Teaching Computers to Learn•Interactive notebook lesson•Free
Train Test Split
No account is required. Learn the material, try the examples, and mark it complete locally when you are ready to move on.
Teaching Computers to Learn - Train/Test Split & Model Evaluation
Goal: Understand how we know whether a machine learning model has actually learned something useful.
So far, we have seen how regression models can learn relationships between input variables and a numerical target. We have also seen that a model can fit the data we give it.
Now we need to answer a much more important question:
How do we know whether our model will work on data it has never seen before?
This is where the train/test split comes in.
1. The Problem: A Model Can Memorize
Imagine that we give a model a dataset containing information about houses and their prices.
We ask the model to learn:
text
House information → House price
The model sees many examples:
text
House A → $300,000
House B → $450,000
House C → $275,000
...
After training, we can ask it to predict prices.
But there is a problem.
If we evaluate the model using the same houses it learned from, we are asking:
"How well can you predict examples you have already seen?"
That isn't quite what we care about.
What we really want to know is:
"How well can you predict a house you've never seen before?"
This is the reason we separate our data into training data and testing data.
2. Train Data vs Test Data
We split our dataset into two parts:
text
Entire Dataset
│
┌──────────┴──────────┐
│ │
Training Testing
Data Data
│ │
▼ │
Train the model │
│ │
└──────────┐ │
▼ ▼
Model Evaluate
Training data
The model is allowed to learn from this data.
We use it to:
find patterns
learn parameters
fit the model
Testing data
The model is not allowed to learn from this data.
We keep it hidden until we want to evaluate the finished model.
This gives us a much better approximation of how the model might perform on new, unseen data.
3. Load the Boston Housing Dataset
We are going to use the classic Boston Housing dataset.
The dataset contains information about houses in different Boston-area neighborhoods and a numerical target representing the median house value.
Important:load_boston() was removed from newer versions of scikit-learn. We can still retrieve the same classic dataset through scikit-learn's fetch_openml() interface.
Let's import what we need:
python
import pandas as pd
from sklearn.datasets import fetch_openml
Now load the dataset:
python
boston = fetch_openml(
name="boston",
version=1,
as_frame=True
)
The features are available through:
python
X = boston.data
And the target is:
python
y = boston.target
Let's inspect the data:
python
X.head()
And:
python
y.head()
Our basic machine learning problem is now:
text
X = information about the house
↓
Machine Learning Model
↓
y = house value
Codepython
import pandas as pandas
from sklearn.datasets import fetch_openml
boston = fetch_openml(
name="boston",
version=1,
as_frame=True
)
Output
/opt/envs/ds/lib/python3.10/site-packages/sklearn/datasets/_openml.py:1022: FutureWarning: The default value of `parser` will change from `'liac-arff'` to `'auto'` in 1.4. You can set `parser='auto'` to silence this warning. Therefore, an `ImportError` will be raised from 1.4 if the dataset is dense and pandas is not installed. Note that the pandas parser may return different data types. See the Notes Section in fetch_openml's API doc for details.
warn(
Train R2: 0.7508856358979672
Test R2: 0.6687594935356338
13. What Does R² Tell Us?
R² gives us a way to describe how much of the variation in the target is explained by our model.
A simplified intuition:
text
R² close to 1
↓
Model explains a lot of the variation
R² around 0
↓
Model explains very little
R² below 0
↓
Model is performing worse than
a simple baseline that predicts
the average target value
For example:
text
Train R² = 0.75
Test R² = 0.68
This suggests that the model performs reasonably well on both the data it learned from and the unseen data.
14. Why Calculate R² Twice?
This is the important part.
We don't just calculate:
python
r2_score(y_test, test_predictions)
We also calculate:
python
r2_score(y_train, train_predictions)
because the difference between train and test performance tells us something about the model.
Consider:
text
Train R² Test R²
Model A 0.72 0.69
Model B 0.99 0.45
Which model would you trust more?
Probably Model A.
Why?
Model B is extremely good at predicting the examples it trained on.
But its performance drops dramatically on unseen examples.
That's a warning sign.
15. The Beginning of Overfitting
Remember the idea of overfitting.
An overfit model has learned the training data too specifically.
It may capture:
useful patterns
noise
random quirks
accidental relationships
The result can look like:
text
Train performance
↓
VERY GOOD
Test performance
↓
BAD
For example:
text
Train R² = 0.99
Test R² = 0.40
The model looks impressive if we only look at training performance.
But the test score tells a different story.
This is why test data matters.
16. Underfitting
The opposite problem is underfitting.
An underfit model is too simple to capture the patterns in the data.
It may look like:
text
Train R² = 0.35
Test R² = 0.30
The model isn't performing particularly well anywhere.
So we can develop a useful intuition:
text
Train Test
Good fit good good
Overfitting great poor
Underfitting poor poor
This is a simplified mental model, not a universal rule.
17. Let's See This in Code
Put everything together:
python
import pandas as pd
from sklearn.datasets import fetch_openml
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score
# -------------------------
# Load the dataset
# -------------------------
boston = fetch_openml(
name="boston",
version=1,
as_frame=True
)
X = boston.data.apply(pd.to_numeric)
y = pd.to_numeric(boston.target)
# -------------------------
# Train-test split
# -------------------------
X_train, X_test, y_train, y_test = train_test_split(
X,
y,
test_size=0.2,
random_state=42
)
# -------------------------
# Create and train model
# -------------------------
model = LinearRegression()
model.fit(
X_train,
y_train
)
# -------------------------
# Predictions
# -------------------------
train_predictions = model.predict(X_train)
test_predictions = model.predict(X_test)
# -------------------------
# Evaluation
# -------------------------
train_r2 = r2_score(
y_train,
train_predictions
)
test_r2 = r2_score(
y_test,
test_predictions
)
print("Train R²:", train_r2)
print("Test R²:", test_r2)
The important part isn't memorizing this code.
It is understanding the sequence:
text
Load data
↓
Separate X and y
↓
Split train/test
↓
Train model using training data
↓
Predict training data
↓
Predict test data
↓
Calculate train metric
↓
Calculate test metric
↓
Compare
Codepython
import pandas as pd
from sklearn.datasets import fetch_openml
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score
# -------------------------
# Load the dataset
# -------------------------
boston = fetch_openml(
name="boston",
version=1,
as_frame=True
)
X = boston.data.apply(pd.to_numeric)
y = pd.to_numeric(boston.target)
# -------------------------
# Train-test split
# -------------------------
X_train, X_test, y_train, y_test = train_test_split(
X,
y,
test_size=0.2,
random_state=42
)
# -------------------------
# Create and train model
# -------------------------
model = LinearRegression()
model.fit(
X_train,
y_train
)
# -------------------------
# Predictions
# -------------------------
train_predictions = model.predict(X_train)
test_predictions = model.predict(X_test)
# -------------------------
# Evaluation
# -------------------------
train_r2 = r2_score(
y_train,
train_predictions
)
test_r2 = r2_score(
y_test,
test_predictions
)
print("Train R²:", train_r2)
print("Test R²:", test_r2)
Output
Train R²: 0.7508856358979672
Test R²: 0.6687594935356338
/opt/envs/ds/lib/python3.10/site-packages/sklearn/datasets/_openml.py:1022: FutureWarning: The default value of `parser` will change from `'liac-arff'` to `'auto'` in 1.4. You can set `parser='auto'` to silence this warning. Therefore, an `ImportError` will be raised from 1.4 if the dataset is dense and pandas is not installed. Note that the pandas parser may return different data types. See the Notes Section in fetch_openml's API doc for details.
warn(
18. What If We Only Used the Training Score?
Imagine we train a complicated model and get:
text
Train R² = 0.98
We might say:
"Amazing! Our model is 98% accurate."
But that conclusion would be premature.
We haven't tested the model on unseen data.
Suppose:
text
Train R² = 0.98
Test R² = 0.41
Now we have a very different interpretation.
The model has learned the training examples extremely well, but it doesn't generalize well.
The test score exposes this problem.
19. What About Classification?
So far, Boston Housing is a regression problem because the target is numerical.
For example:
text
Predict house value:
$250,000
$320,000
$410,000
But many machine learning problems are classification problems.
Classification predicts categories.
For example:
text
Will this customer churn?
Yes
No
Or:
text
Is this email spam?
Spam
Not Spam
For classification, one simple metric is accuracy.
20. Accuracy
Accuracy asks:
What percentage of predictions did the model get correct?
Suppose our model makes 100 predictions.
If 85 are correct:
text
Accuracy = 85 / 100
= 0.85
= 85%
In scikit-learn:
python
from sklearn.metrics import accuracy_score
accuracy = accuracy_score(
y_test,
predictions
)
21. Train Accuracy vs Test Accuracy
Just like regression, we can calculate the metric on both datasets.
text
Train Accuracy Test Accuracy
Model A 91% 89%
Model B 99% 72%
Model B is suspicious.
It performs almost perfectly on its training examples but much worse on unseen examples.
Again:
text
High train performance
+
Low test performance
↓
Possible overfitting
This is the same fundamental idea we saw with R².
The metric changes depending on the problem, but the evaluation principle stays the same.
22. R² vs Accuracy
The metric should match the type of prediction we are making.
Problem
Prediction
Example metric
Regression
Numerical value
R²
Classification
Category
Accuracy
For our Boston Housing problem:
text
Features
↓
Regression model
↓
House value
We use:
python
r2_score()
For a classification problem:
text
Features
↓
Classification model
↓
Category
we might use:
python
accuracy_score()
There are many other metrics, and we will encounter them later.
23. A Very Important Warning About Accuracy
Accuracy can sometimes be misleading.
Imagine a dataset where:
text
95% of customers do NOT churn
5% of customers DO churn
A terrible model could predict:
text
"No churn"
for every customer.
It would get:
text
95 out of 100 correct
So its accuracy would be:
text
95%
That sounds excellent.
But the model completely failed to identify the customers who actually churn.
So:
A metric is only useful when we understand what it is measuring.
We'll explore other classification metrics later.
24. The Golden Rule
There is one idea you should remember from this entire tutorial:
Never judge a machine learning model only by how well it performs on the data it was trained on.
Training performance tells us:
"How well did the model fit the examples it learned from?"
Test performance asks:
"How well does the model perform on examples it never saw during training?"
The second question is much closer to what we actually care about.
25. The Machine Learning Workflow
We can now expand our basic machine learning workflow:
text
DATA
↓
Separate X and y
↓
Train/Test Split
↙ ↘
Training Testing
↓ ↓
Train Model Keep Hidden
↓ │
└──────┬───────┘
↓
Predictions
↓
Evaluation
↓
Train Metric vs Test Metric
↓
Does the model generalize?
This is the beginning of model evaluation.
26. Your Experiment
Now change the model.
Try a few different regression models on the same train/test split.
For example:
python
from sklearn.tree import DecisionTreeRegressor
and:
python
from sklearn.ensemble import RandomForestRegressor
Train each model using:
python
X_train
y_train
Then calculate:
text
Train R²
Test R²
Create a table:
text
Model Train R² Test R²
Linear Regression ...
Decision Tree ...
Random Forest ...
Now ask yourself:
Which model has the highest training score?
Which model has the highest test score?
Which model has the biggest train/test gap?
Is the model with the best training score necessarily the best model?
Which model would you choose for new houses?
Don't just look for the biggest number.
Think about what the numbers are telling you.
The Big Idea
Machine learning is not:
"Train a model and look at its score."
It is:
"Train a model, test it on unseen data, and ask whether it generalizes."
The train/test split gives us a way to simulate the future:
text
Past data
↓
Training
↓
Model
↓
Future / unseen data
↓
Testing
And the difference between training and testing performance gives us one of our first clues about whether the model is learning useful patterns or simply fitting the data it has already seen.
Codepython
Finished this lesson?
Completion is saved in this browser without creating an account.