Train Test Split

Teaching Computers to Learn - Train/Test Split & Model Evaluation

Goal: Understand how we know whether a machine learning model has actually learned something useful.

So far, we have seen how regression models can learn relationships between input variables and a numerical target. We have also seen that a model can fit the data we give it.

Now we need to answer a much more important question:

How do we know whether our model will work on data it has never seen before?

This is where the train/test split comes in.

1. The Problem: A Model Can Memorize

Imagine that we give a model a dataset containing information about houses and their prices.

We ask the model to learn:

House information → House price

The model sees many examples:

House A → $300,000
House B → $450,000
House C → $275,000
...

After training, we can ask it to predict prices.

But there is a problem.

If we evaluate the model using the same houses it learned from, we are asking:

"How well can you predict examples you have already seen?"

That isn't quite what we care about.

What we really want to know is:

"How well can you predict a house you've never seen before?"

This is the reason we separate our data into training data and testing data.

2. Train Data vs Test Data

We split our dataset into two parts:

                    Entire Dataset
                         │
              ┌──────────┴──────────┐
              │                     │
           Training               Testing
             Data                   Data
              │                     │
              ▼                     │
        Train the model              │
              │                     │
              └──────────┐           │
                         ▼           ▼
                       Model      Evaluate

Training data

The model is allowed to learn from this data.

We use it to:

• find patterns
• learn parameters
• fit the model

Testing data

The model is not allowed to learn from this data.

We keep it hidden until we want to evaluate the finished model.

This gives us a much better approximation of how the model might perform on new, unseen data.

3. Load the Boston Housing Dataset

We are going to use the classic Boston Housing dataset.

The dataset contains information about houses in different Boston-area neighborhoods and a numerical target representing the median house value.

Important: load_boston() was removed from newer versions of scikit-learn. We can still retrieve the same classic dataset through scikit-learn's fetch_openml() interface.

Let's import what we need:

import pandas as pd

from sklearn.datasets import fetch_openml

Now load the dataset:

boston = fetch_openml(
    name="boston",
    version=1,
    as_frame=True
)

The features are available through:

X = boston.data

And the target is:

y = boston.target

Let's inspect the data:

X.head()

And:

y.head()

Our basic machine learning problem is now:

X = information about the house

        ↓

Machine Learning Model

        ↓

y = house value

Check for data integrity

• Check if there are any missing values. If there are, substitute them with their column's mean, median or as per business objective.

There are some category columns here. Let's check.

For now to keep things simple we will convert categories to numeric values.

4. What Are X and y?

You'll see this notation constantly in machine learning.

X

X contains the features.

These are the pieces of information we give to the model.

For example, the dataset contains variables describing things such as:

• crime rate
• number of rooms
• accessibility
• property characteristics
• neighborhood information

y

y contains the target.

This is what we want the model to predict.

In this case:

y = house value

So:

X = boston.data
y = boston.target

means:

"Use the columns in X to predict the values in y."

5. Splitting the Data

Scikit-learn gives us a convenient function for this:

from sklearn.model_selection import train_test_split

Now:

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42
)

We have now created four datasets:

X_train → training features
y_train → training answers

X_test  → testing features
y_test  → testing answers

The model gets:

X_train + y_train

It does not get:

y_test

until we want to evaluate it.

6. What Does test_size=0.2 Mean?

This:

test_size=0.2

means:

20% of the data goes into the test set.

The remaining:

80%

goes into the training set.

So if we had 100 examples:

100 examples
│
├── 80 → training
└── 20 → testing

A common split is 80/20, although other splits are possible.

7. Why random_state=42?

The train/test split is random.

If we run:

train_test_split(X, y, test_size=0.2)

we may get a different collection of training and testing examples each time.

That's not necessarily bad.

But while learning, it makes experiments harder to reproduce.

So we use:

random_state=42

This gives us the same split every time we run the code.

The number 42 isn't magical.

You could use another integer.

The important idea is:

Use a fixed random state when you want reproducible experiments.

8. Look at the Shapes

Let's see how much data ended up in each set:

print(X_train.shape)
print(X_test.shape)

print(y_train.shape)
print(y_test.shape)

You should see that the training set contains roughly 80% of the examples and the test set contains roughly 20%.

This is worth checking because it makes the split concrete.

9. Train a Model

Now let's train a simple linear regression model.

from sklearn.linear_model import LinearRegression

model = LinearRegression()

Train it:

model.fit(X_train, y_train)

Notice something important.

We are training using:

X_train
y_train

We are not using:

X_test
y_test

The test set remains untouched.

10. Make Predictions on the Training Data

We can ask the model to predict the training examples:

train_predictions = model.predict(X_train)

Then we can compare:

Actual training values
        vs
Predicted training values

This tells us how well the model fits the data it learned from.

11. Make Predictions on the Test Data

Now comes the important part.

We give the model the test features:

test_predictions = model.predict(X_test)

Notice that we only give it:

X_test

We don't give it:

y_test

The model has never seen the answers.

It has to make predictions.

Only after it has made those predictions do we compare them with:

y_test

This is the basic evaluation workflow:

Training:

X_train + y_train
       ↓
     Model


Testing:

X_test
  ↓
Model
  ↓
Predictions
  ↓
Compare with y_test

12. Measuring Regression Performance with R²

A model's predictions aren't very useful unless we have a way to measure how good they are.

For regression, one metric we can use is R².

Scikit-learn provides:

from sklearn.metrics import r2_score

Calculate the training R²:

train_r2 = r2_score(
    y_train,
    train_predictions
)

And the test R²:

test_r2 = r2_score(
    y_test,
    test_predictions
)

Print them:

print("Train R²:", train_r2)
print("Test R²:", test_r2)

13. What Does R² Tell Us?

R² gives us a way to describe how much of the variation in the target is explained by our model.

A simplified intuition:

R² close to 1
    ↓
Model explains a lot of the variation

R² around 0
    ↓
Model explains very little

R² below 0
    ↓
Model is performing worse than
a simple baseline that predicts
the average target value

For example:

Train R² = 0.75
Test R²  = 0.68

This suggests that the model performs reasonably well on both the data it learned from and the unseen data.

14. Why Calculate R² Twice?

This is the important part.

We don't just calculate:

r2_score(y_test, test_predictions)

We also calculate:

r2_score(y_train, train_predictions)

because the difference between train and test performance tells us something about the model.

Consider:

                 Train R²     Test R²

Model A            0.72         0.69
Model B            0.99         0.45

Which model would you trust more?

Probably Model A.

Why?

Model B is extremely good at predicting the examples it trained on.

But its performance drops dramatically on unseen examples.

That's a warning sign.

15. The Beginning of Overfitting

Remember the idea of overfitting.

An overfit model has learned the training data too specifically.

It may capture:

• useful patterns
• noise
• random quirks
• accidental relationships

The result can look like:

Train performance
       ↓
      VERY GOOD

Test performance
       ↓
       BAD

For example:

Train R² = 0.99
Test R²  = 0.40

The model looks impressive if we only look at training performance.

But the test score tells a different story.

This is why test data matters.

16. Underfitting

The opposite problem is underfitting.

An underfit model is too simple to capture the patterns in the data.

It may look like:

Train R² = 0.35
Test R²  = 0.30

The model isn't performing particularly well anywhere.

So we can develop a useful intuition:

                    Train       Test

Good fit             good        good

Overfitting          great       poor

Underfitting         poor        poor

This is a simplified mental model, not a universal rule.

17. Let's See This in Code

Put everything together:

import pandas as pd

from sklearn.datasets import fetch_openml
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score


# -------------------------
# Load the dataset
# -------------------------

boston = fetch_openml(
    name="boston",
    version=1,
    as_frame=True
)

X = boston.data.apply(pd.to_numeric)
y = pd.to_numeric(boston.target)


# -------------------------
# Train-test split
# -------------------------

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42
)


# -------------------------
# Create and train model
# -------------------------

model = LinearRegression()

model.fit(
    X_train,
    y_train
)


# -------------------------
# Predictions
# -------------------------

train_predictions = model.predict(X_train)
test_predictions = model.predict(X_test)


# -------------------------
# Evaluation
# -------------------------

train_r2 = r2_score(
    y_train,
    train_predictions
)

test_r2 = r2_score(
    y_test,
    test_predictions
)


print("Train R²:", train_r2)
print("Test R²:", test_r2)

The important part isn't memorizing this code.

It is understanding the sequence:

Load data
   ↓
Separate X and y
   ↓
Split train/test
   ↓
Train model using training data
   ↓
Predict training data
   ↓
Predict test data
   ↓
Calculate train metric
   ↓
Calculate test metric
   ↓
Compare

18. What If We Only Used the Training Score?

Imagine we train a complicated model and get:

Train R² = 0.98

We might say:

"Amazing! Our model is 98% accurate."

But that conclusion would be premature.

We haven't tested the model on unseen data.

Suppose:

Train R² = 0.98
Test R²  = 0.41

Now we have a very different interpretation.

The model has learned the training examples extremely well, but it doesn't generalize well.

The test score exposes this problem.

19. What About Classification?

So far, Boston Housing is a regression problem because the target is numerical.

For example:

Predict house value:

$250,000
$320,000
$410,000

But many machine learning problems are classification problems.

Classification predicts categories.

For example:

Will this customer churn?

Yes
No

Or:

Is this email spam?

Spam
Not Spam

For classification, one simple metric is accuracy.

20. Accuracy

Accuracy asks:

What percentage of predictions did the model get correct?

Suppose our model makes 100 predictions.

If 85 are correct:

Accuracy = 85 / 100
         = 0.85
         = 85%

In scikit-learn:

from sklearn.metrics import accuracy_score

accuracy = accuracy_score(
    y_test,
    predictions
)

21. Train Accuracy vs Test Accuracy

Just like regression, we can calculate the metric on both datasets.

                 Train Accuracy    Test Accuracy

Model A               91%              89%

Model B               99%              72%

Model B is suspicious.

It performs almost perfectly on its training examples but much worse on unseen examples.

Again:

High train performance
+
Low test performance
        ↓
Possible overfitting

This is the same fundamental idea we saw with R².

The metric changes depending on the problem, but the evaluation principle stays the same.

22. R² vs Accuracy

The metric should match the type of prediction we are making.

For our Boston Housing problem:

Features
   ↓
Regression model
   ↓
House value

We use:

r2_score()

For a classification problem:

Features
   ↓
Classification model
   ↓
Category

we might use:

accuracy_score()

There are many other metrics, and we will encounter them later.

23. A Very Important Warning About Accuracy

Accuracy can sometimes be misleading.

Imagine a dataset where:

95% of customers do NOT churn
5% of customers DO churn

A terrible model could predict:

"No churn"

for every customer.

It would get:

95 out of 100 correct

So its accuracy would be:

95%

That sounds excellent.

But the model completely failed to identify the customers who actually churn.

So:

A metric is only useful when we understand what it is measuring.

We'll explore other classification metrics later.

24. The Golden Rule

There is one idea you should remember from this entire tutorial:

Never judge a machine learning model only by how well it performs on the data it was trained on.

Training performance tells us:

"How well did the model fit the examples it learned from?"

Test performance asks:

"How well does the model perform on examples it never saw during training?"

The second question is much closer to what we actually care about.

25. The Machine Learning Workflow

We can now expand our basic machine learning workflow:

                DATA
                  ↓
           Separate X and y
                  ↓
          Train/Test Split
             ↙         ↘
        Training       Testing
           ↓              ↓
       Train Model     Keep Hidden
           ↓              │
           └──────┬───────┘
                  ↓
              Predictions
                  ↓
              Evaluation
                  ↓
        Train Metric vs Test Metric
                  ↓
       Does the model generalize?

This is the beginning of model evaluation.

26. Your Experiment

Now change the model.

Try a few different regression models on the same train/test split.

For example:

from sklearn.tree import DecisionTreeRegressor

and:

from sklearn.ensemble import RandomForestRegressor

Train each model using:

X_train
y_train

Then calculate:

Train R²
Test R²

Create a table:

Model                  Train R²       Test R²

Linear Regression       ...
Decision Tree            ...
Random Forest            ...

Now ask yourself:

1. Which model has the highest training score?
2. Which model has the highest test score?
3. Which model has the biggest train/test gap?
4. Is the model with the best training score necessarily the best model?
5. Which model would you choose for new houses?

Don't just look for the biggest number.

Think about what the numbers are telling you.

The Big Idea

Machine learning is not:

"Train a model and look at its score."

It is:

"Train a model, test it on unseen data, and ask whether it generalizes."

The train/test split gives us a way to simulate the future:

Past data
    ↓
Training
    ↓
Model
    ↓
Future / unseen data
    ↓
Testing

And the difference between training and testing performance gives us one of our first clues about whether the model is learning useful patterns or simply fitting the data it has already seen.