Overfitting, underfitting, and generalization

The idea

The goal of a model isn't to fit the training data. It's to generalise: to make good predictions on new data it has never seen. Two things can go wrong:

• Underfitting: the model is too simple to capture the real pattern. It's bad on training data and on new data.
• Overfitting: the model is so flexible it memorises the training data, noise included. It's excellent on training data and bad on new data.

An experiment

Take a small sample of days: ad spend vs coffee units, which has a real but noisy upward relationship. Fit polynomials of increasing degree (a degree-1 polynomial is a straight line; higher degrees can bend more) and compare training and test error.

import numpy as np
import pandas as pd

demand = pd.read_csv("daily_demand.csv")
rng = np.random.default_rng(19)
sample = demand.iloc[rng.choice(len(demand), size=30, replace=False)]
train, test = sample.iloc[:20], sample.iloc[20:]


def fit_and_score(degree):
    coefs = np.polyfit(train["ad_spend"], train["units"], degree)
    train_mse = np.mean((np.polyval(coefs, train["ad_spend"]) - train["units"]) ** 2)
    test_mse = np.mean((np.polyval(coefs, test["ad_spend"]) - test["units"]) ** 2)
    return float(round(train_mse, 1)), float(round(test_mse, 1))


for degree in [0, 1, 3, 9, 15]:
    print(degree, fit_and_score(degree))

0 (138.8, 170.5)
1 (85.7, 94.2)
3 (84.0, 110.8)
9 (31.0, 531.1)
15 (3.8, 9636.3)

(You may also see a RankWarning: NumPy warning that a degree-15 polynomial on 20 points is numerically shaky. That's the point.)

Read the table from top to bottom:

• Degree 0 (a flat line: "always predict the average") underfits: both errors are high.
• Degree 1 captures the real trend; training and test errors are similar.
• As the degree grows, training error keeps falling, because the curve can wiggle through more points. But test error explodes: the wiggles are fitted to noise, and they swing wildly between the training points.

With 20 training points, a degree-15 polynomial has enough flexibility to almost pass through every point. It "learned" the noise.

The two symptoms

The gap between training and test error is the signature of overfitting.

Memorising: the extreme case

A "model" that just stores the training data in a dictionary has zero training error:

lookup = dict(zip(train["date"], train["units"]))
train_errors = [lookup[d] - u for d, u in zip(train["date"], train["units"])]
print("training MSE:", np.mean(np.square(train_errors)))
print("can predict a new date?", test["date"].iloc[0] in lookup)

training MSE: 0.0
can predict a new date? False

Perfect on the past, useless for the future. Every overfit model is a softer version of this lookup table.

What helps generalisation

• More data. The more examples, the harder it is to memorise noise, so flexible models become safer.
• Simpler models. Prefer the simplest model that captures the pattern. Start simple, add complexity only when validation error improves.
• Good features. One informative feature (days since last order) beats ten noisy ones.
• Regularisation. Penalise complexity during training (a later module).
• Honest evaluation. A separate validation set or cross-validation to choose the model, and a test set touched once at the end (the module after next).

A quick check you can always do

How do you know whether your own model is overfitting? Compare how wrong it is on the data it trained on with how wrong it is on data it has never seen. Here's that check for a model that predicts daily units sold from the weather, the day of the week, promotions and ad spend:

from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

X = demand[["temperature", "weekend", "promo", "ad_spend"]]
y = demand["units"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0)
model = LinearRegression().fit(X_train, y_train)
train_mse = np.mean((model.predict(X_train) - y_train) ** 2)
test_mse = np.mean((model.predict(X_test) - y_test) ** 2)
print(round(train_mse, 1), round(test_mse, 1))

23.4 29.4

Training and test errors are close: this simple linear model generalises well. If the test error were several times the training error, you'd have an overfitting problem to fix before trusting it.

Why this matters for clean code

Always reporting both training and test performance is the ML equivalent of writing tests: it catches the bug (overfitting) that looks like success. Build the comparison into your workflow (a small evaluate(model) function that prints both), and you'll never ship a model that only works on yesterday's data.

Recap

• The goal is generalisation: good predictions on unseen data.
• Underfitting: too simple, bad everywhere. Overfitting: too flexible, great on training data, bad on new data.
• Training error always falls with flexibility; judge models on held-out data.
• A large gap between training and test error signals overfitting.
• More data, simpler models, better features, regularisation and honest evaluation all help.