Train/validation/test split and cross-validation

The idea

You'll rarely train just one model. You'll compare models, try feature sets and tune settings like "how deep can a tree go?" Each comparison is a decision based on data. If you make those decisions using the test set, you slowly fit your choices to it, and its score stops being an honest estimate. The test set has to stay a surprise.

The fix: a third set for decisions.

A three-way split

If you try several models and keep the one that does best on the test set, the test set has quietly become part of the training: you've chosen the model that happens to suit it. The fix is a third slice of data, the validation set, for comparing models, with the test set kept back for one final check. You get there by splitting twice:

import pandas as pd
from sklearn.model_selection import train_test_split

churn = pd.read_csv("churn.csv")
FEATURES = ["tenure_months", "orders_90d", "days_since_last_order", "support_tickets", "app_user"]
X, y = churn[FEATURES], churn["churned"]

X_temp, X_test, y_temp, y_test = train_test_split(X, y, test_size=0.2, random_state=0, stratify=y)
X_train, X_val, y_train, y_val = train_test_split(X_temp, y_temp, test_size=0.25, random_state=0, stratify=y_temp)
print(len(X_train), len(X_val), len(X_test))

240 80 80

60% train, 20% validation, 20% test. Now compare candidate models on the validation set:

from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier

candidates = {
    "logistic": LogisticRegression(max_iter=1000),
    "tree depth 2": DecisionTreeClassifier(max_depth=2, random_state=0),
    "tree depth 10": DecisionTreeClassifier(max_depth=10, random_state=0),
}
for name, model in candidates.items():
    model.fit(X_train, y_train)
    print(f"{name:14} train {model.score(X_train, y_train):.3f}  validation {model.score(X_val, y_val):.3f}")

logistic       train 0.833  validation 0.812
tree depth 2   train 0.783  validation 0.713
tree depth 10  train 0.992  validation 0.775

The deep tree memorises the training data (near-perfect training accuracy) but scores lower on validation: overfitting, caught without touching the test set. Choose by validation score, then evaluate only the winner on the test set, once.

The problem with one split

With 80 validation customers, a single unlucky split can make a model look 5 points better or worse than it really is. Try a few different random splits and watch the score move:

for seed in [0, 1, 2, 3, 4]:
    Xa, Xb, ya, yb = train_test_split(X_temp, y_temp, test_size=0.25, random_state=seed, stratify=y_temp)
    print(seed, round(LogisticRegression(max_iter=1000).fit(Xa, ya).score(Xb, yb), 3))

0 0.812
1 0.775
2 0.812
3 0.887
4 0.812

Same model, same data, scores spread over several points. Small datasets make single splits noisy.

k-fold cross-validation

Cross-validation uses every example for validation exactly once. With k = 5:

1. Split the (non-test) data into 5 equal folds.
2. Train on 4 folds, validate on the 5th.
3. Repeat 5 times, each fold taking a turn as validation.
4. Average the 5 scores.

from sklearn.model_selection import cross_val_score

scores = cross_val_score(LogisticRegression(max_iter=1000), X_temp, y_temp, cv=5)
print(scores.round(3))
print(f"mean {scores.mean():.3f} ± {scores.std():.3f}")

[0.828 0.891 0.812 0.828 0.812]
mean 0.834 ± 0.029

You get an average and a sense of the spread. For classification, scikit-learn stratifies the folds automatically. The cost is k times the training time, which is nothing for models like these.

Tuning with a grid search

Settings you choose before training (tree depth, regularisation strength) are called hyperparameters, as opposed to the parameters the model learns. GridSearchCV tries every combination with cross-validation and keeps the best:

from sklearn.model_selection import GridSearchCV

search = GridSearchCV(
    DecisionTreeClassifier(random_state=0),
    param_grid={"max_depth": [1, 2, 3, 4, 6, 8], "min_samples_leaf": [1, 5, 20]},
    cv=5,
)
search.fit(X_temp, y_temp)
print(search.best_params_, round(search.best_score_, 3))
print("test:", round(search.score(X_test, y_test), 3))

{'max_depth': 3, 'min_samples_leaf': 1} 0.791
test: 0.738

best_score_ is the cross-validated score; after the search, search refits the best settings on all of X_temp, and only then do we look at the test set. Here the test score is a few points below the CV score. With only 80 test customers that's within the noise you saw in the split experiment above; a much bigger drop would mean the search had overfitted the validation folds.

Special cases

• Time series: never shuffle. Validate on later periods than you train on (TimeSeriesSplit), because in real use you'll always predict the future from the past.
• Groups: if one customer has many rows, keep all their rows in the same fold (GroupKFold), or the model is tested on people it has already seen.

Why this matters for clean code

A validation process is part of your code, not an afterthought: a fixed split or CV scheme with seeds, one function that scores a model, and a test set that's evaluated exactly once in a clearly marked final step. That structure is what lets you (and reviewers) trust a reported number.

Recap

• Training set fits; validation set (or CV) chooses; test set reports, once.
• One split can be noisy; k-fold cross-validation averages k validation scores.
• cross_val_score(model, X, y, cv=5) returns one score per fold.
• Hyperparameters (like max_depth) are tuned with GridSearchCV(model, param_grid, cv=5).
• Time series: validate on later data; grouped data: keep groups together.