Classification metrics: accuracy, precision, recall, F1

The idea

"The model is 80% accurate" sounds good, until you remember that predicting "nobody churns" is already 66.5% accurate on this data, and that a fraud model can be 99% accurate while catching no fraud at all. Accuracy treats every mistake the same. In business, mistakes have different costs:

• Missing a churner (no voucher sent, customer lost) costs a customer.
• Flagging a loyal customer as a churner costs a voucher.

To see the trade-off, count the four possible outcomes.

The four outcomes

For a "positive" class (churned = 1):

import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

churn = pd.read_csv("churn.csv")
FEATURES = ["tenure_months", "orders_90d", "days_since_last_order", "support_tickets", "app_user"]
X_train, X_test, y_train, y_test = train_test_split(
    churn[FEATURES], churn["churned"], test_size=0.25, random_state=42, stratify=churn["churned"])
model = LogisticRegression(max_iter=1000).fit(X_train, y_train)

y_true = y_test.to_numpy()
y_pred = model.predict(X_test)
tp = int(((y_pred == 1) & (y_true == 1)).sum())
fp = int(((y_pred == 1) & (y_true == 0)).sum())
fn = int(((y_pred == 0) & (y_true == 1)).sum())
tn = int(((y_pred == 0) & (y_true == 0)).sum())
print(tp, fp, fn, tn)

24 11 9 56

Precision and recall

Two questions, two metrics:

• Precision = TP / (TP + FP): of the customers we flagged, how many really churned? High precision = few wasted vouchers.
• Recall = TP / (TP + FN): of the customers who churned, how many did we catch? High recall = few lost customers slipped through.

precision = tp / (tp + fp)
recall = tp / (tp + fn)
accuracy = (tp + tn) / len(y_true)
print(round(precision, 3), round(recall, 3), round(accuracy, 3))

0.686 0.727 0.8

They pull in opposite directions. Lower the threshold and you flag more people: recall rises (you catch more churners), precision falls (more false alarms). Raise it and the reverse happens.

probs = model.predict_proba(X_test)[:, 1]
for threshold in [0.2, 0.35, 0.5, 0.65, 0.8]:
    pred = (probs >= threshold).astype(int)
    tp_t = ((pred == 1) & (y_true == 1)).sum()
    p = tp_t / max(pred.sum(), 1)
    r = tp_t / y_true.sum()
    print(f"{threshold:.2f}  precision {p:.2f}  recall {r:.2f}")

0.20  precision 0.53  recall 0.97
0.35  precision 0.69  recall 0.88
0.50  precision 0.69  recall 0.73
0.65  precision 0.79  recall 0.58
0.80  precision 0.87  recall 0.39

F1: one number for both

When you need one number that balances them, the F1 score is their harmonic mean:

F1 = 2 × precision × recall / (precision + recall)

The harmonic mean punishes imbalance: precision 1.0 with recall 0.1 gives F1 0.18, not 0.55. F1 is high only when both are decent.

With scikit-learn

You've now worked out each metric by hand from the four counts, which is the best way to understand them. In practice, scikit-learn calculates them for you, and classification_report prints a full summary in one call:

from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, classification_report

print(round(precision_score(y_true, y_pred), 3), round(recall_score(y_true, y_pred), 3), round(f1_score(y_true, y_pred), 3))
print(classification_report(y_true, y_pred, target_names=["stayed", "churned"]))

0.686 0.727 0.706
              precision    recall  f1-score   support

      stayed       0.86      0.84      0.85        67
     churned       0.69      0.73      0.71        33

    accuracy                           0.80       100
   macro avg       0.77      0.78      0.78       100
weighted avg       0.80      0.80      0.80       100

classification_report shows precision, recall and F1 for each class, plus the number of examples (support). Read the row of the class you care about, usually the rarer one.

Argument order matters: metrics take (y_true, y_pred). Swapping them silently swaps precision and recall.

Choosing the metric

Why this matters for clean code

Metrics are the contract between a model and the business. Write down the chosen metric and threshold, and why, next to the code that computes them. When the model is retrained, the same function scores it the same way, and nobody can quietly switch to a more flattering number.

Recap

• Four outcomes: TP, FP (false alarm), FN (missed), TN.
• Precision = TP/(TP+FP): how trustworthy a positive prediction is.
• Recall = TP/(TP+FN): how many actual positives are found.
• F1 = harmonic mean of precision and recall.
• precision_score, recall_score, f1_score, classification_report, always (y_true, y_pred).
• Pick the metric and threshold from the cost of each kind of mistake.