Logistic regression and classification

The idea

Linear regression predicts any number. For a yes/no question ("will this customer churn?") we want a probability between 0 and 1. Logistic regression keeps the weighted sum from linear regression and squeezes it into that range with the sigmoid function. Despite its name, it's a classification model, and it's the first one to try for any yes/no problem.

The sigmoid

A linear model produces a score that can be any number, like -6 or 4.2. But "will this customer churn?" needs an answer between 0 and 1, a probability. The sigmoid function squashes any score into that range:

import numpy as np


def sigmoid(z):
    return 1 / (1 + np.exp(-z))


for z in [-6, -2, 0, 2, 6]:
    print(z, round(sigmoid(z), 3))

-6 0.002
-2 0.119
0 0.5
2 0.881
6 0.998

• Large negative scores become probabilities near 0, large positive scores near 1, and a score of 0 becomes exactly 0.5.
• It's smooth and always between 0 and 1, so the output can be read as a probability.

The model: score = w₁x₁ + w₂x₂ + ... + b, then probability = sigmoid(score). Training finds the weights that minimise log loss (from the model-and-loss module) on the training data.

Predicting churn

Now the real task: train a model on the churn data and see how well it predicts which customers will leave. scikit-learn's LogisticRegression combines the linear score and the sigmoid for you:

import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

churn = pd.read_csv("churn.csv")
FEATURES = ["tenure_months", "orders_90d", "days_since_last_order", "support_tickets", "app_user"]
X_train, X_test, y_train, y_test = train_test_split(
    churn[FEATURES], churn["churned"], test_size=0.25, random_state=42, stratify=churn["churned"])

model = LogisticRegression(max_iter=1000)
model.fit(X_train, y_train)

probabilities = model.predict_proba(X_test)[:, 1]
print(probabilities[:5].round(2))
print(model.predict(X_test)[:5])
print("accuracy:", round(model.score(X_test, y_test), 3))

[0.47 0.13 0.54 0.   0.59]
[0 0 1 0 1]
accuracy: 0.8

• predict_proba returns two columns: P(class 0) and P(class 1). [:, 1] takes the churn probability.
• predict turns probabilities into 0/1 using a threshold of 0.5.
• score gives accuracy on the test set. Compare with the 66.5% "nobody churns" baseline: the model is clearly better.

The threshold is a business decision

0.5 isn't sacred. If a retention voucher is cheap and losing a customer is expensive, you'd rather contact more people: lower the threshold.

for threshold in [0.3, 0.5, 0.7]:
    predicted = (probabilities >= threshold).astype(int)
    contacted = predicted.sum()
    caught = ((predicted == 1) & (y_test.to_numpy() == 1)).sum()
    print(f"threshold {threshold}: contact {contacted} customers, catch {caught} of {y_test.sum()} churners")

threshold 0.3: contact 48 customers, catch 30 of 33 churners
threshold 0.5: contact 35 customers, catch 24 of 33 churners
threshold 0.7: contact 22 customers, catch 18 of 33 churners

A lower threshold catches more churners but contacts more customers who weren't leaving. Choosing the threshold is a trade-off between the cost of missed churners and the cost of wasted vouchers, which the metrics module turns into precise numbers (precision and recall).

Reading the coefficients

Like linear regression, each coefficient shows a feature's direction and strength, but on the log-odds scale:

coefs = pd.Series(model.coef_[0], index=FEATURES).round(3)
print(coefs.sort_values())

orders_90d              -0.783
app_user                -0.736
tenure_months           -0.048
days_since_last_order    0.065
support_tickets          1.043
dtype: float64

• Positive coefficients push towards churn: more days since the last order, more support tickets.
• Negative ones push away: more recent orders, longer tenure, using the app.

For a feature x with coefficient w, one extra unit multiplies the odds of churning by exp(w):

print(round(np.exp(coefs["support_tickets"]), 2))

2.84

"Each extra support ticket multiplies the odds of churning by about that much." That's a finding the customer-service team can use, with the usual caveat: association, not proof of cause.

Note that coefficient sizes depend on feature units (a coefficient per day looks small next to one per ticket). To compare importance, scale features first (the preprocessing module).

The decision boundary

Logistic regression draws a straight boundary through feature space: churn on one side, stay on the other. With one feature, that's a single cut-off value:

one = LogisticRegression().fit(churn[["days_since_last_order"]], churn["churned"])
w, b = one.coef_[0][0], one.intercept_[0]
print("50% churn probability at", round(-b / w, 1), "days")

50% churn probability at 45.3 days

At that many days without an order, the model rates a customer 50/50. When the true boundary is curved or depends on combinations of features, a straight line isn't enough, and trees or ensembles (later modules) do better.

Why this matters for clean code

Logistic regression is the classification baseline: fast, stable, explainable, with probabilities that support decisions. Keep predict_proba and the threshold separate in your code (probabilities → decide(probabilities, threshold)), so the business rule can change without retraining the model.

Recap

• Logistic regression = weighted sum → sigmoid → probability; trained by minimising log loss.
• predict_proba(X)[:, 1] gives P(class 1); predict applies a 0.5 threshold.
• Choose the threshold from business costs; lower thresholds catch more positives and raise more false alarms.
• Coefficients: sign = direction; exp(w) = odds multiplier per unit. Scale features to compare them.
• The decision boundary is straight; curved problems need other models.