The idea of a model and a loss function · Machine Learning

See a model as a function with parameters, measure how wrong it is with a loss function, and find the best parameters by searching for the lowest loss.

The idea

Two words carry most of machine learning:

A model is a function that turns features into a prediction, with some adjustable numbers inside, its parameters.
A loss function is a single number that says how wrong the model's predictions are on the training data. Lower is better.

Training = finding the parameters with the lowest loss. Everything else (linear regression, neural networks) is a variation on that sentence.

A model for coffee demand

daily_demand.csv has a year of daily coffee sales with the day's ad spend (in million Rupiah):

import numpy as np
import pandas as pd

demand = pd.read_csv("daily_demand.csv")
ad = demand["ad_spend"].to_numpy()
units = demand["units"].to_numpy()
print(demand.head(3))
print(len(demand))

         date  temperature  weekend  promo  ad_spend  units
0  2024-01-01         26.4        0      0      2.45     57
1  2024-01-02         27.4        0      0      2.62     63
2  2024-01-03         27.2        0      0      2.05     51
366

A simple model: units sold = w × ad spend + b. Two parameters: w (extra units per million Rupiah of ads) and b (units sold with no ads).

def predict(ad, w, b):
    return w * ad + b


print(predict(np.array([0.0, 1.0, 2.0]), w=8, b=45))

[45. 53. 61.]

Different w and b give different models. Which pair is best?

Measuring wrongness: the loss

For each day, the error (residual) is prediction minus actual. To summarise many errors in one number, the most common choice for regression is the mean squared error (MSE): square each error, then average.

def mse(predicted, actual):
    return np.mean((predicted - actual) ** 2)


for w, b in [(0, 56), (8, 45), (15, 30)]:
    print(w, b, round(mse(predict(ad, w, b), units), 1))

0 56 131.3
8 45 90.0
15 30 140.6

The middle model has the lowest loss of the three, so it fits best. Why square the errors?

Errors of +5 and −5 shouldn't cancel out; squaring makes both positive.
Big errors count much more than small ones (an error of 10 costs 100; two errors of 5 cost 50 together). That's usually what you want: a model that's occasionally very wrong is worse than one that's always slightly off.
Squares are smooth, which makes the maths of finding the minimum easy (gradient descent, later in this track).

The alternative mean absolute error (MAE), np.mean(np.abs(errors)), treats all errors proportionally and is easier to explain ("off by 4 units on average"). The regression metrics module compares them.

Training = searching for the lowest loss

With only two parameters, we can try a grid of values and keep the best, exactly like the threshold search in the first module:

best = None
for w in np.arange(0, 15.5, 0.5):
    for b in np.arange(30, 61, 1.0):
        loss = mse(predict(ad, w, b), units)
        if best is None or loss < best[0]:
            best = (loss, w, b)
print(f"loss {best[0]:.1f} at w={best[1]}, b={best[2]}")

loss 88.6 at w=7.5, b=45.0

The data says: about 7.5 extra units per million Rupiah of ads, and around 45 units with no ads at all. That's a learned business insight.

A grid search like this explodes as parameters grow (10 parameters × 30 values each = 30¹⁰ combinations). Real training uses smarter methods: an exact formula for linear regression (two modules from now), and gradient descent for nearly everything else, including neural networks.

The loss landscape

Picture the loss as a landscape: each point on the ground is a choice of (w, b), and the height is the loss. Training means finding the lowest valley. For linear regression with MSE, the landscape is a smooth bowl with exactly one bottom, which is why it's easy to train. Neural networks have bumpy landscapes with many valleys, which is why training them is an art.

for w in [4, 6, 7.5, 9, 11]:
    print(w, round(mse(predict(ad, w, 45), units), 1))

Moving w away from the best value in either direction raises the loss: the walls of the bowl.

Loss for classification

For yes/no predictions, models usually output a probability (0.8 = "80% likely to churn"), and the standard loss is log loss (cross-entropy): confident wrong answers are punished heavily.

def log_loss(p, y):
    p = np.clip(p, 1e-12, 1 - 1e-12)
    return -np.mean(y * np.log(p) + (1 - y) * np.log(1 - p))


actual = np.array([1, 0, 1])
print(round(log_loss(np.array([0.9, 0.1, 0.8]), actual), 3))   # confident and right
print(round(log_loss(np.array([0.6, 0.4, 0.5]), actual), 3))   # unsure
print(round(log_loss(np.array([0.1, 0.9, 0.2]), actual), 3))   # confident and wrong

0.145
0.572
2.072

You'll meet it again in logistic regression and in PyTorch, where it's called BCELoss / CrossEntropyLoss.

Why this matters for clean code

"Model + loss + search" is a clean decomposition you can apply to any ML problem: write the model as a function, the loss as a function, and the training as a loop or a library call that connects them. When results look wrong, you can check each part separately: are the predictions sensible? Is the loss computed correctly? Is the search converging?

Recap

A model is a function with parameters; training finds the parameters with the lowest loss.
Regression loss: MSE (mean of squared errors) punishes big errors; MAE (mean absolute error) is proportional and easy to explain.
Grid search works for a couple of parameters; real training uses formulas or gradient descent.
The loss landscape: for linear regression with MSE, a single smooth bowl.
Classification loss: log loss on predicted probabilities; confident mistakes cost the most.