Autograd: how gradients work

The idea

Gradient descent (ML track) needs the gradient of the loss with respect to every parameter. For linear regression you derived it by hand. For a network with a million parameters, nobody can. Autograd does it automatically: PyTorch records every operation on tensors that "require gradients", and when you call .backward() on the loss, it walks that record backwards applying the chain rule, filling in .grad for every parameter.

That's the whole trick behind training neural networks.

A first gradient

In the ML track you worked out gradients by hand, with calculus. Autograd does it for you. Here's the simplest possible case: one weight, w, and a loss that's smallest when w is 5. Ask PyTorch which way to move w:

import torch

w = torch.tensor(3.0, requires_grad=True)
loss = (w - 5) ** 2          # minimum at w = 5
loss.backward()
print(loss.item(), w.grad)

4.0 tensor(-4.)

The derivative of (w − 5)² is 2(w − 5), which at w = 3 is −4. PyTorch computed exactly that. The negative sign says "increase w to lower the loss", which gradient descent will do.

The recorded graph

Every result remembers the operation that created it:

x = torch.tensor([1.0, 2.0, 3.0])
y = torch.tensor([3.0, 5.0, 7.0])
w = torch.tensor(0.0, requires_grad=True)
b = torch.tensor(0.0, requires_grad=True)

pred = w * x + b
loss = ((pred - y) ** 2).mean()
print(type(pred.grad_fn).__name__, type(loss.grad_fn).__name__)
loss.backward()
print(w.grad, b.grad)

AddBackward0 MeanBackward0
tensor(-22.6667) tensor(-10.)

These are exactly the MSE gradients you derived in the gradient-descent module: mean(2·error·x) and mean(2·error). The grad_fn entries are the recorded steps (an addition, a mean) that backward() walks through.

A training loop with autograd

Replace the hand-written gradients from the ML track with autograd:

w = torch.tensor(0.0, requires_grad=True)
b = torch.tensor(0.0, requires_grad=True)
learning_rate = 0.05

for step in range(500):
    loss = ((w * x + b - y) ** 2).mean()
    loss.backward()                       # compute w.grad and b.grad
    with torch.no_grad():                 # update without recording these operations
        w -= learning_rate * w.grad
        b -= learning_rate * b.grad
    w.grad.zero_()                        # reset for the next step
    b.grad.zero_()

print(round(w.item(), 3), round(b.item(), 3), round(loss.item(), 6))

2.0 1.0 0.0

It finds w = 2, b = 1 (the data is y = 2x + 1). Three details matter:

1. torch.no_grad() around the update: the update itself shouldn't become part of the recorded graph.
2. Zeroing gradients: PyTorch adds new gradients to .grad instead of replacing them. Without zero_(), each step would use the sum of all previous gradients, and training goes haywire.
3. The loop is predict → loss → backward() → update → zero. In the next modules, optimizer.step() and optimizer.zero_grad() do steps 4 and 5 for you.

Gradients accumulate: see it

That third detail, zeroing the gradients, is the one everyone forgets once. Here's what happens if you call backward() repeatedly without clearing w.grad in between:

w = torch.tensor(3.0, requires_grad=True)
for _ in range(3):
    loss = (w - 5) ** 2
    loss.backward()
    print(w.grad)

tensor(-4.)
tensor(-8.)
tensor(-12.)

Same loss, called three times: −4, then −8, then −12. That's accumulation. It's occasionally useful (to simulate bigger batches), but almost always it's a bug when you forget zero_grad().

Turning tracking off

Recording costs memory and time. During evaluation and prediction you don't need gradients:

with torch.no_grad():
    predictions = w * x + b
print(predictions.requires_grad)
print((w * x).detach().requires_grad)

False
False

torch.no_grad() turns recording off for a block; .detach() returns a tensor cut off from the graph (needed before .numpy(), as the previous module showed).

Checking gradients numerically

How do you know a gradient is right? Compare it with a finite difference: nudge the parameter a tiny bit and see how much the loss changes. This works in plain NumPy:

import numpy as np

x = np.array([1.0, 2.0, 3.0])
y = np.array([3.0, 5.0, 7.0])


def loss(w, b):
    return np.mean((w * x + b - y) ** 2)


w, b, eps = 0.5, 0.2, 1e-6
numeric_w = (loss(w + eps, b) - loss(w - eps, b)) / (2 * eps)
analytic_w = np.mean(2 * (w * x + b - y) * x)
print(round(numeric_w, 4), round(analytic_w, 4))

-17.2 -17.2

They agree. Library developers test autograd exactly this way (torch.autograd.gradcheck), and it's a great way to convince yourself that "the gradient is the slope" isn't just a slogan.

Why this matters for clean code

Autograd removes the most error-prone part of ML code: hand-derived gradients. In return, it asks for discipline: zero gradients every step, wrap updates and evaluation in no_grad, and detach before leaving PyTorch. These three habits prevent the majority of "my training does something weird" bugs.

Recap

• requires_grad=True makes PyTorch record operations on a tensor.
• loss.backward() computes gradients via the chain rule into each parameter's .grad.
• Gradients accumulate: zero them every step (.grad.zero_() or optimizer.zero_grad()).
• Update parameters inside torch.no_grad(); use it (or .detach()) for evaluation and predictions.
• Finite differences check gradients numerically: (f(w+ε) − f(w−ε)) / 2ε.