Gradient Descent — M. Nagesh

Every time a neural network trains — every time ChatGPT got smarter — Gradient Descent was quietly doing all the work. Let's understand exactly how, with real numbers at every step.

Gradient Descent = A simple way to slide down the error curve step by step until we reach the lowest point — that lowest point is where the model has learned.

CONCEPT 01 What is a Loss Function?

Before gradient descent can do anything, it needs something to minimize. A loss function is that thing — it measures how wrong the model's prediction is. The bigger the loss, the worse the model.

🇬🇧 EnglishLoss = how far the model's prediction is from the actual value.

🇮🇳 TeluguLoss = model prediction అసలు value నుండి ఎంత దూరంలో ఉందో.

🇮🇳 HindiLoss = model की prediction असली value से कितनी दूर है।

House Price Example — Computing MSE Step by Step

1
Actual price: 60
2
Model prediction: 50
3
Error = Actual − Predicted:
60 − 50 = 10
4
Loss (Squared Error) = Error²:
(60 − 50)² = 10² = 100
5
Over many examples — Mean Squared Error (MSE):
MSE = (1/n) × Σ (actual − predicted)²

➕

Why square the error?

Squaring makes all errors positive — negative and positive errors won't cancel out.

📈

Why not use |Error|?

Squaring penalizes large errors much more — it forces the model to fix big mistakes first.

CONCEPT 02 What is Gradient Descent?

📐

Gradient

The slope or steepness of the loss function at any given point. It tells us which direction is uphill and by how much.

⬇️

Descent

Going downward — moving in the direction that reduces the loss, step by step, toward the minimum.

CONCEPT 03 The Intuition — Foggy Hill Analogy

⛰️ Foggy Hill Analogy

Imagine standing on a hill in thick fog. You cannot see the bottom of the valley — your goal. The only thing you can feel is the slope under your feet.

What do you do? Feel which direction the ground goes down and take a step that way. Steep slope → big step. Gentle slope → small step. Keep repeating until the ground is flat — you reached the bottom!

The Hill = Loss Function | The Valley = Minimum Loss | The Slope = Gradient | Each Step = Weight Update

CONCEPT 04 The Math — Derivative & Update Rule

What is a Derivative? — The Slope of a Function

💡

A derivative tells us how much a function's output changes when we change the input by a tiny amount.
📝

For function f(w), derivative is written as:
f'(w) = df(w) / dw

+
Positive f'(w) → function going uphill at this point
−
Negative f'(w) → function going downhill at this point
0
Zero f'(w) → function is flat — we are at a minimum ✓

Why subtract the gradient? — The Beauty of the Minus Sign

📌
We want to go DOWNHILL (reduce loss).
1

Gradient is positive (slope going up):
w_new = w_old − α × (+value)
→ moves LEFT → downhill ✓
2

Gradient is negative (slope going down):
w_new = w_old − α × (−value) = w_old + something
→ moves RIGHT → downhill ✓
✓
No matter which side of the minimum we are on, subtracting the gradient ALWAYS moves us toward the minimum. This is the beauty of the minus sign!

STEP-BY-STEP 05 Full Numeric Example — 5 Iterations

Setup — Our Loss Function

f
Loss function (minimum at w=3):
f(w) = (w − 3)²
(3−3)² = 0 → minimum is exactly at w = 3
f'
Derivative using chain rule:
f'(w) = 2 × (w − 3)
α
Learning rate: α = 0.1
w₀
Starting weight (model's initial guess): w = 0

ITERATION 1 — w starts at 0

1
Current w: 0
2
Gradient:
f'(0) = 2 × (0 − 3) = 2 × (−3) = −6
3
Update rule:
w_new = 0 − 0.1 × (−6) = 0 + 0.6 = 0.6
4
New loss:
f(0.6) = (0.6−3)² = (−2.4)² = 5.76

ITERATION 2 — w = 0.6

1
Current w: 0.6
2
Gradient:
f'(0.6) = 2 × (0.6 − 3) = 2 × (−2.4) = −4.8
3
Update rule:
w_new = 0.6 − 0.1 × (−4.8) = 0.6 + 0.48 = 1.08
4
New loss:
f(1.08) = (1.08−3)² = (−1.92)² = 3.686

ITERATION 3 — w = 1.08

1
Current w: 1.08
2
Gradient:
f'(1.08) = 2 × (1.08 − 3) = 2 × (−1.92) = −3.84
3
Update rule:
w_new = 1.08 − 0.1 × (−3.84) = 1.08 + 0.384 = 1.464
4
New loss:
f(1.464) = (1.464−3)² = (−1.536)² = 2.359

Step	w (before)	Gradient: 2(w−3)	α × Gradient	w_new	Loss f(w_new)
1	0	−6.000	−0.600	0.600	5.760
2	0.600	−4.800	−0.480	1.080	3.686
3	1.080	−3.840	−0.384	1.464	2.359
4	1.464	−3.072	−0.307	1.771	1.510
5	1.771	−2.458	−0.246	2.017	0.966 ↓

Key Observation: w is getting closer to 3 with every step. The gradient is also shrinking — meaning steps get smaller as we approach the minimum. This is how gradient descent converges.

CONCEPT 06 Multiple Parameters & Partial Derivatives

📺 TV Knobs Analogy

Imagine a TV with two knobs — volume and brightness. To understand each knob's effect, you turn one at a time while keeping the other fixed.

A partial derivative does exactly this — it measures how loss changes with respect to one weight while all others stay fixed.

Two-Parameter Update — L(w1, w2)

1
Partial derivative w.r.t. w1 (w2 held fixed):
∂L / ∂w1 → how much does loss change if we nudge w1?
2
Partial derivative w.r.t. w2 (w1 held fixed):
∂L / ∂w2
3
The gradient = collection of all partial derivatives:
∇L = [ ∂L/∂w1, ∂L/∂w2 ]
4
Update rule for each weight independently:
w1_new = w1_old − α × ∂L/∂w1
w2_new = w2_old − α × ∂L/∂w2
✓
This scales to millions or billions of parameters — each weight gets its own independent gradient update every step.

CONCEPT 07 The Role of Learning Rate (α)

The learning rate controls step size. Too small = painfully slow. Too large = overshoots and diverges. Let's prove this with actual numbers.

⏳ α = 0.01 — Too Small

Step 1: grad=−6
w = 0+0.01×6 = 0.06
Step 2: grad=−5.88
w = 0.06+0.0588 = 0.119

After 20 steps: w ≈ 0.997
Still far from 3 ❌

Extremely slow convergence.

✅ α = 0.1 — Just Right

Step 1: w → 0.600
Step 2: w → 1.080
Step 3: w → 1.464
Step 4: w → 1.771
Step 5: w → 2.017

Step 50: w = 3.000 ✓

Smooth, steady convergence.

💥 α = 1.5 — Too Large

Step 1: grad=−6
w = 0+1.5×6 = 9 ❌ overshot!
Step 2: grad=2(9−3)=12
w = 9−1.5×12 = −9 ❌

0 → 9 → −9 → ...

Diverging — never converges.

Learning Rate Quick Reference

Too Small (0.001)	Converges but extremely slowly.	SLOW
Just Right (0.01–0.1)	Smooth convergence to the minimum.	IDEAL ✓
Too Large (1.5+)	Overshoots minimum, bounces wildly, diverges.	DIVERGES

In practice: start with 0.001 or 0.01. For large Transformer models: 1e-4 or smaller.

CONCEPT 08 Types of Gradient Descent

Type	Data Per Step	Speed	Gradient Quality	Best For
Batch GD	Entire dataset	Slow	High (exact)	Small datasets
SGD	1 random point	Fast	Noisy	Online learning
Mini-Batch GD ⭐	Batch of 32–1024	Moderate	Balanced	Most real-world models

In Practice: Mini-Batch GD is used for almost all deep learning today. PyTorch's torch.optim.SGD is actually mini-batch SGD.

CODE 09 Gradient Descent in Python

Let's run the exact same math in Python — f(w) = (w−3)², starting from w=0, learning rate=0.1.

PYTHON

# Gradient Descent on f(w) = (w - 3)²

# Minimum is at w = 3  →  loss = 0

w = 0.0                     # starting weight

learning_rate = 0.1         # alpha (α)

for step in range(50):

    # Step 1: compute gradient  →  f'(w) = 2(w - 3)

    gradient = 2 * (w - 3)

    # Step 2: update rule  →  w = w - α × gradient

    w = w - learning_rate * gradient

    # Step 3: compute new loss  →  f(w) = (w - 3)²

    loss = (w - 3) ** 2

    print(f"Step {step+1:>2}:  w = {w:.4f}   loss = {loss:.6f}")

Output — Selected Steps

1
Step 1: w = 0.6000 loss = 5.760000
5
Step 5: w = 2.0170 loss = 0.966289
10
Step 10: w = 2.6508 loss = 0.121577
25
Step 25: w = 2.9722 loss = 0.000773
50
Step 50: w = 3.0000 loss = 0.000000 ✓ MINIMUM FOUND

After 50 steps, w = 3 exactly and loss = 0. In real deep learning frameworks like PyTorch and TensorFlow, gradients are computed automatically via backpropagation — the same update rule applies.

FINAL Putting It All Together

🎯 Key Takeaways

The loss function measures how wrong the model is — our goal is to minimize it.
Gradient Descent adjusts weights step by step in the direction that reduces loss.
The derivative (gradient) gives us the slope — which direction is uphill and by how much.
The update rule w_new = w_old − α × f'(w) always moves us toward the minimum.
The minus sign is crucial — it ensures we always go downhill regardless of which side we are on.
Learning rate α controls step size — too small is slow, too large diverges.
Mini-batch gradient descent is the standard in practice — balanced speed and accuracy.
In real models, gradients are computed automatically via backpropagation — but the same math applies.

Gradient Descent:The Math Behind How AI Learns