🧠 ANN Backpropagation: Weight & Bias Update A simple numerical example showing how an ANN calculates the loss, sends the error backward, and updates its weights and biases.

🧠 ANN Backpropagation: Weight & Bias Update

A simple numerical example showing how an ANN calculates the loss, sends the error backward, and updates its weights and biases.

1. What Are We Going to Learn?

During training, an ANN initially has randomly selected weights and biases. The network makes a prediction. That prediction is compared with the correct answer.

Input ↓ Forward Propagation ↓ Prediction ↓ Loss ↓ Backpropagation ↓ Gradients ↓ Update Weight & Bias ↓ Better Prediction

The goal is to make the loss smaller.

2. Small ANN Used in This Example

Input X ↓ Hidden Neuron H ↓ Output Neuron Y

1 Input → 1 Hidden → 1 Output

We deliberately use one neuron in each layer so that every calculation can be understood easily.

3. Training Data

Input X	Correct Output Y
1	1

The ANN receives:

Input: x = 1

Actual / Target: y = 1

4. Initial Weight and Bias

Suppose the ANN starts with:

Parameter	Initial Value
Input → Hidden Weight	0.5
Hidden Bias	0.1
Hidden → Output Weight	0.4
Output Bias	0.1
Learning Rate η	0.1

Learning Rate: η = 0.1

1 Forward Propagation — Hidden Layer

First calculate the weighted sum of the hidden neuron.

zₕ = x × w₁ + bₕ

Substitute the values:

zₕ = (1 × 0.5) + 0.1

zₕ = 0.6

Now apply sigmoid activation.

Sigmoid(z) = 1 / (1 + e⁻ᶻ)

h = 1 / (1 + e⁻⁰·⁶)

h ≈ 0.646

2 Forward Propagation — Output Layer

The hidden neuron output becomes the input to the output neuron.

zₒ = h × w₂ + bₒ

Substitute:

zₒ = (0.646 × 0.4) + 0.1

zₒ = 0.2584 + 0.1

zₒ = 0.3584

Apply sigmoid:

ŷ = 1 / (1 + e⁻⁰·³⁵⁸⁴)

ŷ ≈ 0.589

Predicted Output ≈ 0.589

3 Calculate the Loss

For this demonstration, we use Mean Squared Error for one example:

Loss = ½(y - ŷ)²

Actual value:

y = 1
ŷ = 0.589

Therefore:

Loss = ½(1 - 0.589)²

= ½(0.411)²

= ½(0.1689)

Loss ≈ 0.08445

4 Backpropagation — Output Error

Now the ANN asks:

"How much did the output weight and output bias contribute to the error?"

For sigmoid:

σ'(z) = σ(z)(1 - σ(z))

Therefore:

σ'(zₒ) = 0.589(1 - 0.589)

= 0.589 × 0.411

≈ 0.242

The derivative of the loss with respect to prediction is:

∂L/∂ŷ = ŷ - y

= 0.589 - 1

= -0.411

Therefore the output error signal is:

δₒ = (ŷ - y) × σ'(zₒ)

= (-0.411)(0.242)

δₒ ≈ -0.09946

5 Calculate Gradient of Output Weight

The output weight connects hidden neuron h to output neuron.

∂L/∂w₂ = δₒ × h

Substitute:

∂L/∂w₂ = (-0.09946)(0.646)

≈ -0.06425

6 Calculate Gradient of Output Bias

∂L/∂bₒ = δₒ

∂L/∂bₒ ≈ -0.09946

The bias gradient is simply the output error signal.

7 Update Output Weight

The basic gradient-descent update rule is:

New Weight = Old Weight - Learning Rate × Gradient

For the output weight:

w₂(new) = 0.4 - (0.1 × -0.06425)

= 0.4 + 0.006425

w₂(new) ≈ 0.40643

8 Update Output Bias

bₒ(new) = bₒ - η(∂L/∂bₒ)

= 0.1 - (0.1 × -0.09946)

= 0.1 + 0.009946

bₒ(new) ≈ 0.10995

9 Send Error Back to Hidden Neuron

The hidden neuron also needs to know how much it contributed to the final error.

δₕ = δₒ × w₂ × σ'(zₕ)

We already know:

δₒ = -0.09946
w₂ = 0.4
h = 0.646

Calculate the hidden sigmoid derivative:

σ'(zₕ) = h(1-h)

= 0.646(1-0.646)

= 0.646 × 0.354

≈ 0.2287

Therefore:

δₕ = (-0.09946)(0.4)(0.2287)

δₕ ≈ -0.00910

10 Calculate Gradient of Hidden Weight

∂L/∂w₁ = δₕ × x

Since:

x = 1
δₕ ≈ -0.00910

Therefore:

∂L/∂w₁ = (-0.00910)(1)

≈ -0.00910

11 Calculate Hidden Bias Gradient

∂L/∂bₕ = δₕ

∂L/∂bₕ ≈ -0.00910

12 Update Hidden Weight

w₁(new) = w₁ - η(∂L/∂w₁)

= 0.5 - (0.1 × -0.00910)

= 0.5 + 0.00091

w₁(new) ≈ 0.50091

13 Update Hidden Bias

bₕ(new) = bₕ - η(∂L/∂bₕ)

= 0.1 - (0.1 × -0.00910)

= 0.1 + 0.00091

bₕ(new) ≈ 0.10091

14. Before vs After Weight & Bias

Parameter	Before	Gradient	After
w₁	0.50000	-0.00910	0.50091
bₕ	0.10000	-0.00910	0.10091
w₂	0.40000	-0.06425	0.40643
bₒ	0.10000	-0.09946	0.10995

15. Why Did the Weights Increase?

Notice that the gradients are negative.

New Weight = Old Weight - Learning Rate × Negative Gradient

Therefore: New Weight > Old Weight

The network predicted:

Prediction ≈ 0.589
Target = 1

The prediction was too low. Gradient descent therefore moves the parameters in a direction that should increase the prediction and reduce the loss.

16. Perform Another Forward Pass

Now use the updated parameters.

w₁ = 0.50091
bₕ = 0.10091
w₂ = 0.40643
bₒ = 0.10995

Hidden Layer:

zₕ = (1 × 0.50091) + 0.10091

zₕ ≈ 0.60182

h ≈ 0.6462

Output Layer:

zₒ = (0.6462 × 0.40643) + 0.10995

zₒ ≈ 0.3726

ŷ ≈ 0.5921

New Prediction ≈ 0.592

Old Prediction ≈ 0.589

17. New Loss

New Loss = ½(1 - 0.5921)²

= ½(0.4079)²

New Loss ≈ 0.08318

Old Loss ≈ 0.08445

New Loss ≈ 0.08318

✅ LOSS DECREASED

18. Complete Training Cycle

INPUT x = 1 ↓ FORWARD PROPAGATION ↓ Prediction = 0.589 ↓ CALCULATE LOSS Loss = 0.08445 ↓ BACKPROPAGATION ↓ Calculate Gradients ↓ UPDATE WEIGHTS w(new) = w(old) - η × gradient ↓ UPDATE BIASES b(new) = b(old) - η × gradient ↓ FORWARD PROPAGATION AGAIN ↓ Prediction ≈ 0.592 ↓ NEW LOSS ≈ 0.08318

19. Most Important Formulas

Weighted Sum

z = wx + b

Sigmoid

σ(z) = 1 / (1 + e⁻ᶻ)

Sigmoid Derivative

σ'(z) = σ(z)(1 - σ(z))

Mean Squared Error

L = ½(y - ŷ)²

Gradient Descent

w(new) = w(old) - η × ∂L/∂w

b(new) = b(old) - η × ∂L/∂b

20. Interactive Step-by-Step

① Forward Pass

zₕ = wx + b
h = Sigmoid(zₕ)

zₒ = hw₂ + bₒ
ŷ = Sigmoid(zₒ)

② Prediction

ŷ ≈ 0.589

③ Loss

L = ½(y - ŷ)²

L ≈ 0.08445

④ Backpropagation

Error flows backward:

Output Error ↓ Output Weight Gradient ↓ Hidden Error ↓ Hidden Weight Gradient

⑤ Update Weights

w(new) = w(old) - η × gradient

⑥ Update Bias

b(new) = b(old) - η × gradient

⑦ New Loss

Old Loss ≈ 0.08445
New Loss ≈ 0.08318

🎯 The ANN learned because the loss decreased.

21. Important Terms

Weight Bias Loss Gradient Gradient Descent Learning Rate Backpropagation Forward Propagation Sigmoid Derivative Error Target Prediction