📡 Entropy of an Information Source

📡 Entropy of an Information Source

Information Theory • Mathematical Explanation • Numerical Examples

📖 1. What is Entropy?

In Information Theory, entropy measures the average amount of uncertainty or information produced by an information source.

If the outcome of a source is highly uncertain, its entropy is high. If the outcome is almost certain, its entropy is low.

Simple idea:

🎲 More uncertainty → More information → Higher entropy

🎯 Less uncertainty → Less information → Lower entropy

📐 2. Information Content of an Event

Suppose an event x occurs with probability P(x). The information contained in that event is:

I(x) = −log₂ P(x)

The unit is called a bit when the logarithm is base 2.

Example

Suppose an event has probability:

P(x) = 1/2

Then:

I(x) = −log₂(1/2)
= −(−1)
= 1 bit

Therefore, an event having probability 1/2 carries 1 bit of information.

🧮 3. Mathematical Formula of Entropy

Suppose an information source can produce n different symbols:

X = {x₁, x₂, x₃, ..., xₙ}

with probabilities:

P = {p₁, p₂, p₃, ..., pₙ}

where:

p₁ + p₂ + ... + pₙ = 1

The entropy of the source is:

H(X) = − Σ pᵢ log₂(pᵢ)

or explicitly:

H(X) = −[p₁log₂p₁ + p₂log₂p₂ + ... + pₙlog₂pₙ]

📊 4. Why Do We Take an Average?

The information associated with an individual symbol is:

I(xᵢ) = −log₂(pᵢ)

The average information is obtained by multiplying each information value by its probability:

H(X) = Σ pᵢ I(xᵢ)

Substituting:

H(X) = Σ pᵢ[−log₂(pᵢ)]

Therefore:

H(X) = −Σ pᵢlog₂(pᵢ)

🎯 Example 1 — Fair Coin

Consider a fair coin with two possible outcomes:

Symbol	Probability
Head (H)	0.5
Tail (T)	0.5

Using:

H(X) = −Σpᵢlog₂(pᵢ)

Therefore:

H(X) = −[0.5log₂(0.5) + 0.5log₂(0.5)]

Since:

log₂(0.5) = −1

Therefore:

H(X) = −[0.5(−1) + 0.5(−1)]
= −[−0.5 − 0.5]
= 1 bit

Answer: Entropy of a fair coin = 1 bit.

🎲 Example 2 — Biased Coin

Suppose:

Outcome	Probability
Head	0.8
Tail	0.2

Entropy:

H(X) = −[0.8log₂(0.8) + 0.2log₂(0.2)]

Using approximate values:

log₂(0.8) ≈ −0.3219
log₂(0.2) ≈ −2.3219

Therefore:

H(X) = −[0.8(−0.3219) + 0.2(−2.3219)]

≈ 0.722 bits

Answer: Entropy ≈ 0.722 bits.

Notice that this is less than 1 bit because the coin is biased, so there is less uncertainty.

📦 Example 3 — Three Symbol Source

Consider a source:

X = {A, B, C}

with probabilities:

Symbol	Probability
A	0.5
B	0.3
C	0.2

Check:

0.5 + 0.3 + 0.2 = 1

Entropy:

H(X) = −[0.5log₂0.5 + 0.3log₂0.3 + 0.2log₂0.2]

Using approximate logarithms:

H(X) ≈ 1.485 bits

🏆 5. Maximum Entropy

For a source having n equally probable symbols:

p₁ = p₂ = ... = pₙ = 1/n

The entropy becomes:

H(X) = −n(1/n)log₂(1/n)

Therefore:

H_max = log₂(n)

Example

For 4 equally probable symbols:

H_max = log₂(4) = 2 bits

For 8 equally probable symbols:

H_max = log₂(8) = 3 bits

⬇️ 6. Minimum Entropy

If one symbol has probability 1 and all other symbols have probability 0:

P(X) = {1,0,0,...,0}

Then:

H(X) = 0 bits

There is no uncertainty because the outcome is completely predictable.

⭐ 7. Important Properties of Entropy

1️⃣ Non-Negative

H(X) ≥ 0

Entropy cannot be negative.

2️⃣ Certain Event

P(X)=1 ⇒ H(X)=0

A completely predictable source has zero entropy.

3️⃣ Maximum

H_max=log₂n

Maximum entropy occurs when all symbols are equally probable.

4️⃣ Unit

When log₂ is used, entropy is measured in bits/symbol.

🎬 Animated Entropy Calculation

The following animation demonstrates how the entropy of a source is calculated from probability → information → weighted information → total entropy.

📡 Probability → Information → Entropy

P(x) Probability

I(x) Information

pI(x) Weighted

H(X) Entropy

Press ▶ Start Animation

H(X)

🧮 General Numerical Example

Suppose an information source produces four symbols:

Symbol	Probability	Information
A	0.4	−log₂(0.4)
B	0.3	−log₂(0.3)
C	0.2	−log₂(0.2)
D	0.1	−log₂(0.1)

Therefore:

H(X) = −[0.4log₂(0.4) +0.3log₂(0.3) +0.2log₂(0.2) +0.1log₂(0.1)]

Approximate calculation gives:

H(X) ≈ 1.846 bits/symbol

🌍 8. Real-Life Applications

📡 Communication

Entropy helps measure the information generated by a communication source.

🗜️ Data Compression

Entropy provides a theoretical limit for lossless data compression.

💻 Computer Networks

It can be used to analyze information transmitted through communication channels.

🤖 Machine Learning

Entropy is used in decision trees to measure impurity and choose useful splits.

📌 Important Formulas for Examination

Information:
I(x) = −log₂P(x)

Entropy:
H(X) = −Σpᵢlog₂pᵢ

Maximum Entropy:
H_max = log₂n

Minimum Entropy:
H_min = 0

🎯 Quick Revision

• 📡 Entropy measures the average uncertainty of an information source.
• 🧮 Information of an event = −log₂P(x).
• 📊 Entropy = −Σpᵢlog₂pᵢ.
• 🎲 A fair binary source has entropy 1 bit.
• 📉 A biased source has lower entropy than an equally probable source with the same number of symbols.
• 🏆 Maximum entropy for n symbols is log₂n.
• ⬇️ Minimum entropy is 0.
• 💾 Entropy is fundamental in information theory and data compression.

🌟 Created by Bijan Krishna Paul 🌟
B.K.PAUL Bhairab Ganguly College