🌳 Decision Tree Using Iris Dataset

🌳 Decision Tree – Step-by-Step Mathematics

Understanding Decision Tree Classification using Entropy and Information Gain

What is a Decision Tree?

A Decision Tree is a supervised machine learning algorithm used for classification and regression. It repeatedly asks questions about the input features and divides the dataset into smaller groups.

Example:
Is Study Hours ≤ 5?
↓
Yes → Low Performance
No → High Performance

1 Training Dataset

Consider the following student dataset.

Student	Study Hours	Attendance	Result
A	2	Low	Fail
B	3	Low	Fail
C	4	High	Pass
D	5	High	Pass
E	8	Low	Pass
9	9	High	Pass

2 Calculate Entropy

Entropy(S) = − Σ pᵢ log₂(pᵢ)

There are:

Pass = 4
Fail = 2
Total = 6

P(Pass) = 4/6 = 0.667
P(Fail) = 2/6 = 0.333

Entropy(S) = −(0.667 log₂0.667) −(0.333 log₂0.333)

≈ 0.918

3 Choose a Feature

The tree needs to decide which feature should become the root node.

Possible features:

✓ Study Hours
✓ Attendance

We calculate Information Gain for each feature.

4 Information Gain

Information Gain = Entropy(Parent) − Weighted Entropy(Children)

The feature producing the largest Information Gain is selected as the decision node.

5 Split Using Study Hours

Study Hours ≤ 5

Left group:

A → Fail
B → Fail
C → Pass
D → Pass

Right group:

E → Pass
F → Pass

6 Calculate Child Entropy

Left Child

Pass = 2
Fail = 2
Total = 4

P(Pass)=2/4=0.5
P(Fail)=2/4=0.5

Entropy = −(0.5 log₂0.5) −(0.5 log₂0.5)

= 1

Right Child

Pass = 2
Fail = 0

Entropy = −(1 log₂1) −(0 log₂0)

= 0

7 Weighted Entropy

Weighted Entropy = (4/6 × 1) + (2/6 × 0)

= 4/6

= 0.667

8 Information Gain

IG = Entropy(S) − Weighted Entropy

= 0.918 − 0.667

= 0.251

Information Gain for Study Hours ≈ 0.251

9 Compare Another Feature

The algorithm can also calculate Information Gain for Attendance.

IG(Study Hours) ≈ 0.251

IG(Attendance) = calculated similarly

The feature with the highest Information Gain becomes the decision node.

Rule:

Highest Information Gain ↓ Best Split ↓ Decision Node

10 Build the Decision Tree

Study Hours ≤ 5?

YES

Attendance?

LOW

FAIL

HIGH

PASS

NO

PASS

11 Predict a New Student

New Student:

Study Hours = 7
Attendance = Low

Step 1:
Study Hours = 7

7 > 5

Therefore follow the NO branch.

NO branch → PASS

🎯 Prediction = PASS

⭐ Interactive Step-by-Step Decision Tree

① Dataset

The algorithm receives student information:

Study Hours Attendance Result

② Entropy

Entropy measures impurity.

Entropy(S) = −Σ p log₂p

For our dataset: Entropy ≈ 0.918

③ Feature Selection

Possible features:

Study Hours
Attendance

The algorithm tests each feature.

④ Information Gain

IG = Parent Entropy − Child Entropy

Higher Information Gain means a better split.

⑤ Split

Study Hours ≤ 5?

YES → Continue left
NO → Continue right

⑥ Decision Tree

Study Hours ≤ 5?

YES

Continue Decision

NO

PASS

New Student

Study Hours = 7

7 > 5

🌳 Decision Tree → PASS

12. Complete Decision Tree Algorithm

Dataset ↓ Calculate Entropy ↓ Select Feature ↓ Calculate Information Gain ↓ Select Highest Gain ↓ Create Decision Node ↓ Split Dataset ↓ Repeat Process ↓ Leaf Node ↓ Prediction

13. Decision Tree vs ANN

Decision Tree	ANN
Tree-based model	Neural-network model
Uses nodes and branches	Uses neurons
Uses Entropy / Gini	Uses activation functions
Uses Information Gain	Uses weights and biases
Easy to interpret	Usually harder to interpret
Can work without feature scaling	Scaling is often useful

B.K. PAUL

Bhairab Ganguly College

Artificial Intelligence & Machine Learning Learning Module

© 2026 B.K. PAUL • Bhairab Ganguly College