📊 Feature Selection Laboratory

Complete Mathematical & Interactive Implementation

5 Features × 10 Records Dataset

1. Dataset Used for All Methods

We will use exactly 5 input features and 10 records. The target variable is ExamScore.

Student	StudyHours	Attendance	SleepHours	Assignments	ScreenTime	ExamScore
1	2	60	5	3	8	45
2	3	65	6	4	7	50
3	4	70	6	5	6	55
4	5	75	7	6	5	62
5	6	80	7	7	5	68
6	7	82	8	8	4	74
7	8	85	8	8	3	80
8	9	90	9	9	3	86
9	10	92	9	10	2	92
10	10	95	10	10	1	96

Features: StudyHours, Attendance, SleepHours, Assignments, ScreenTime
Target: ExamScore
Records: 10
Features: 5

2. Mathematical Representation

The dataset can be represented by a feature matrix X and target vector y.

X = [ x₁₁ x₁₂ x₁₃ x₁₄ x₁₅
x₂₁ x₂₂ x₂₃ x₂₄ x₂₅
⋮
x₁₀₁ x₁₀₂ x₁₀₃ x₁₀₄ x₁₀₅ ]

Therefore:

X ∈ R^{10 × 5}

and:

y ∈ R¹⁰

The objective is to determine whether all five features are useful or whether a smaller subset can provide similar predictive information.

3. Feature Selection Categories

🔵 Filter Methods

Correlation, Variance Threshold, Chi-Square, ANOVA and Mutual Information.

🟣 Wrapper Methods

Forward Selection, Backward Elimination and Recursive Feature Elimination.

🟢 Embedded Methods

L1/Lasso, Decision Tree and Random Forest importance.

🟠 Feature Extraction

PCA creates new components rather than selecting original columns.

4. Variance Threshold

Variance measures how much a feature changes across observations. A feature with extremely small variance may contain little information.

Variance(X) = Σ(Xᵢ − X̄)² / n

Example: StudyHours

StudyHours = 2, 3, 4, 5, 6, 7, 8, 9, 10, 10

Mean:

X̄ = 64 / 10 = 6.4

Population variance:

σ² = [(2−6.4)² + (3−6.4)² + ... + (10−6.4)²] / 10

The feature has substantial variation, so it would not be removed by a reasonable variance threshold.

Python

from sklearn.feature_selection import VarianceThreshold

selector = VarianceThreshold(threshold=1)

X_new = selector.fit_transform(X) 

5. Correlation-Based Feature Selection

Correlation measures the linear relationship between a feature and the target.

r = Σ(Xᵢ − X̄)(Yᵢ − Ȳ) / √[Σ(Xᵢ − X̄)² × Σ(Yᵢ − Ȳ)²]

Interpretation:

|r|	Interpretation
0	No linear relationship
0 to 0.3	Weak
0.3 to 0.7	Moderate
0.7 to 1	Strong
1	Perfect linear relationship

Python

import pandas as pd

correlation = df.corr(numeric_only=True)

print(correlation["ExamScore"].sort_values(
ascending=False
)) 

A feature having a high correlation with the target does not automatically mean that it should always be selected. Correlation mainly measures linear association.

6. Chi-Square Feature Selection

The Chi-Square test measures statistical dependence between categorical variables. In machine learning it is commonly used for classification problems.

χ² = Σ (O − E)² / E

where:

• O = observed frequency
• E = expected frequency

The expected frequency is:

E = (Row Total × Column Total) / Grand Total

Python

from sklearn.feature_selection import SelectKBest
from sklearn.feature_selection import chi2

selector = SelectKBest(
score_func=chi2,
k=3
)

X_selected = selector.fit_transform(X, y) 

Chi-square requires suitable non-negative feature values and is mainly appropriate when the target is categorical.

7. ANOVA F-Test

ANOVA compares variation between groups with variation within groups. For classification, it can evaluate numerical features against a categorical target.

F = MS_between / MS_within

where:

MS_between = SS_between / (k−1)

MS_within = SS_within / (N−k)

Python

from sklearn.feature_selection import SelectKBest
from sklearn.feature_selection import f_classif

selector = SelectKBest(
score_func=f_classif,
k=3
)

X_selected = selector.fit_transform(X, y) 

8. Mutual Information

Mutual Information measures how much knowing one variable reduces uncertainty about another.

I(X;Y) = Σₓ Σᵧ p(x,y) log [ p(x,y) / (p(x)p(y)) ]

Higher mutual information indicates greater statistical dependency.

Python

from sklearn.feature_selection import mutual_info_regression

scores = mutual_info_regression(X, y)

for feature, score in zip(
feature_names, scores
):
print(feature, score) 

9. Forward Selection

Forward selection begins with zero features and adds one feature at a time. At each step, the feature that produces the best validation performance is considered.

S₀ = ∅

Then:

S₁ = S₀ ∪ {best feature}

S₂ = S₁ ∪ {next best feature}

The process continues until the required number of features is reached.

Conceptual Python

selected = []

remaining = feature_names.copy()

while remaining:

```
best_feature = None
best_score = -float("inf")

for feature in remaining:

    current = selected + [feature]

    # train model using current
    # calculate validation score

    # if score > best_score:
    #     best_score = score
    #     best_feature = feature

selected.append(best_feature)
remaining.remove(best_feature)
```

10. Backward Elimination

Backward elimination starts with all five features and removes one feature at a time.

S₀ = {StudyHours, Attendance, SleepHours, Assignments, ScreenTime}

At every iteration the least useful feature is removed.

S₁ = S₀ − {least useful feature}

S₂ = S₁ − {least useful feature}

Conceptual Python

selected = feature_names.copy()

while len(selected) > 2:

```
# evaluate removing each feature

# remove feature whose removal
# causes the smallest performance loss

pass
```

11. Recursive Feature Elimination — RFE

RFE repeatedly fits a model and removes the least important feature.

5 features → 4 → 3 → 2 → selected subset

Python

from sklearn.feature_selection import RFE
from sklearn.linear_model import LinearRegression

model = LinearRegression()

rfe = RFE(
estimator=model,
n_features_to_select=2
)

rfe.fit(X, y)

print(rfe.support_)
print(rfe.ranking_) 

Feature	Selected?	Rank
StudyHours	✓ / ✗	1–5
Attendance	✓ / ✗	1–5
SleepHours	✓ / ✗	1–5
Assignments	✓ / ✗	1–5
ScreenTime	✓ / ✗	1–5

12. L1 Regularization / Lasso

L1 regularization adds the absolute value of coefficients to the loss function.

Loss = Σ(yᵢ − ŷᵢ)² + λΣ|βⱼ|

If a coefficient becomes zero:

βⱼ = 0

the corresponding feature can effectively be excluded from the linear model.

Python

from sklearn.linear_model import Lasso

model = Lasso(alpha=0.1)

model.fit(X, y)

for feature, coefficient in zip(
feature_names,
model.coef_
):
print(feature, coefficient) 

L1 regularization is an embedded method because feature selection occurs during model training.

13. Decision Tree Feature Importance

Decision trees can estimate feature importance from the reduction in impurity produced by splits.

Importance(j) = Σ (weight of node) × impurity decrease

The normalized feature importances usually sum to 1.

from sklearn.tree import DecisionTreeRegressor

model = DecisionTreeRegressor(
random_state=42
)

model.fit(X, y)

for feature, importance in zip(
feature_names,
model.feature_importances_
):
print(feature, importance) 

14. Random Forest Feature Importance

Random Forest combines many decision trees. Feature importance can be obtained from the fitted forest.

Importance(feature) = Average importance across trees

Python

from sklearn.ensemble import RandomForestRegressor

model = RandomForestRegressor(
n_estimators=100,
random_state=42
)

model.fit(X, y)

importance = model.feature_importances_

for f, score in zip(
feature_names,
importance
):
print(f, score) 

15. PCA — Feature Extraction

PCA is included here for comparison, but technically it is a feature-extraction/dimensionality-reduction technique rather than selection of original columns.

Step 1 — Standardization

z = (x − μ) / σ

Step 2 — Covariance Matrix

C = 1/(n−1) XᵀX

Step 3 — Eigenvalue Problem

Cv = λv

The eigenvectors corresponding to the largest eigenvalues become the principal components.

PC₁ = w₁X₁ + w₂X₂ + ... + w₅X₅

Explained Variance Ratio

EVRᵢ = λᵢ / Σλⱼ

Python

from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA

X_scaled = StandardScaler().fit_transform(X)

pca = PCA(n_components=2)

X_pca = pca.fit_transform(X_scaled)

print(pca.explained_variance_ratio_)
print(X_pca) 

16. Interactive Feature Selection Simulator

The following JavaScript calculates several basic statistics directly from the 10-record dataset.

Click a button to calculate the results.

17. Complete Python Implementation

import pandas as pd
from sklearn.feature_selection import (
    SelectKBest,
    f_classif,
    mutual_info_regression,
    RFE,
    VarianceThreshold
)
from sklearn.linear_model import Lasso, LinearRegression
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import RandomForestRegressor
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA

# ------------------------------------------------

# DATASET

# ------------------------------------------------

df = pd.DataFrame({

```
"StudyHours":
[2,3,4,5,6,7,8,9,10,10],

"Attendance":
[60,65,70,75,80,82,85,90,92,95],

"SleepHours":
[5,6,6,7,7,8,8,9,9,10],

"Assignments":
[3,4,5,6,7,8,8,9,10,10],

"ScreenTime":
[8,7,6,5,5,4,3,3,2,1],

"ExamScore":
[45,50,55,62,68,74,80,86,92,96]
```

})

features = [
"StudyHours",
"Attendance",
"SleepHours",
"Assignments",
"ScreenTime"
]

X = df[features]
y = df["ExamScore"]

# ------------------------------------------------

# CORRELATION

# ------------------------------------------------

print("\nCORRELATION")
print(df.corr(numeric_only=True)["ExamScore"])

# ------------------------------------------------

# VARIANCE THRESHOLD

# ------------------------------------------------

print("\nVARIANCE")

vt = VarianceThreshold(
threshold=1
)

vt.fit(X)

for feature, selected in zip(
features,
vt.get_support()
):
print(feature, selected)

# ------------------------------------------------

# MUTUAL INFORMATION

# ------------------------------------------------

print("\nMUTUAL INFORMATION")

mi = mutual_info_regression(
X,
y,
random_state=42
)

for feature, score in zip(
features,
mi
):
print(feature, score)

# ------------------------------------------------

# RFE

# ------------------------------------------------

print("\nRFE")

rfe = RFE(
LinearRegression(),
n_features_to_select=2
)

rfe.fit(X, y)

for feature, selected, rank in zip(
features,
rfe.support_,
rfe.ranking_
):
print(
feature,
selected,
rank
)

# ------------------------------------------------

# LASSO

# ------------------------------------------------

print("\nLASSO")

lasso = Lasso(alpha=0.1)

lasso.fit(X, y)

for feature, coefficient in zip(
features,
lasso.coef_
):
print(
feature,
coefficient
)

# ------------------------------------------------

# DECISION TREE

# ------------------------------------------------

print("\nDECISION TREE")

tree = DecisionTreeRegressor(
random_state=42
)

tree.fit(X, y)

for feature, importance in zip(
features,
tree.feature_importances_
):
print(
feature,
importance
)

# ------------------------------------------------

# RANDOM FOREST

# ------------------------------------------------

print("\nRANDOM FOREST")

forest = RandomForestRegressor(
n_estimators=100,
random_state=42
)

forest.fit(X, y)

for feature, importance in zip(
features,
forest.feature_importances_
):
print(
feature,
importance
)

# ------------------------------------------------

# PCA

# ------------------------------------------------

print("\nPCA")

scaler = StandardScaler()

X_scaled = scaler.fit_transform(X)

pca = PCA(
n_components=2
)

X_pca = pca.fit_transform(
X_scaled
)

print(
"Explained variance:",
pca.explained_variance_ratio_
)

print(
"PCA data:"
)

print(X_pca) 

18. Method Comparison

Method	Category	Uses Target?	Original Features?	Main Mathematics
Variance Threshold	Filter	No	Yes	Variance
Correlation	Filter	Yes	Yes	Pearson r
Chi-Square	Filter	Yes	Yes	χ²
ANOVA	Filter	Yes	Yes	F-statistic
Mutual Information	Filter	Yes	Yes	Information theory
Forward Selection	Wrapper	Yes	Yes	Validation score
Backward Elimination	Wrapper	Yes	Yes	Validation score
RFE	Wrapper	Yes	Yes	Model importance
L1/Lasso	Embedded	Yes	Yes	L1 penalty
Decision Tree	Embedded	Yes	Yes	Impurity reduction
Random Forest	Embedded	Yes	Yes	Average tree importance
PCA	Extraction	No	No	Eigenvalues/eigenvectors

19. Important Machine Learning Rule

Never perform feature selection using the complete dataset before cross-validation or testing.

The feature-selection procedure should be fitted only on the training portion of the data. Otherwise information from the validation/test data can influence the selected features, producing data leakage.

Correct Pipeline

Raw Data → Train/Test Split → Feature Selection on Training Data → Model Training → Test Evaluation

With Cross-Validation

Pipeline( Feature Selection + Model ) → Cross Validation → Final Test

20. Final Summary

Technique	What It Does
Variance	Removes features with very little variation.
Correlation	Measures linear relationship with target or between features.
Chi-Square	Measures categorical statistical dependence.
ANOVA	Compares between-group and within-group variation.
Mutual Information	Measures information dependency.
Forward Selection	Adds features progressively.
Backward Elimination	Removes features progressively.
RFE	Recursively removes less important features.
Lasso	Uses L1 regularization to shrink coefficients.
Decision Tree	Uses impurity reduction for feature importance.
Random Forest	Aggregates importance across multiple trees.
PCA	Transforms original features into principal components.

Key idea: Feature selection attempts to reduce the number of input variables while retaining useful predictive information. The appropriate method depends on the dataset, target type, model and computational constraints.

FEATURE SELECTION LABORATORY
5 FEATURES × 10 RECORDS
CREATED BY BIJAN KRISHNA PAUL