š Feature Selection Laboratory
Complete Mathematical & Interactive Implementation
5 Features × 10 Records Dataset
1. Dataset Used for All Methods
We will use exactly 5 input features and 10 records. The target variable is ExamScore.
| Student | StudyHours | Attendance | SleepHours | Assignments | ScreenTime | ExamScore |
|---|---|---|---|---|---|---|
| 1 | 2 | 60 | 5 | 3 | 8 | 45 |
| 2 | 3 | 65 | 6 | 4 | 7 | 50 |
| 3 | 4 | 70 | 6 | 5 | 6 | 55 |
| 4 | 5 | 75 | 7 | 6 | 5 | 62 |
| 5 | 6 | 80 | 7 | 7 | 5 | 68 |
| 6 | 7 | 82 | 8 | 8 | 4 | 74 |
| 7 | 8 | 85 | 8 | 8 | 3 | 80 |
| 8 | 9 | 90 | 9 | 9 | 3 | 86 |
| 9 | 10 | 92 | 9 | 10 | 2 | 92 |
| 10 | 10 | 95 | 10 | 10 | 1 | 96 |
Target: ExamScore
Records: 10
Features: 5
2. Mathematical Representation
The dataset can be represented by a feature matrix X and target vector y.
x₂₁ x₂₂ x₂₃ x₂₄ x₂₅
⋮
x₁₀₁ x₁₀₂ x₁₀₃ x₁₀₄ x₁₀₅ ]
Therefore:
and:
The objective is to determine whether all five features are useful or whether a smaller subset can provide similar predictive information.
3. Feature Selection Categories
šµ Filter Methods
Correlation, Variance Threshold, Chi-Square, ANOVA and Mutual Information.
š£ Wrapper Methods
Forward Selection, Backward Elimination and Recursive Feature Elimination.
š¢ Embedded Methods
L1/Lasso, Decision Tree and Random Forest importance.
š Feature Extraction
PCA creates new components rather than selecting original columns.
4. Variance Threshold
Variance measures how much a feature changes across observations. A feature with extremely small variance may contain little information.
Example: StudyHours
StudyHours = 2, 3, 4, 5, 6, 7, 8, 9, 10, 10
Mean:
Population variance:
Python
from sklearn.feature_selection import VarianceThreshold selector = VarianceThreshold(threshold=1) X_new = selector.fit_transform(X)
5. Correlation-Based Feature Selection
Correlation measures the linear relationship between a feature and the target.
Interpretation:
| |r| | Interpretation |
|---|---|
| 0 | No linear relationship |
| 0 to 0.3 | Weak |
| 0.3 to 0.7 | Moderate |
| 0.7 to 1 | Strong |
| 1 | Perfect linear relationship |
Python
import pandas as pd correlation = df.corr(numeric_only=True) print(correlation["ExamScore"].sort_values( ascending=False ))
6. Chi-Square Feature Selection
The Chi-Square test measures statistical dependence between categorical variables. In machine learning it is commonly used for classification problems.
where:
- O = observed frequency
- E = expected frequency
The expected frequency is:
Python
from sklearn.feature_selection import SelectKBest from sklearn.feature_selection import chi2 selector = SelectKBest( score_func=chi2, k=3 ) X_selected = selector.fit_transform(X, y)
7. ANOVA F-Test
ANOVA compares variation between groups with variation within groups. For classification, it can evaluate numerical features against a categorical target.
where:
Python
from sklearn.feature_selection import SelectKBest from sklearn.feature_selection import f_classif selector = SelectKBest( score_func=f_classif, k=3 ) X_selected = selector.fit_transform(X, y)
8. Mutual Information
Mutual Information measures how much knowing one variable reduces uncertainty about another.
Higher mutual information indicates greater statistical dependency.
Python
from sklearn.feature_selection import mutual_info_regression scores = mutual_info_regression(X, y) for feature, score in zip( feature_names, scores ): print(feature, score)
9. Forward Selection
Forward selection begins with zero features and adds one feature at a time. At each step, the feature that produces the best validation performance is considered.
Then:
The process continues until the required number of features is reached.
Conceptual Python
selected = []
remaining = feature_names.copy()
while remaining:
```
best_feature = None
best_score = -float("inf")
for feature in remaining:
current = selected + [feature]
# train model using current
# calculate validation score
# if score > best_score:
# best_score = score
# best_feature = feature
selected.append(best_feature)
remaining.remove(best_feature)
```
10. Backward Elimination
Backward elimination starts with all five features and removes one feature at a time.
At every iteration the least useful feature is removed.
Conceptual Python
selected = feature_names.copy() while len(selected) > 2: ``` # evaluate removing each feature # remove feature whose removal # causes the smallest performance loss pass ```
11. Recursive Feature Elimination — RFE
RFE repeatedly fits a model and removes the least important feature.
Python
from sklearn.feature_selection import RFE from sklearn.linear_model import LinearRegression model = LinearRegression() rfe = RFE( estimator=model, n_features_to_select=2 ) rfe.fit(X, y) print(rfe.support_) print(rfe.ranking_)
| Feature | Selected? | Rank |
|---|---|---|
| StudyHours | ✓ / ✗ | 1–5 |
| Attendance | ✓ / ✗ | 1–5 |
| SleepHours | ✓ / ✗ | 1–5 |
| Assignments | ✓ / ✗ | 1–5 |
| ScreenTime | ✓ / ✗ | 1–5 |
12. L1 Regularization / Lasso
L1 regularization adds the absolute value of coefficients to the loss function.
If a coefficient becomes zero:
the corresponding feature can effectively be excluded from the linear model.
Python
from sklearn.linear_model import Lasso model = Lasso(alpha=0.1) model.fit(X, y) for feature, coefficient in zip( feature_names, model.coef_ ): print(feature, coefficient)
13. Decision Tree Feature Importance
Decision trees can estimate feature importance from the reduction in impurity produced by splits.
The normalized feature importances usually sum to 1.
from sklearn.tree import DecisionTreeRegressor model = DecisionTreeRegressor( random_state=42 ) model.fit(X, y) for feature, importance in zip( feature_names, model.feature_importances_ ): print(feature, importance)
14. Random Forest Feature Importance
Random Forest combines many decision trees. Feature importance can be obtained from the fitted forest.
Python
from sklearn.ensemble import RandomForestRegressor model = RandomForestRegressor( n_estimators=100, random_state=42 ) model.fit(X, y) importance = model.feature_importances_ for f, score in zip( feature_names, importance ): print(f, score)
15. PCA — Feature Extraction
PCA is included here for comparison, but technically it is a feature-extraction/dimensionality-reduction technique rather than selection of original columns.
Step 1 — Standardization
Step 2 — Covariance Matrix
Step 3 — Eigenvalue Problem
The eigenvectors corresponding to the largest eigenvalues become the principal components.
Explained Variance Ratio
Python
from sklearn.preprocessing import StandardScaler from sklearn.decomposition import PCA X_scaled = StandardScaler().fit_transform(X) pca = PCA(n_components=2) X_pca = pca.fit_transform(X_scaled) print(pca.explained_variance_ratio_) print(X_pca)
16. Interactive Feature Selection Simulator
The following JavaScript calculates several basic statistics directly from the 10-record dataset.
17. Complete Python Implementation
import pandas as pd
from sklearn.feature_selection import (
SelectKBest,
f_classif,
mutual_info_regression,
RFE,
VarianceThreshold
)
from sklearn.linear_model import Lasso, LinearRegression
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import RandomForestRegressor
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
# ------------------------------------------------
# DATASET
# ------------------------------------------------
df = pd.DataFrame({
```
"StudyHours":
[2,3,4,5,6,7,8,9,10,10],
"Attendance":
[60,65,70,75,80,82,85,90,92,95],
"SleepHours":
[5,6,6,7,7,8,8,9,9,10],
"Assignments":
[3,4,5,6,7,8,8,9,10,10],
"ScreenTime":
[8,7,6,5,5,4,3,3,2,1],
"ExamScore":
[45,50,55,62,68,74,80,86,92,96]
```
})
features = [
"StudyHours",
"Attendance",
"SleepHours",
"Assignments",
"ScreenTime"
]
X = df[features]
y = df["ExamScore"]
# ------------------------------------------------
# CORRELATION
# ------------------------------------------------
print("\nCORRELATION")
print(df.corr(numeric_only=True)["ExamScore"])
# ------------------------------------------------
# VARIANCE THRESHOLD
# ------------------------------------------------
print("\nVARIANCE")
vt = VarianceThreshold(
threshold=1
)
vt.fit(X)
for feature, selected in zip(
features,
vt.get_support()
):
print(feature, selected)
# ------------------------------------------------
# MUTUAL INFORMATION
# ------------------------------------------------
print("\nMUTUAL INFORMATION")
mi = mutual_info_regression(
X,
y,
random_state=42
)
for feature, score in zip(
features,
mi
):
print(feature, score)
# ------------------------------------------------
# RFE
# ------------------------------------------------
print("\nRFE")
rfe = RFE(
LinearRegression(),
n_features_to_select=2
)
rfe.fit(X, y)
for feature, selected, rank in zip(
features,
rfe.support_,
rfe.ranking_
):
print(
feature,
selected,
rank
)
# ------------------------------------------------
# LASSO
# ------------------------------------------------
print("\nLASSO")
lasso = Lasso(alpha=0.1)
lasso.fit(X, y)
for feature, coefficient in zip(
features,
lasso.coef_
):
print(
feature,
coefficient
)
# ------------------------------------------------
# DECISION TREE
# ------------------------------------------------
print("\nDECISION TREE")
tree = DecisionTreeRegressor(
random_state=42
)
tree.fit(X, y)
for feature, importance in zip(
features,
tree.feature_importances_
):
print(
feature,
importance
)
# ------------------------------------------------
# RANDOM FOREST
# ------------------------------------------------
print("\nRANDOM FOREST")
forest = RandomForestRegressor(
n_estimators=100,
random_state=42
)
forest.fit(X, y)
for feature, importance in zip(
features,
forest.feature_importances_
):
print(
feature,
importance
)
# ------------------------------------------------
# PCA
# ------------------------------------------------
print("\nPCA")
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
pca = PCA(
n_components=2
)
X_pca = pca.fit_transform(
X_scaled
)
print(
"Explained variance:",
pca.explained_variance_ratio_
)
print(
"PCA data:"
)
print(X_pca) 18. Method Comparison
| Method | Category | Uses Target? | Original Features? | Main Mathematics |
|---|---|---|---|---|
| Variance Threshold | Filter | No | Yes | Variance |
| Correlation | Filter | Yes | Yes | Pearson r |
| Chi-Square | Filter | Yes | Yes | Ļ² |
| ANOVA | Filter | Yes | Yes | F-statistic |
| Mutual Information | Filter | Yes | Yes | Information theory |
| Forward Selection | Wrapper | Yes | Yes | Validation score |
| Backward Elimination | Wrapper | Yes | Yes | Validation score |
| RFE | Wrapper | Yes | Yes | Model importance |
| L1/Lasso | Embedded | Yes | Yes | L1 penalty |
| Decision Tree | Embedded | Yes | Yes | Impurity reduction |
| Random Forest | Embedded | Yes | Yes | Average tree importance |
| PCA | Extraction | No | No | Eigenvalues/eigenvectors |
19. Important Machine Learning Rule
The feature-selection procedure should be fitted only on the training portion of the data. Otherwise information from the validation/test data can influence the selected features, producing data leakage.
Correct Pipeline
With Cross-Validation
20. Final Summary
| Technique | What It Does |
|---|---|
| Variance | Removes features with very little variation. |
| Correlation | Measures linear relationship with target or between features. |
| Chi-Square | Measures categorical statistical dependence. |
| ANOVA | Compares between-group and within-group variation. |
| Mutual Information | Measures information dependency. |
| Forward Selection | Adds features progressively. |
| Backward Elimination | Removes features progressively. |
| RFE | Recursively removes less important features. |
| Lasso | Uses L1 regularization to shrink coefficients. |
| Decision Tree | Uses impurity reduction for feature importance. |
| Random Forest | Aggregates importance across multiple trees. |
| PCA | Transforms original features into principal components. |