🔎 Filter Methods for Feature Selection

Machine Learning Tutorial with Mathematics, Dataset and Python

8 Features × 8 Records

1. What is a Filter Method?

A Filter Method is a feature-selection technique that evaluates features using statistical or mathematical properties of the dataset before the machine-learning model is trained.

The basic idea is:

Dataset → Statistical Measure → Feature Score → Ranking → Select Top Features

Unlike wrapper methods, a filter method does not repeatedly train a machine-learning model for every possible feature subset.

Important idea:

A filter method normally calculates an independent score for each feature. Features with poor scores can then be removed before model training.

2. Dataset: 8 Features × 8 Records

We use a small student-performance dataset. There are 8 input features, 8 observations, and one categorical target called Result.

Student	StudyHours	Attendance	Assignment	SleepHours	ScreenTime	PracticeTests	LibraryVisits	PreviousScore	Result
1	2	60	3	5	8	1	1	45	Fail
2	3	65	4	6	7	2	1	50	Fail
3	4	70	5	6	6	2	2	55	Fail
4	5	75	6	7	5	3	3	62	Pass
5	6	80	7	7	5	4	4	68	Pass
6	7	85	8	8	4	5	5	74	Pass
7	8	90	9	8	3	6	6	80	Pass
8	9	95	10	9	2	7	7	88	Pass

Input features = 8
StudyHours, Attendance, Assignment, SleepHours, ScreenTime, PracticeTests, LibraryVisits, PreviousScore

Target = Result
Fail / Pass

3. Major Filter Methods

📌 Variance Threshold

Removes features with very small variance.

📈 Pearson Correlation

Measures linear relationship between numerical variables.

📊 Spearman Correlation

Measures monotonic relationship using ranks.

χ² Chi-Square

Measures dependence between categorical variables.

📐 ANOVA F-Test

Measures differences between class groups.

🧠 Mutual Information

Measures information shared between feature and target.

🎯 Information Gain

Measures reduction in entropy after observing a feature.

⚖️ Fisher Score

Measures separation between classes.

🔗 Redundancy Analysis

Identifies highly correlated features that provide similar information.

4. Filter Method 1 — Variance Threshold

Variance measures how much the values of a feature change around their mean.

μ = Σxᵢ / n

σ² = Σ(xᵢ − μ)² / n

Example: StudyHours

StudyHours = 2, 3, 4, 5, 6, 7, 8, 9

μ = (2+3+4+5+6+7+8+9) / 8

μ = 44 / 8 = 5.5

Therefore:

σ² = [(2−5.5)² +(3−5.5)² +(4−5.5)² +(5−5.5)² +(6−5.5)² +(7−5.5)² +(8−5.5)² +(9−5.5)²] / 8

σ² = 5.25

A feature with variance close to zero may be removed.

Python

from sklearn.feature_selection import VarianceThreshold

selector = VarianceThreshold(
    threshold=1
)

X_selected = selector.fit_transform(X)

print(selector.get_support())

5. Filter Method 2 — Pearson Correlation

Pearson correlation measures the strength and direction of a linear relationship between two numerical variables.

r = Σ(xᵢ−x̄)(yᵢ−ȳ) / √[Σ(xᵢ−x̄)² × Σ(yᵢ−ȳ)²]

Interpretation

Correlation	Interpretation
+1	Perfect positive relationship
+0.7 to +1	Strong positive relationship
0	No linear relationship
-0.7 to -1	Strong negative relationship
-1	Perfect negative relationship

Example

StudyHours and PreviousScore increase together in this dataset. Therefore their Pearson correlation is expected to be strongly positive.

import pandas as pd

correlation =
df.corr(numeric_only=True)

print(
    correlation["PreviousScore"]
)

6. Filter Method 3 — Spearman Rank Correlation

Spearman correlation first converts values into ranks and then calculates correlation between those ranks.

dᵢ = Rank(Xᵢ) − Rank(Yᵢ)

ρ = 1 − [6Σdᵢ² / n(n²−1)]

Example

For a feature with ranks:

1,2,3,4,5,6,7,8

and target ranks:

1,2,3,4,5,6,7,8

Then:

dᵢ = 0

ρ = 1

Python

from scipy.stats import spearmanr

rho, p_value = spearmanr(
    df["StudyHours"],
    df["PreviousScore"]
)

print("Spearman:", rho)
print("P-value:", p_value)

Pearson vs Spearman

Pearson → linear relationship.
Spearman → monotonic relationship based on ranks.

7. Filter Method 4 — Chi-Square Test

The Chi-Square test evaluates whether two categorical variables are statistically independent.

χ² = Σ (O − E)² / E

where:

• O = Observed frequency
• E = Expected frequency

Expected Frequency

E = (Row Total × Column Total) / Grand Total

Simple Example

Suppose we convert StudyHours into two categories:

Study Level	Fail	Pass
Low	3	0
High	0	5

The expected frequency for Low–Fail is:

E = (3 × 3) / 8 = 1.125

The Chi-Square statistic is obtained by summing:

χ² = Σ [(Observed − Expected)² / Expected]

Python

from sklearn.feature_selection import (
    SelectKBest,
    chi2
)

selector = SelectKBest(
    score_func=chi2,
    k=4
)

X_selected = selector.fit_transform(
    X,
    y
)

print(selector.scores_)

Important: scikit-learn's chi2 feature score requires non-negative feature values. It is commonly used with classification problems.

8. Filter Method 5 — ANOVA F-Test

ANOVA evaluates whether the means of a numerical feature differ between target classes.

F = MS_between / MS_within

Between-Group Mean Square

MS_between = SS_between / (k−1)

Within-Group Mean Square

MS_within = SS_within / (N−k)

where:

• N = total number of observations
• k = number of classes

Interpretation

A larger F-statistic indicates that the feature's group means differ more relative to the variation within groups.

Python

from sklearn.feature_selection import (
    SelectKBest,
    f_classif
)

selector = SelectKBest(
    score_func=f_classif,
    k=4
)

X_selected = selector.fit_transform(
    X,
    y
)

print(selector.scores_)

9. Filter Method 6 — Mutual Information

Mutual Information measures the amount of information shared between a feature X and target Y.

I(X;Y) = Σₓ Σᵧ p(x,y) log [ p(x,y) / (p(x)p(y)) ]

If X and Y are independent:

I(X;Y) = 0

A higher value means stronger statistical dependence.

Python — Classification

from sklearn.feature_selection import (
    mutual_info_classif
)

scores = mutual_info_classif(
    X,
    y,
    random_state=42
)

for feature, score in zip(
    feature_names,
    scores
):
    print(feature, score)

10. Filter Method 7 — Information Gain

Information Gain measures how much uncertainty about the target is reduced after observing a feature.

Entropy

H(Y) = −Σ p(y) log₂ p(y)

In our dataset:

• Fail = 3
• Pass = 5
• Total = 8

Therefore:

H(Y) = −(3/8)log₂(3/8) −(5/8)log₂(5/8)

H(Y) ≈ 0.9544 bits

Conditional Entropy

H(Y|X) = Σ p(x)H(Y|X=x)

Information Gain

IG(Y,X) = H(Y) − H(Y|X)

Higher Information Gain means that the feature provides more information about the target.

Python Concept

from sklearn.feature_selection import (
    mutual_info_classif
)

information_gain = mutual_info_classif(
    X,
    y,
    random_state=42
)

print(information_gain)

For classification, mutual information is closely related to the information-theoretic idea behind information gain, although the exact implementation and estimation procedure may differ.

11. Filter Method 8 — Fisher Score

Fisher Score evaluates how well a feature separates different classes. A feature is useful when its class means are far apart while its within-class variation is small.

Fisher Score = Σₖ nₖ(μₖ−μ)² / Σₖ nₖσₖ²

where:

• nₖ = number of observations in class k
• μₖ = mean of class k
• μ = overall mean
• σₖ² = variance within class k

A larger Fisher Score generally indicates better class separation.

Python Concept

import numpy as np

def fisher_score(feature, y):

    classes = np.unique(y)

    overall_mean = np.mean(feature)

    numerator = 0
    denominator = 0

    for c in classes:

        group = feature[y == c]

        n = len(group)
        mean_c = np.mean(group)
        var_c = np.var(group)

        numerator += n * (
            mean_c - overall_mean
        ) ** 2

        denominator += n * var_c

    return numerator / denominator


for i, feature in enumerate(
    feature_names
):

    score = fisher_score(
        X[:, i],
        y
    )

    print(feature, score)

12. Filter Method 9 — Redundancy Analysis

Two features may individually have strong relationships with the target but contain almost the same information.

For example:

Correlation(StudyHours, PreviousScore) → Strong Positive Relationship

If two features are highly correlated with each other, retaining both may provide little additional information for some models.

Feature-Feature Correlation

corr = df[
    feature_names
].corr()

print(corr)

A high correlation between two features does not automatically mean one must be removed. The decision depends on the model, objective, interpretability and validation results.

13. Ranking Features Using Filter Scores

After calculating a statistical score, features can be ranked.

Feature Ranking = Sort Features by Score

Feature	Variance	Pearson	ANOVA	Mutual Information	Decision
StudyHours	High	High	High	High	Candidate
Attendance	High	High	High	High	Candidate
Assignment	High	High	High	High	Candidate
SleepHours	Moderate	Moderate	Moderate	Evaluate	Candidate
ScreenTime	High	Negative	High	High	Candidate
PracticeTests	High	High	High	High	Candidate
LibraryVisits	High	High	High	High	Candidate
PreviousScore	High	High	High	High	Candidate

Important: The table above illustrates the ranking process conceptually. Actual statistical scores should be calculated from the data rather than manually assigned.

14. Select Top-K Features

Suppose we want only four features from the original eight.

Original Features = 8

Selected Features = 4

The filter procedure is:

Calculate Score → Sort → Select Top 4 → Remove Remaining 4

Python

from sklearn.feature_selection import (
    SelectKBest,
    f_classif
)

selector = SelectKBest(
    score_func=f_classif,
    k=4
)

X_new = selector.fit_transform(
    X,
    y
)

print(
    selector.get_support()
)

15. Complete Python Implementation

import pandas as pd

from sklearn.feature_selection import (
    VarianceThreshold,
    SelectKBest,
    chi2,
    f_classif,
    mutual_info_classif
)

from scipy.stats import spearmanr


# ============================================
# DATASET
# ============================================

df = pd.DataFrame({

    "StudyHours":
    [2,3,4,5,6,7,8,9],

    "Attendance":
    [60,65,70,75,80,85,90,95],

    "Assignment":
    [3,4,5,6,7,8,9,10],

    "SleepHours":
    [5,6,6,7,7,8,8,9],

    "ScreenTime":
    [8,7,6,5,5,4,3,2],

    "PracticeTests":
    [1,2,2,3,4,5,6,7],

    "LibraryVisits":
    [1,1,2,3,4,5,6,7],

    "PreviousScore":
    [45,50,55,62,68,74,80,88],

    "Result":
    [
        "Fail",
        "Fail",
        "Fail",
        "Pass",
        "Pass",
        "Pass",
        "Pass",
        "Pass"
    ]
})


features = [
    "StudyHours",
    "Attendance",
    "Assignment",
    "SleepHours",
    "ScreenTime",
    "PracticeTests",
    "LibraryVisits",
    "PreviousScore"
]

X = df[features]

y = df["Result"]


# ============================================
# 1. VARIANCE
# ============================================

print("\nVARIANCE")

print(
    X.var()
)


# ============================================
# 2. PEARSON
# ============================================

print("\nPEARSON")

numeric_df = df.copy()

numeric_df["ResultCode"] = (
    numeric_df["Result"]
    .map({
        "Fail":0,
        "Pass":1
    })
)

print(
    numeric_df[
        features + ["ResultCode"]
    ].corr()["ResultCode"]
)


# ============================================
# 3. SPEARMAN
# ============================================

print("\nSPEARMAN")

for feature in features:

    rho, p = spearmanr(
        df[feature],
        df["Result"].map({
            "Fail":0,
            "Pass":1
        })
    )

    print(
        feature,
        "rho =",
        rho,
        "p =",
        p
    )


# ============================================
# 4. CHI-SQUARE
# ============================================

print("\nCHI-SQUARE")

y_code = y.map({
    "Fail":0,
    "Pass":1
})

chi_selector = SelectKBest(
    score_func=chi2,
    k="all"
)

chi_selector.fit(
    X,
    y_code
)

for feature, score, p in zip(
    features,
    chi_selector.scores_,
    chi_selector.pvalues_
):

    print(
        feature,
        "Chi2 =",
        score,
        "p =",
        p
    )


# ============================================
# 5. ANOVA
# ============================================

print("\nANOVA")

anova_selector = SelectKBest(
    score_func=f_classif,
    k="all"
)

anova_selector.fit(
    X,
    y_code
)

for feature, score, p in zip(
    features,
    anova_selector.scores_,
    anova_selector.pvalues_
):

    print(
        feature,
        "F =",
        score,
        "p =",
        p
    )


# ============================================
# 6. MUTUAL INFORMATION
# ============================================

print("\nMUTUAL INFORMATION")

mi = mutual_info_classif(
    X,
    y_code,
    random_state=42
)

for feature, score in zip(
    features,
    mi
):

    print(
        feature,
        "MI =",
        score
    )


# ============================================
# 7. SELECT TOP FEATURES
# ============================================

selector = SelectKBest(
    score_func=f_classif,
    k=4
)

X_selected = selector.fit_transform(
    X,
    y_code
)

print("\nSELECTED FEATURES")

for feature, selected in zip(
    features,
    selector.get_support()
):

    if selected:
        print(feature)

16. Complete Filter Method Workflow

Dataset ↓ Data Cleaning ↓ Remove Constant Features ↓ Variance Threshold ↓ Correlation Analysis ↓ Statistical Test ↓ Feature Score ↓ Feature Ranking ↓ Select Top-K ↓ Machine Learning Model

17. Comparison of Filter Methods

Method	Target Type	Mathematical Basis	Main Purpose
Variance Threshold	Any	Variance	Remove low-variation features
Pearson	Numerical	Correlation coefficient	Linear relationship
Spearman	Ordinal/Numerical	Rank correlation	Monotonic relationship
Chi-Square	Categorical	χ² statistic	Categorical dependence
ANOVA	Categorical	F statistic	Compare class means
Mutual Information	Classification/Regression	Information theory	Statistical dependency
Information Gain	Classification	Entropy	Reduction in uncertainty
Fisher Score	Classification	Between/within class variation	Class separation
Redundancy Analysis	Numerical	Feature-feature correlation	Remove duplicate information

18. Advantages of Filter Methods

⚡ Fast

Most filter techniques are computationally inexpensive compared with large wrapper searches.

📊 Model Independent

The feature score can usually be calculated without selecting a particular machine-learning estimator.

📉 Dimensionality Reduction

Unimportant variables can be removed before model training.

🧠 Easy to Interpret

Statistical scores provide an understandable basis for ranking features.

19. Limitations of Filter Methods

• A filter method may evaluate features individually and therefore may not fully capture feature interactions.
• A feature with a weak individual score may become useful when combined with another feature.
• Different statistical measures can produce different rankings.
• A high statistical score does not guarantee improved test accuracy.
• Feature selection must be performed carefully to avoid data leakage.

Important: Feature selection should normally be fitted using only the training data when a separate test set or cross-validation is used.

20. Quick Revision

Question	Answer
What is a filter method?	A model-independent statistical feature-selection approach.
Variance Threshold?	Removes features with low variance.
Pearson correlation?	Measures linear association.
Spearman correlation?	Measures rank-based monotonic association.
Chi-Square?	Measures dependence between categorical variables.
ANOVA?	Compares between-group and within-group variation.
Mutual Information?	Measures statistical information shared by variables.
Information Gain?	Measures reduction in entropy.
Fisher Score?	Measures separation between classes.
Why rank features?	To select the most useful features according to a chosen criterion.

FILTER METHODS FOR FEATURE SELECTION
8 FEATURES × 8 RECORDS
CREATED BY BIJAN KRISHNA PAUL