Bijan Krishna Paul : 🎯 Wrapper Methods for Feature Selection

Saturday, September 26, 2026

🎯 Wrapper Methods for Feature Selection

Machine Learning Tutorial with Mathematics, Dataset and Python

8 Features × 8 Records

1. What is a Wrapper Method?

A Wrapper Method selects features by repeatedly training and evaluating a machine-learning model using different subsets of features.

Features → Feature Subset → Train Model → Evaluate Model → Compare Score → Select Better Subset

The important difference from a filter method is that the usefulness of a feature is judged according to the performance of a selected machine-learning algorithm.

Example: Suppose there are 8 features. A wrapper algorithm may test:

{StudyHours}
{StudyHours, Attendance}
{StudyHours, Attendance, Assignment}
...

and compare the model performance for each subset.

2. Dataset: 8 Features × 8 Records

The following small dataset is used throughout this tutorial.

Student	StudyHours	Attendance	Assignment	SleepHours	ScreenTime	PracticeTests	LibraryVisits	PreviousScore	Result
1	2	60	3	5	8	1	1	45	Fail
2	3	65	4	6	7	2	1	50	Fail
3	4	70	5	6	6	2	2	55	Fail
4	5	75	6	7	5	3	3	62	Pass
5	6	80	7	7	5	4	4	68	Pass
6	7	85	8	8	4	5	5	74	Pass
7	8	90	9	8	3	6	6	80	Pass
8	9	95	10	9	2	7	7	88	Pass

Number of records = 8
Number of input features = 8
Target = Result

3. Types of Wrapper Methods

➡️ Forward Selection

Starts with no features and adds features one by one.

⬅️ Backward Elimination

Starts with all features and removes features one by one.

🔄 Sequential Forward Selection

Sequentially adds the feature that improves the selected evaluation score.

🔄 Sequential Backward Selection

Sequentially removes the feature whose removal gives the best result.

🎯 RFE

Recursive Feature Elimination repeatedly removes the least important features.

🔍 RFECV

RFE combined with cross-validation to determine the feature count.

🧮 Exhaustive Search

Evaluates possible feature subsets systematically.

🏆 Best Subset Selection

Searches for the subset producing the chosen evaluation criterion.

4. Mathematical Idea Behind Wrapper Selection

Let the complete feature set be:

F = {X₁, X₂, X₃, ..., X₈}

A subset S is selected from F:

S ⊆ F

A machine-learning model is trained using S:

Model = f(X_S)

The model produces an evaluation score:

Score(S) = Evaluation(Model(S))

The wrapper searches for a subset that optimizes the chosen criterion. For example, when maximizing accuracy:

S* = argmax_{S ⊆ F} Accuracy(S)

For minimizing an error:

S* = argmin_{S ⊆ F} Error(S)

5. Model Evaluation

A wrapper method needs an evaluation criterion. For classification, common criteria include:

Accuracy

Accuracy = (TP + TN) / (TP + TN + FP + FN)

Precision

Precision = TP / (TP + FP)

Recall

Recall = TP / (TP + FN)

F1 Score

F1 = 2 × Precision × Recall / (Precision + Recall)

In practice, cross-validation is commonly used rather than relying on training accuracy alone.

6. Forward Selection

Forward selection begins with an empty feature set. At each step, one feature is added.

S₀ = ∅

Suppose the eight features are:

X₁ = StudyHours
X₂ = Attendance
X₃ = Assignment
X₄ = SleepHours
X₅ = ScreenTime
X₆ = PracticeTests
X₇ = LibraryVisits
X₈ = PreviousScore

Step 1

Test each feature independently:

Score(X₁), Score(X₂), ..., Score(X₈)

Select the feature giving the best validation score.

S₁ = {X_best}

Step 2

Now test adding every remaining feature:

S₁ ∪ {X₂}
S₁ ∪ {X₃}
...

Select the addition producing the best score.

S₂ = S₁ ∪ {X_{best next}}

Process

∅ → 1 Feature → 2 Features → 3 Features → 4 Features → ...

Python

from mlxtend.feature_selection import (
    SequentialFeatureSelector
)

from sklearn.linear_model import LogisticRegression

model = LogisticRegression(
    max_iter=1000
)

sfs = SequentialFeatureSelector(
    model,
    k_features=4,
    forward=True,
    floating=False,
    scoring="accuracy",
    cv=5
)

sfs.fit(X, y)

print(
    sfs.k_feature_names_
)

7. Backward Elimination

Backward elimination starts with all features.

S₀ = {X₁,X₂,X₃,X₄,X₅,X₆,X₇,X₈}

Now one feature is removed at a time.

Step 1

S₀ − {X₁}
S₀ − {X₂}
...
S₀ − {X₈}

The subset producing the best evaluation score is retained.

S₁ = S₀ − {X_worst}

Process

8 Features → 7 Features → 6 Features → 5 Features → 4 Features

Python

sbs = SequentialFeatureSelector(
    model,
    k_features=4,
    forward=False,
    floating=False,
    scoring="accuracy",
    cv=5
)

sbs.fit(X, y)

print(
    sbs.k_feature_names_
)

8. Sequential Forward Selection (SFS)

SFS is a systematic forward search procedure.

S₀ = ∅

At each iteration:

X* = argmax_{X ∉ S} Score(S ∪ {X})

Then:

S ← S ∪ {X*}

The process continues until the required number of features is selected.

9. Sequential Backward Selection (SBS)

SBS begins with all features.

S₀ = F

At each iteration, one feature is removed.

X* = argmax_{X ∈ S} Score(S − {X})

Then:

S ← S − {X*}

10. Recursive Feature Elimination — RFE

RFE starts with all features and repeatedly removes the least important features according to the selected estimator.

F₀ = {X₁,X₂,...,X₈}

Train the model:

Model(F₀)

Determine feature importance:

Importance(X₁), ..., Importance(X₈)

Remove the least important feature:

F₁ = F₀ − {X_{least important}}

Repeat until the desired number of features remains.

8 → 7 → 6 → 5 → 4

Python

from sklearn.feature_selection import RFE
from sklearn.linear_model import LogisticRegression

model = LogisticRegression(
    max_iter=1000
)

rfe = RFE(
    estimator=model,
    n_features_to_select=4
)

rfe.fit(X, y)

for feature, rank, selected in zip(
    feature_names,
    rfe.ranking_,
    rfe.support_
):

    print(
        feature,
        "Rank:", rank,
        "Selected:", selected
    )

11. RFE Feature Importance

For a linear model, the absolute value of a coefficient can be used as an importance measure.

Importance(Xᵢ) = |βᵢ|

Suppose a model produces:

Feature	Coefficient β	\|β\|
StudyHours	1.82	1.82
Attendance	1.41	1.41
Assignment	1.22	1.22
SleepHours	0.31	0.31
ScreenTime	-0.88	0.88
PracticeTests	1.15	1.15
LibraryVisits	0.76	0.76
PreviousScore	2.01	2.01

The sign indicates direction, while the absolute magnitude is commonly used to represent coefficient-based importance.

12. RFECV — Recursive Feature Elimination with Cross-Validation

RFECV extends RFE by using cross-validation to determine an appropriate number of features.

RFE + Cross-Validation = RFECV

For example, RFECV can evaluate:

Number of Features	Mean CV Score
1	0.75
2	0.80
3	0.84
4	0.88
5	0.86
6	0.85
7	0.84
8	0.83

The cross-validation score is used to determine the useful feature count.

Python

from sklearn.feature_selection import RFECV
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold

model = LogisticRegression(
    max_iter=1000
)

cv = StratifiedKFold(
    n_splits=4,
    shuffle=True,
    random_state=42
)

rfecv = RFECV(
    estimator=model,
    step=1,
    cv=cv,
    scoring="accuracy",
    min_features_to_select=1
)

rfecv.fit(X, y)

print(
    "Selected feature count:",
    rfecv.n_features_
)

print(
    rfecv.support_
)

13. Cross-Validation in Wrapper Methods

A wrapper method can evaluate each feature subset using cross-validation.

For 5-fold cross-validation:

CV Score = (Score₁ + Score₂ + Score₃ + Score₄ + Score₅) / 5

For example, suppose a feature subset produces:

0.80, 0.90, 0.85, 0.80, 0.95

Then:

Mean CV Score = (0.80+0.90+0.85+0.80+0.95)/5

Mean CV Score = 4.30 / 5 = 0.86

Therefore the subset receives an average validation score of:

86%

14. Exhaustive Feature Selection

Exhaustive search evaluates every possible subset of the features.

For n features, the number of all possible subsets is:

2ⁿ

For our 8-feature dataset:

2⁸ = 256 subsets

If the empty subset is excluded:

2⁸ − 1 = 255 non-empty subsets

This illustrates why exhaustive wrapper methods become expensive as the number of features increases.

Number of subsets by size

Features Selected	Number of Subsets
1	C(8,1) = 8
2	C(8,2) = 28
3	C(8,3) = 56
4	C(8,4) = 70
5	C(8,5) = 56
6	C(8,6) = 28
7	C(8,7) = 8
8	C(8,8) = 1
Total	255

15. Best-Subset Selection

Suppose we specifically want four features. The number of four-feature subsets is:

C(8,4) = 8! / [4!(8−4)!]

C(8,4) = 70

Therefore, an exhaustive four-feature wrapper would train and evaluate 70 different models.

Best Subset = argmax_S:|S|=4 CVScore(S)

16. Exhaustive Search — Python

from itertools import combinations

from sklearn.model_selection import (
    cross_val_score
)

from sklearn.linear_model import (
    LogisticRegression
)

model = LogisticRegression(
    max_iter=1000
)

best_score = -1
best_features = None

for combination in combinations(
    feature_names,
    4
):

    X_subset = X[
        list(combination)
    ]

    scores = cross_val_score(
        model,
        X_subset,
        y,
        cv=4,
        scoring="accuracy"
    )

    mean_score = scores.mean()

    print(
        combination,
        mean_score
    )

    if mean_score > best_score:

        best_score = mean_score
        best_features = combination


print(
    "\nBest Features:",
    best_features
)

print(
    "Best CV Score:",
    best_score
)

17. Sequential Feature Selection with scikit-learn

from sklearn.feature_selection import (
    SequentialFeatureSelector
)

from sklearn.linear_model import LogisticRegression

model = LogisticRegression(
    max_iter=1000
)

sfs = SequentialFeatureSelector(
    model,
    n_features_to_select=4,
    direction="forward",
    scoring="accuracy",
    cv=4
)

sfs.fit(X, y)

selected = X.columns[
    sfs.get_support()
]

print(
    "Selected Features:"
)

print(
    selected.tolist()
)

Backward Selection

sbs = SequentialFeatureSelector(
    model,
    n_features_to_select=4,
    direction="backward",
    scoring="accuracy",
    cv=4
)

sbs.fit(X, y)

selected = X.columns[
    sbs.get_support()
]

print(
    selected.tolist()
)

18. Complete Python Dataset

import pandas as pd

data = {

    "StudyHours":
    [2,3,4,5,6,7,8,9],

    "Attendance":
    [60,65,70,75,80,85,90,95],

    "Assignment":
    [3,4,5,6,7,8,9,10],

    "SleepHours":
    [5,6,6,7,7,8,8,9],

    "ScreenTime":
    [8,7,6,5,5,4,3,2],

    "PracticeTests":
    [1,2,2,3,4,5,6,7],

    "LibraryVisits":
    [1,1,2,3,4,5,6,7],

    "PreviousScore":
    [45,50,55,62,68,74,80,88],

    "Result":
    [
        "Fail",
        "Fail",
        "Fail",
        "Pass",
        "Pass",
        "Pass",
        "Pass",
        "Pass"
    ]
}

df = pd.DataFrame(data)

print(df)

19. Comparison of Wrapper Methods

Method	Starting Point	Operation	Uses Model?	Search Direction
Forward Selection	Empty set	Add feature	Yes	Forward
Backward Elimination	All features	Remove feature	Yes	Backward
SFS	Empty set	Sequential addition	Yes	Forward
SBS	All features	Sequential removal	Yes	Backward
RFE	All features	Remove least important	Yes	Backward
RFECV	All features	RFE + CV	Yes	Backward
Exhaustive Search	All possible subsets	Evaluate subsets	Yes	All directions
Best Subset	Fixed subset size	Compare combinations	Yes	Combinatorial

20. Filter Method vs Wrapper Method

Property	Filter	Wrapper
Model required?	Usually No	Yes
Uses statistical score?	Yes	May use model score
Model dependent?	Usually model independent	Model dependent
Computational cost	Usually lower	Usually higher
Feature interactions	May miss interactions	Can capture subset effects
Example	Chi-Square	RFE
Example	ANOVA	Forward Selection
Example	Mutual Information	Backward Elimination

21. Complete Wrapper Workflow

Prepare Dataset
Separate X and y.

Select Machine-Learning Model
Example: Logistic Regression.

Generate Feature Subset
Forward, backward, recursive or exhaustive search.

Train Model
Train the model using the current subset.

Evaluate
Calculate cross-validation score.

Compare
Compare the current subset with candidate subsets.

Select
Keep the subset according to the chosen criterion.

Final Model
Train the final model using the selected features.

22. Computational Cost

Wrapper methods can become expensive because the machine-learning model may need to be trained many times.

Exhaustive Search

Number of subsets = 2ⁿ − 1

Features	Non-empty subsets
5	31
8	255
10	1,023
20	1,048,575
30	1,073,741,823

Important: Exhaustive wrapper search becomes computationally expensive very quickly as the number of features increases.

23. Practical Example: Select 4 of 8 Features

Suppose we want exactly four features from our eight features.

n = 8

k = 4

Number of possible subsets = C(8,4)

= 70

An exhaustive wrapper can evaluate all 70 combinations.

For each combination:

Feature Subset → Train Model → Cross Validation → Mean Score → Store Score

Finally:

Best Subset = argmax Score(S)

24. Advantages of Wrapper Methods

🎯 Model Specific

The selected features are evaluated according to actual model performance.

🔗 Feature Interaction

Wrapper approaches can discover useful combinations of features.

📈 Performance Based

The selection criterion can directly use accuracy, F1, RMSE or another model metric.

🔍 Subset Evaluation

The usefulness of a group of features can be evaluated together.

25. Limitations of Wrapper Methods

They can be computationally expensive.
Repeated model training can take considerable time.
Exhaustive search becomes impractical for high-dimensional datasets.
The selected features can depend on the chosen machine-learning model.
Careless evaluation can cause overfitting.
Cross-validation should be designed carefully to prevent information leakage.

Important: If feature selection is performed before cross-validation using the entire dataset, information from validation folds can leak into the selection process. In a proper workflow, feature selection should be inside the cross-validation/training pipeline.

26. Quick Revision

Question	Answer
What is a wrapper method?	A feature-selection method that evaluates subsets using a machine-learning model.
Forward selection?	Starts with zero features and adds features.
Backward elimination?	Starts with all features and removes features.
SFS?	Sequential Forward Selection.
SBS?	Sequential Backward Selection.
RFE?	Recursive Feature Elimination.
RFECV?	RFE combined with cross-validation.
Exhaustive search?	Evaluates all possible feature subsets.
Number of subsets for 8 features?	2⁸ = 256 including the empty subset; 255 non-empty subsets.
Number of 4-feature subsets from 8?	C(8,4) = 70.

SEM 1	SEM 2	SEM 3
SEM 4	SEM 5	SEM 6

SEM 1	SEM 2	SEM 3
SEM 4	SEM 5	SEM 6

SEM 1	SEM 2	SEM 3
SEM 4	SEM 5	SEM 6

CLASS-4	CLASS-5	CLASS-6
CLASS-7	CLASS-8	CLASS-9
CLASS10	CLASS11 application	CLASS12 application
CLASS11 science	CLASS12 science

C	C++	CORE JAVA	SQL	PYTHON
MS OFFICE	HTML	VISUAL BASIC	advanced java	8085
PROLOG	ASSEMBLY LANGUAGE	JAVA SCRIPT	SHELL PROGRAMMING	R
DIGITAL ELECTRONICS	COMPUTER ARCHITECTURE	DATA STRUCTURE	OPERATING SYSTEM	GRAPH THEORY
DISCRETE MATHEMATICS	NUMERICAL ALGORITHM	AUTOMATA	MICROPROCESSOR	NETWORKING
GRAPHICS	SOFTWARE ENGINEERING	DATABSE	ANALYSIS OF ALGORITHM	IMAGE PROCESSING
ARTIFICIAL INTELLIGENCE	BIG DATA	CLOUD COMPUTING	ALL Types Of Machine Learning	machine learning using python

CU BSC computer science old syllabus	WBSU BSC computer science old syllabus
CU cbcs BSC computer science HONOURS syllabus 2018	WBSU cbcs BSc computer science HONOURS syllabus 2018
CU cbcs BSC computer science GENERAL syllabus 2018	WBSU cbcs BSC computer science GENERAL syllabus 2018

Total Pageviews

Saturday, September 26, 2026

🎯 Wrapper Methods for Feature Selection

🎯 Wrapper Methods for Feature Selection

1. What is a Wrapper Method?

2. Dataset: 8 Features × 8 Records

3. Types of Wrapper Methods

➡️ Forward Selection

⬅️ Backward Elimination

🔄 Sequential Forward Selection

🔄 Sequential Backward Selection

🎯 RFE

🔍 RFECV

🧮 Exhaustive Search

🏆 Best Subset Selection

4. Mathematical Idea Behind Wrapper Selection

5. Model Evaluation

Accuracy

Precision

Recall

F1 Score

6. Forward Selection

Step 1

Step 2

Process

Python

7. Backward Elimination

Step 1

Process

Python

8. Sequential Forward Selection (SFS)

9. Sequential Backward Selection (SBS)

10. Recursive Feature Elimination — RFE

Python

11. RFE Feature Importance

12. RFECV — Recursive Feature Elimination with Cross-Validation

Python

13. Cross-Validation in Wrapper Methods

14. Exhaustive Feature Selection

Number of subsets by size

15. Best-Subset Selection

16. Exhaustive Search — Python

17. Sequential Feature Selection with scikit-learn

Backward Selection

18. Complete Python Dataset

19. Comparison of Wrapper Methods

20. Filter Method vs Wrapper Method

21. Complete Wrapper Workflow

22. Computational Cost

Exhaustive Search

23. Practical Example: Select 4 of 8 Features

24. Advantages of Wrapper Methods

🎯 Model Specific

🔗 Feature Interaction

📈 Performance Based

🔍 Subset Evaluation

25. Limitations of Wrapper Methods

26. Quick Revision

No comments:

Post a Comment

About Me