Saturday, September 26, 2026

📊 Feature Selection in Machine Learning

```

📊 Feature Selection in Machine Learning

Concepts, Techniques, Python Implementation and Comparison

1. What is Feature Selection?

Feature Selection is the process of choosing the most useful input variables from a dataset for building a machine learning model. Instead of using every available column, we retain features that provide useful information for predicting the target.

Example:
Suppose a student-performance dataset contains: Study Hours, Attendance, Sleep Hours, Favorite Color, Student ID and Exam Score. If the target is Exam Score, some variables may provide little useful predictive information. Feature selection helps identify the useful variables.

Why is Feature Selection Important?

Reduces the number of input variables.
Can reduce model training time.
Can make models easier to interpret.
Can reduce the effect of irrelevant or redundant variables.
Can help reduce overfitting when irrelevant features are removed.
Can simplify machine learning pipelines.

2. Feature Selection vs Feature Extraction

Feature Selection	Feature Extraction
Selects existing features from the dataset.	Creates new features from existing features.
Original feature meaning is retained.	New transformed variables may be less directly interpretable.
Examples: Chi-Square, RFE, SelectKBest.	Examples: PCA, ICA, NMF.

Important: PCA is generally considered a dimensionality-reduction / feature-extraction technique rather than conventional feature selection, because it transforms the original variables into new components.

3. Main Types of Feature Selection

🔵 Filter Methods

Features are evaluated using statistical or information-based measures, usually independently of a particular ML model.

🟣 Wrapper Methods

Different feature subsets are evaluated using a machine learning model.

🟢 Embedded Methods

Feature selection happens as part of model training, such as through regularization or tree-based importance.

4. Filter Methods

Filter methods evaluate features using properties of the data, rather than repeatedly training a particular predictive model.

4.1 Correlation

Correlation measures the strength and direction of association between numerical variables. Highly correlated predictor variables can contain redundant information.

Pearson Correlation: r = Cov(X,Y) / (σX × σY)

```

import pandas as pd

df = pd.DataFrame({
    "Age": [20,21,22,23,24],
    "StudyHours": [2,4,5,7,8],
    "Score": [45,55,65,78,88]
})

print(df.corr())

```

             Age  StudyHours  Score
Age         1.00       0.99    0.99
StudyHours  0.99       1.00    0.99
Score       0.99       0.99    1.00

```

4.2 Chi-Square Test

The Chi-Square test is commonly used for testing the relationship between categorical features and a categorical target. In scikit-learn, chi2 can be used with non-negative feature values.

χ² = Σ (O − E)² / E

```

from sklearn.feature_selection import SelectKBest, chi2
from sklearn.datasets import load_iris

X, y = load_iris(return_X_y=True)

selector = SelectKBest(score_func=chi2, k=2)

X_new = selector.fit_transform(X, y)

print("Original shape:", X.shape)
print("Selected shape:", X_new.shape)

```

Original shape: (150, 4)
Selected shape: (150, 2)

```

4.3 Mutual Information

Mutual information measures the amount of information that one variable provides about another. It can detect nonlinear relationships that simple correlation may not capture.

```

from sklearn.feature_selection import mutual_info_classif
from sklearn.datasets import load_iris

X, y = load_iris(return_X_y=True)

scores = mutual_info_classif(X, y, random_state=42)

for i, score in enumerate(scores):
    print("Feature", i + 1, ":", round(score, 4))

```

4.4 ANOVA F-Test

The ANOVA F-test can be used to evaluate the relationship between numerical input features and a categorical target.

```

from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.datasets import load_iris

X, y = load_iris(return_X_y=True)

selector = SelectKBest(f_classif, k=2)

X_new = selector.fit_transform(X, y)

print(X_new.shape)

```

4.5 Variance Threshold

A feature with almost no variation across observations may provide little information. VarianceThreshold removes features whose variance is below a specified threshold.

```

from sklearn.feature_selection import VarianceThreshold

X = [
    [1, 10, 5],
    [1, 20, 5],
    [1, 30, 5],
    [1, 40, 5]
]

selector = VarianceThreshold(threshold=0)

X_new = selector.fit_transform(X)

print(X_new)

```

[[10]
 [20]
 [30]
 [40]]

```

5. Wrapper Methods

Wrapper methods evaluate feature subsets by training and testing a chosen machine learning model. They can be computationally expensive because many candidate subsets may need to be evaluated.

5.1 Forward Selection

Forward selection starts with no selected features and adds features one at a time according to a chosen evaluation criterion.

Basic idea:
Start → Select the best feature → Add another useful feature → Continue until the desired number of features is reached.

5.2 Backward Elimination

Backward elimination starts with all available features and removes features step by step according to the selected evaluation criterion.

5.3 Recursive Feature Elimination — RFE

RFE repeatedly trains a model, ranks features according to model importance, and removes less important features until the requested number remains.

```

from sklearn.datasets import load_iris
from sklearn.feature_selection import RFE
from sklearn.linear_model import LogisticRegression

X, y = load_iris(return_X_y=True)

model = LogisticRegression(max_iter=1000)

rfe = RFE(
    estimator=model,
    n_features_to_select=2
)

rfe.fit(X, y)

print("Selected:", rfe.support_)
print("Ranking:", rfe.ranking_)

```

Selected: [False False  True  True]
Ranking:  [3 2 1 1]

```

5.4 Exhaustive Feature Selection

Exhaustive selection evaluates many or all possible feature combinations within a specified range and chooses a subset according to a model-performance criterion.

Limitation: If there are many features, the number of possible combinations can become extremely large, making exhaustive search computationally expensive.

6. Embedded Methods

Embedded methods perform feature selection during the process of fitting a model. The model itself provides information about feature relevance.

6.1 L1 Regularization

L1 regularization adds a penalty based on the absolute values of model coefficients. For some linear models, this can drive coefficients exactly to zero, effectively removing corresponding features.

```

from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression

X, y = load_iris(return_X_y=True)

model = LogisticRegression(
    penalty="l1",
    solver="liblinear",
    max_iter=1000
)

model.fit(X, y)

print(model.coef_)

```

6.2 Tree-Based Feature Importance

Decision trees and ensemble methods such as Random Forest can provide feature-importance measures derived from the fitted trees.

```

from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier

X, y = load_iris(return_X_y=True)

model = RandomForestClassifier(
    n_estimators=100,
    random_state=42
)

model.fit(X, y)

for name, importance in zip(
    load_iris().feature_names,
    model.feature_importances_
):
    print(name, round(importance, 4))

```

7. Dimensionality Reduction and Feature Transformation

Some techniques reduce the dimensionality of data by creating a smaller representation of the original variables. These techniques should be distinguished from selecting original columns.

7.1 Principal Component Analysis — PCA

PCA transforms correlated numerical variables into a smaller set of principal components. The components are ordered according to the variance they explain.

```

from sklearn.datasets import load_wine
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA

wine = load_wine()

X = wine.data

X_scaled = StandardScaler().fit_transform(X)

pca = PCA(n_components=2)

X_pca = pca.fit_transform(X_scaled)

print("Original:", X.shape)
print("After PCA:", X_pca.shape)

print("Explained variance:")
print(pca.explained_variance_ratio_)

```

Original: (178, 13)
After PCA: (178, 2)

Explained variance:
[approximately 0.36, approximately 0.19]

```

7.2 ICA

Independent Component Analysis attempts to represent data using statistically independent components. It is particularly associated with signal separation and source-separation problems.

7.3 NMF

Non-negative Matrix Factorization decomposes a non-negative data matrix into lower-dimensional non-negative matrices. It can be useful for topics, images and other non-negative datasets.

7.4 t-SNE

t-SNE is primarily a nonlinear dimensionality-reduction and visualization technique. It is commonly used to visualize high-dimensional data in two or three dimensions rather than as a conventional feature-selection method.

7.5 Autoencoders

An autoencoder is a neural network trained to reconstruct its input. Its lower-dimensional bottleneck representation can be used as a learned representation of the original data.

8. Complete Python Example — SelectKBest

The following example demonstrates a simple feature-selection workflow using the Iris dataset.

```

from sklearn.datasets import load_iris
from sklearn.feature_selection import SelectKBest
from sklearn.feature_selection import f_classif

# Load dataset
iris = load_iris()

X = iris.data
y = iris.target

# Select the best two features
selector = SelectKBest(
    score_func=f_classif,
    k=2
)

X_selected = selector.fit_transform(X, y)

# Display results
print("Original shape:", X.shape)
print("Selected shape:", X_selected.shape)

print("\nSelected features:")

for feature, selected in zip(
    iris.feature_names,
    selector.get_support()
):
    if selected:
        print(feature)

```

Original shape: (150, 4)
Selected shape: (150, 2)

Selected features:
petal length (cm)
petal width (cm)

```

9. Recommended Feature Selection Workflow

Step 1

Understand the dataset and identify the target variable.

Step 2

Remove obvious identifiers or data-leakage variables.

Step 3

Handle missing values and encode categorical variables appropriately.

Step 4

Split the data into training and test sets.

Step 5

Perform feature selection using only the training data.

Step 6

Train the final model using the selected features.

Step 7

Evaluate performance on unseen test data.

Important — Data Leakage: Feature selection should normally be fitted inside the training process, especially when cross-validation is being used. Selecting features using the complete dataset before validation can allow information from the validation/test data to influence the selection.

10. Comparison of Feature Selection Methods

Method	Uses ML Model?	Main Idea	Typical Advantage	Typical Limitation
Correlation	No	Measures association	Simple and fast	May miss nonlinear relationships
Chi-Square	No	Statistical association	Useful for categorical classification problems	Requires appropriate non-negative feature representation
Mutual Information	No	Measures information dependence	Can detect nonlinear dependency	Scores can require careful interpretation
Forward Selection	Yes	Add features progressively	Model-specific	Can be computationally expensive
Backward Elimination	Yes	Remove features progressively	Model-specific	Can require many model fits
RFE	Yes	Recursively eliminate features	Works with many estimators	Repeated model fitting
L1 Regularization	Yes	Penalizes coefficients	Can produce sparse models	Depends on model and regularization
Random Forest Importance	Yes	Uses tree-based importance	Captures nonlinear relationships	Importance measures require interpretation
PCA	No target required	Creates principal components	Strong dimensionality reduction	Components are transformed variables

11. Advantages of Feature Selection

📉 Reduces dimensionality.
⚡ May reduce computational cost.
🎯 Can focus the model on informative variables.
🧠 Can improve interpretability.
🛡️ Can help control overfitting when irrelevant variables are removed.
📊 Can simplify visualization and analysis.

12. Limitations

A selected feature may be useful only in combination with another feature.
Different methods can produce different feature subsets.
Wrapper methods can require substantial computation.
Feature selection can be affected by preprocessing choices.
Feature selection does not automatically guarantee better test performance.
Selection must be performed carefully to avoid data leakage.

13. Quick Revision

Question	Answer
What is feature selection?	Selecting useful input variables from the original feature set.
Three major supervised categories?	Filter, Wrapper and Embedded methods.
Example of a filter method?	Chi-Square, correlation, mutual information or ANOVA F-test.
Example of a wrapper method?	RFE, forward selection or backward elimination.
Example of an embedded method?	L1 regularization or tree-based feature importance.
What does RFE mean?	Recursive Feature Elimination.
What does PCA do?	Transforms data into principal components with reduced dimensionality.
Main purpose?	Reduce unnecessary information and build a simpler, potentially more effective model.

```

SEM 1	SEM 2	SEM 3
SEM 4	SEM 5	SEM 6

SEM 1	SEM 2	SEM 3
SEM 4	SEM 5	SEM 6

SEM 1	SEM 2	SEM 3
SEM 4	SEM 5	SEM 6

CLASS-4	CLASS-5	CLASS-6
CLASS-7	CLASS-8	CLASS-9
CLASS10	CLASS11 application	CLASS12 application
CLASS11 science	CLASS12 science

C	C++	CORE JAVA	SQL	PYTHON
MS OFFICE	HTML	VISUAL BASIC	advanced java	8085
PROLOG	ASSEMBLY LANGUAGE	JAVA SCRIPT	SHELL PROGRAMMING	R
DIGITAL ELECTRONICS	COMPUTER ARCHITECTURE	DATA STRUCTURE	OPERATING SYSTEM	GRAPH THEORY
DISCRETE MATHEMATICS	NUMERICAL ALGORITHM	AUTOMATA	MICROPROCESSOR	NETWORKING
GRAPHICS	SOFTWARE ENGINEERING	DATABSE	ANALYSIS OF ALGORITHM	IMAGE PROCESSING
ARTIFICIAL INTELLIGENCE	BIG DATA	CLOUD COMPUTING	ALL Types Of Machine Learning	machine learning using python

CU BSC computer science old syllabus	WBSU BSC computer science old syllabus
CU cbcs BSC computer science HONOURS syllabus 2018	WBSU cbcs BSc computer science HONOURS syllabus 2018
CU cbcs BSC computer science GENERAL syllabus 2018	WBSU cbcs BSC computer science GENERAL syllabus 2018

Total Pageviews

Saturday, September 26, 2026