🔍 Feature Selection — Boston Housing

Choosing the most relevant features to improve model performance

1

Correlation-based selection

Drop features with very low correlation to the target (MEDV), and check for multicollinearity between features themselves.

corr_matrix = df_clean.corr()
target_corr = corr_matrix['MEDV'].abs().sort_values(ascending=False)
print(target_corr)

# Drop features with correlation below a threshold
low_corr_features = target_corr[target_corr < 0.2].index
print("Weakly correlated features:", list(low_corr_features))

2

Variance Threshold

Remove features that barely vary across samples, since they carry little predictive information.

from sklearn.feature_selection import VarianceThreshold

selector = VarianceThreshold(threshold=0.01)
X_var = selector.fit_transform(X_train_scaled)
print("Features kept:", X_train_scaled.columns[selector.get_support()].tolist())

3

SelectKBest (Univariate)

Score each feature individually against the target using a statistical test (f_regression) and keep the top K.

from sklearn.feature_selection import SelectKBest, f_regression

selector = SelectKBest(score_func=f_regression, k=8)
X_kbest = selector.fit_transform(X_train_scaled, y_train)

selected_cols = X_train_scaled.columns[selector.get_support()]
print("Top features:", list(selected_cols))

4

Recursive Feature Elimination (RFE)

Iteratively fit a model, rank features by importance, and remove the weakest ones until the desired number remains.

from sklearn.feature_selection import RFE
from sklearn.linear_model import LinearRegression

model = LinearRegression()
rfe = RFE(model, n_features_to_select=8)
rfe.fit(X_train_scaled, y_train)

selected_features = X_train_scaled.columns[rfe.support_]
print("RFE selected features:", list(selected_features))

5

Feature Importance (Tree-based)

Use a Random Forest to rank features by how much they reduce prediction error, which also captures non-linear relationships.

from sklearn.ensemble import RandomForestRegressor

rf = RandomForestRegressor(random_state=42)
rf.fit(X_train_scaled, y_train)

importances = pd.Series(rf.feature_importances_, index=X_train_scaled.columns)
importances.sort_values(ascending=False).plot(kind='bar', figsize=(10,5))
plt.title("Feature Importance (Random Forest)")
plt.show()

6

Finalize selected features

Rebuild the training and test sets using only the chosen features before modeling.

final_features = selected_features  # or selected_cols, based on method chosen

X_train_final = X_train_scaled[final_features]
X_test_final = X_test_scaled[final_features]

Created by Bijan Krishna Paul