🧪 Feature Extraction — Boston Housing Dataset

Creating new, more compact features from existing ones

1

What is Feature Extraction?

Instead of selecting existing columns, feature extraction combines them into new features. The most common beginner-friendly method is PCA (Principal Component Analysis) — it compresses many correlated features into fewer "components" while keeping most of the important information.

2

Import PCA

Bring in PCA from sklearn's decomposition module.

from sklearn.decomposition import PCA

3

Apply PCA

Reduce the scaled features into a smaller number of new components. Here we keep 5 components — you can change this number.

pca = PCA(n_components=5)
X_train_pca = pca.fit_transform(X_train_scaled)
X_test_pca = pca.transform(X_test_scaled)

print("Original shape:", X_train_scaled.shape)
print("Reduced shape:", X_train_pca.shape)

4

Check how much information is kept

Each component explains a percentage of the total variance (information) in the data. Add them up to see how much you kept.

print("Variance explained by each component:")
print(pca.explained_variance_ratio_)

print("Total variance kept:", sum(pca.explained_variance_ratio_))

5

Visualize the variance (Elbow plot)

Plotting cumulative variance helps decide how many components are "enough" — usually where the curve flattens out.

pca_full = PCA().fit(X_train_scaled)
plt.plot(range(1, len(pca_full.explained_variance_ratio_)+1),
         pca_full.explained_variance_ratio_.cumsum(), marker='o')
plt.xlabel('Number of Components')
plt.ylabel('Cumulative Variance Explained')
plt.title('PCA — How many components do we need?')
plt.grid(True)
plt.show()

6

Use the new features in a model

The PCA-transformed data can now be fed into any model, like KNN, just like before.

knn_pca = KNeighborsRegressor(n_neighbors=5)
knn_pca.fit(X_train_pca, y_train)

y_pred_pca = knn_pca.predict(X_test_pca)
print("R2 Score with PCA features:", r2_score(y_test, y_pred_pca))

7

Simple manual feature creation (optional)

Feature extraction doesn't always need PCA — you can also create new features manually using domain knowledge. Example: combining rooms and age into a single "livability" score.

df_clean['ROOMS_PER_AGE'] = df_clean['RM'] / (df_clean['AGE'] + 1)

Created by Bijan Krishna Paul