🏠 Boston Housing Dataset — Preprocessing

A colorful step-by-step pipeline for cleaning & preparing the data

1

Import libraries

Load pandas, numpy, matplotlib, seaborn, sklearn utilities.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

2

Load the dataset

Fetch data from the source and build a DataFrame.

data_url = "http://lib.stat.cmu.edu/datasets/boston"
raw_df = pd.read_csv(data_url, sep=r"\s+", skiprows=22, header=None)
data = np.hstack([raw_df.values[::2, :], raw_df.values[1::2, :2]])
target = raw_df.values[1::2, 2]

columns = ['CRIM','ZN','INDUS','CHAS','NOX','RM','AGE','DIS',
           'RAD','TAX','PTRATIO','B','LSTAT']
df = pd.DataFrame(data, columns=columns)
df['MEDV'] = target

3

Basic exploration

Check shape, types, and summary statistics.

print(df.shape)
print(df.info())
print(df.describe())

4

Handle missing values

Identify and impute nulls if present.

print(df.isnull().sum())
# df.fillna(df.mean(), inplace=True)

5

Remove duplicates

Drop repeated rows.

print(df.duplicated().sum())
df.drop_duplicates(inplace=True)

6

Handle outliers (IQR)

Filter extreme values in the target using IQR.

def remove_outliers_iqr(df, column):
    Q1 = df[column].quantile(0.25)
    Q3 = df[column].quantile(0.75)
    IQR = Q3 - Q1
    lower, upper = Q1 - 1.5*IQR, Q3 + 1.5*IQR
    return df[(df[column] >= lower) & (df[column] <= upper)]

df_clean = remove_outliers_iqr(df, 'MEDV')

7

Check correlations

Visualize relationships with a heatmap.

plt.figure(figsize=(12,8))
sns.heatmap(df_clean.corr(), annot=True, cmap='coolwarm', fmt=".2f")
plt.show()

8

Split features & target

Separate predictors (X) from label (y).

X = df_clean.drop('MEDV', axis=1)
y = df_clean['MEDV']

9

Train-test split

Reserve 20% of the data for testing.

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

10

Feature scaling

Standardize features to a comparable scale.

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

11

Rebuild as DataFrames

Restore column names after scaling.

(At this point, X_train_scaled, X_test_scaled, y_train, y_test are ready to feed into any regression model (Linear Regression, Random Forest, XGBoost, etc.).)

X_train_scaled = pd.DataFrame(X_train_scaled, columns=X.columns)
X_test_scaled = pd.DataFrame(X_test_scaled, columns=X.columns)

Created by Bijan Krishna Paul