๐ Complete Data Preprocessing in Machine Learning
Theory + Python Code + Output + Outlier Detection Methods
Practical Dataset: Employee Salary & Performance Dataset
๐ Topics Covered
1 Dataset Creation and Loading
๐ Theory
Data preprocessing starts with loading the raw dataset. In this practical we use an Employee dataset instead of the diabetes dataset used in the reference article.
The dataset contains numerical and categorical variables, missing values, duplicate records and intentionally introduced extreme salary values so that different preprocessing techniques can be demonstrated.
| Employee | Age | Experience | Salary | Department | Performance | LeftCompany |
|---|---|---|---|---|---|---|
| 101 | 25 | 2 | 35000 | IT | 72 | 0 |
| 102 | 28 | 4 | 45000 | HR | 80 | 0 |
| 103 | 35 | 10 | 65000 | IT | 88 | 0 |
| 104 | 42 | 16 | 85000 | Finance | 91 | 0 |
| 105 | 29 | 5 | 47000 | IT | 76 | 1 |
| 106 | 51 | 25 | 250000 | Finance | 95 | 0 |
import pandas as pd
import numpy as np
data = {
'EmployeeID': [101,102,103,104,105,106,107,108,109,110],
'Age': [25,28,35,42,29,51,31,np.nan,38,45],
'Experience': [2,4,10,16,5,25,7,6,12,np.nan],
'Salary': [35000,45000,65000,85000,47000,
250000,55000,52000,72000,300000],
'Department': ['IT','HR','IT','Finance','IT',
'Finance','HR','IT',np.nan,'Finance'],
'Performance': [72,80,88,91,76,95,82,79,89,94],
'LeftCompany': [0,0,0,0,1,0,1,0,0,0]
}
df = pd.DataFrame(data)
print(df)
A DataFrame containing 10 employee records with numerical and categorical columns.
The dataset intentionally contains missing values and extreme salary values for demonstration.
2 Data Inspection
๐ Theory
Before modifying data, we should understand its structure, number of records, column names, data types and missing values.
df.head() df.shape df.info() df.dtypes isnull()
print(df.head())
print("Shape:", df.shape)
df.info()
print(df.dtypes)
print(df.isnull().sum())
Shape: (10, 7) Age 1 Experience 1 Salary 0 Department 1 Performance 0 LeftCompany 0 EmployeeID 0
3 Missing Value Detection and Handling
๐ Theory
Missing values occur when information is unavailable. Machine learning algorithms generally require a complete numerical matrix, so missing values must be handled.
Common techniques include:
- Delete rows
- Delete columns
- Mean imputation
- Median imputation
- Mode imputation
- Constant-value imputation
- KNN imputation
- Iterative imputation
3.1 Detect Missing Values
print(df.isnull().sum())
print(df.isnull().mean() * 100)
3.2 Mean Imputation
Mean imputation replaces a missing numerical value with the average of that column.
df['Age'] = df['Age'].fillna(df['Age'].mean())
3.3 Median Imputation
Median is often preferred when the data contains extreme values because it is less affected by outliers.
df['Experience'] = df['Experience'].fillna(
df['Experience'].median()
)
3.4 Mode Imputation
Mode is the most frequently occurring value and is commonly used for categorical columns.
df['Department'] = df['Department'].fillna(
df['Department'].mode()[0]
)
3.5 KNN Imputation
from sklearn.impute import KNNImputer
numeric_columns = [
'Age',
'Experience',
'Salary',
'Performance'
]
imputer = KNNImputer(n_neighbors=3)
df[numeric_columns] = imputer.fit_transform(
df[numeric_columns]
)
After imputation, the numerical and categorical columns no longer contain missing values.
4 Duplicate Data
๐ Theory
Duplicate records can give excessive importance to particular observations and may distort statistical analysis and model training.
print("Duplicate rows:",
df.duplicated().sum())
df = df.drop_duplicates()
print("Shape after removing duplicates:",
df.shape)
Duplicate rows: 0 Shape after removing duplicates: (10, 7)
5 Statistical Summary
๐ Theory
Statistical summaries help us understand the central tendency, dispersion and range of numerical variables.
- Mean
- Standard deviation
- Minimum
- 25th percentile
- Median
- 75th percentile
- Maximum
print(df.describe())
A statistical table containing count, mean, standard deviation, minimum, quartiles and maximum values.
6 Complete Outlier Detection and Handling
⚠️ What is an Outlier?
An outlier is an observation that is unusually far away from the majority of observations.
Example: If most employee salaries are between ₹30,000 and ₹90,000 but one value is ₹300,000, that observation may be an outlier.
An outlier is not automatically an error. It may represent a genuine observation, so the appropriate treatment depends on the data-generating process.
Method 1: IQR Method
The Interquartile Range method identifies observations outside the range defined by Q1 and Q3.
Lower Bound = Q1 - 1.5 × IQR
Upper Bound = Q3 + 1.5 × IQR
Q1 = df['Salary'].quantile(0.25)
Q3 = df['Salary'].quantile(0.75)
IQR = Q3 - Q1
lower = Q1 - 1.5 * IQR
upper = Q3 + 1.5 * IQR
df_iqr = df[
(df['Salary'] >= lower) &
(df['Salary'] <= upper)
]
print(df_iqr)
Method 2: Z-Score Method
Z-score measures how many standard deviations an observation is away from the mean.
A common rule is to investigate observations where |Z| > 3.
from scipy.stats import zscore
z_scores = zscore(df['Salary'])
df_zscore = df[
abs(z_scores) < 3
]
print(df_zscore)
Method 3: Modified Z-Score / MAD
Modified Z-score uses the median and Median Absolute Deviation (MAD). It is useful when the data is highly skewed or contains extreme observations.
median = df['Salary'].median()
MAD = np.median(
np.abs(df['Salary'] - median)
)
modified_z = (
0.6745 *
(df['Salary'] - median) /
MAD
)
df_mad = df[
abs(modified_z) < 3.5
]
print(df_mad)
Method 4: Percentile Method
Extreme observations can be detected by defining acceptable percentile limits, such as the 1st and 99th percentiles.
lower = df['Salary'].quantile(0.01)
upper = df['Salary'].quantile(0.99)
df_percentile = df[
(df['Salary'] >= lower) &
(df['Salary'] <= upper)
]
print(df_percentile)
Method 5: Percentile Capping
Instead of deleting outliers, values outside selected percentiles can be replaced by the boundary values.
lower = df['Salary'].quantile(0.01)
upper = df['Salary'].quantile(0.99)
df['Salary_Capped'] = df['Salary'].clip(
lower=lower,
upper=upper
)
print(
df[['Salary', 'Salary_Capped']]
)
Method 6: Winsorization
Winsorization replaces extreme values with specified percentile limits instead of removing observations.
from scipy.stats.mstats import winsorize
df['Salary_Winsorized'] = winsorize(
df['Salary'],
limits=[0.05, 0.05]
)
print(
df[['Salary',
'Salary_Winsorized']]
)
Method 7: RobustScaler
RobustScaler uses the median and IQR instead of the mean and standard deviation. It is useful when outliers should remain in the dataset but their influence should be reduced.
from sklearn.preprocessing import RobustScaler
robust = RobustScaler()
df['Salary_Robust'] = robust.fit_transform(
df[['Salary']]
)
print(
df[['Salary',
'Salary_Robust']]
)
Method 8: Box Plot for Outlier Visualization
import matplotlib.pyplot as plt
import seaborn as sns
plt.figure(figsize=(8,5))
sns.boxplot(
x=df['Salary']
)
plt.title(
'Salary Distribution and Outliers'
)
plt.xlabel('Salary')
plt.show()
A boxplot is displayed. Individual points beyond the whiskers represent potential outliers.
๐ Comparison of Outlier Techniques
| Method | Basic Idea | Removes Data? | Useful When |
|---|---|---|---|
| IQR | Uses Q1, Q3 and IQR | Yes | Skewed data |
| Z-Score | Uses mean and standard deviation | Yes | Approximately normal data |
| Modified Z-Score | Uses median and MAD | Yes | Highly skewed data |
| Percentile | Uses percentile boundaries | Yes | Known extreme tails |
| Capping | Limits extreme values | No | Want to preserve observations |
| Winsorization | Replaces extreme observations | No | Reduce extreme influence |
| RobustScaler | Scales using median and IQR | No | Outliers are genuine |
7 Correlation Analysis
๐ Theory
Correlation measures the relationship between two numerical variables.
- +1 → Strong positive relationship
- 0 → No linear relationship
- -1 → Strong negative relationship
numeric_df = df.select_dtypes(
include='number'
)
corr = numeric_df.corr()
print(corr)
plt.figure(figsize=(9,6))
sns.heatmap(
corr,
annot=True,
cmap='coolwarm',
fmt='.2f'
)
plt.title('Correlation Heatmap')
plt.show()
A correlation matrix and heatmap showing relationships between Age, Experience, Salary, Performance and the target.
8 Categorical Data Encoding
๐ Theory
Machine learning algorithms generally require numerical input. Categorical text such as IT, HR and Finance therefore needs to be converted into numerical representation.
8.1 One-Hot Encoding
One-hot encoding creates separate binary columns for each category.
df_encoded = pd.get_dummies(
df,
columns=['Department'],
dtype=int
)
print(df_encoded.head())
8.2 Label Encoding
Label encoding assigns an integer to each category.
from sklearn.preprocessing import LabelEncoder
encoder = LabelEncoder()
df['Department_Label'] = encoder.fit_transform(
df['Department']
)
print(df[['Department',
'Department_Label']])
Label encoding can introduce an artificial ordering between categories. For nominal variables such as Department, one-hot encoding is generally more appropriate.
9 Feature and Target Separation
๐ Theory
Features are the input variables used by a machine learning model. The target is the variable that the model attempts to predict.
In this dataset:
- Features → Age, Experience, Salary, Department, Performance
- Target → LeftCompany
X = df.drop(
columns=['LeftCompany']
)
y = df['LeftCompany']
print("Features:")
print(X.head())
print("Target:")
print(y.head())
10 Feature Scaling
๐ Theory
Features can have very different numerical ranges. For example, Age may range from 20–60 while Salary may range from 30,000–300,000.
Scaling prevents large numerical ranges from dominating algorithms that depend on distances or feature magnitude.
10.1 Min-Max Normalization
Min-Max scaling transforms values into a selected range, commonly 0 to 1.
from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler()
X_scaled = scaler.fit_transform(
X_numeric
)
print(X_scaled[:5])
10.2 Standardization
Standardization transforms data so that it has approximately mean 0 and standard deviation 1.
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_standardized = scaler.fit_transform(
X_numeric
)
print(X_standardized[:5])
10.3 Robust Scaling
RobustScaler uses median and IQR, making it useful when extreme values are present.
from sklearn.preprocessing import RobustScaler
scaler = RobustScaler()
X_robust = scaler.fit_transform(
X_numeric
)
print(X_robust[:5])
| Scaler | Transformation | Outlier Sensitivity |
|---|---|---|
| MinMaxScaler | 0 to 1 | High |
| StandardScaler | Mean 0, SD 1 | Medium |
| RobustScaler | Median / IQR | Lower |
11 Train-Test Split
๐ Theory
The dataset should normally be divided into training and testing portions. The model learns from the training data and is evaluated on unseen testing data.
A common split is 80% training and 20% testing.
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
X,
y,
test_size=0.20,
random_state=42
)
print("Training:", X_train.shape)
print("Testing:", X_test.shape)
Scaling and other learned preprocessing operations should generally be fitted using the training data only and then applied to the test data. This prevents information from the test set from influencing training.
12 Complete Preprocessing Pipeline
๐ Overall Workflow
Raw Dataset → Inspect → Missing Values → Duplicate Removal → Outlier Detection → Encoding → Feature/Target Separation → Train/Test Split → Scaling → Machine Learning Model
# ==========================================
# COMPLETE DATA PREPROCESSING
# ==========================================
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import (
OneHotEncoder,
StandardScaler
)
# ------------------------------------------
# Load Dataset
# ------------------------------------------
df = pd.read_csv(
"employee_data.csv"
)
# ------------------------------------------
# Remove duplicates
# ------------------------------------------
df = df.drop_duplicates()
# ------------------------------------------
# Define Features and Target
# ------------------------------------------
X = df.drop(
columns=['LeftCompany']
)
y = df['LeftCompany']
# ------------------------------------------
# Identify columns
# ------------------------------------------
numeric_features = [
'Age',
'Experience',
'Salary',
'Performance'
]
categorical_features = [
'Department'
]
# ------------------------------------------
# Numerical preprocessing
# ------------------------------------------
numeric_pipeline = Pipeline(
steps=[
(
'imputer',
SimpleImputer(
strategy='median'
)
),
(
'scaler',
StandardScaler()
)
]
)
# ------------------------------------------
# Categorical preprocessing
# ------------------------------------------
categorical_pipeline = Pipeline(
steps=[
(
'imputer',
SimpleImputer(
strategy='most_frequent'
)
),
(
'encoder',
OneHotEncoder(
handle_unknown='ignore'
)
)
]
)
# ------------------------------------------
# Combine preprocessing
# ------------------------------------------
preprocessor = ColumnTransformer(
transformers=[
(
'num',
numeric_pipeline,
numeric_features
),
(
'cat',
categorical_pipeline,
categorical_features
)
]
)
# ------------------------------------------
# Train-Test Split
# ------------------------------------------
X_train, X_test, y_train, y_test = train_test_split(
X,
y,
test_size=0.20,
random_state=42
)
# ------------------------------------------
# Fit preprocessing ONLY on training data
# ------------------------------------------
X_train_processed = (
preprocessor.fit_transform(
X_train
)
)
X_test_processed = (
preprocessor.transform(
X_test
)
)
print(
"Training shape:",
X_train_processed.shape
)
print(
"Testing shape:",
X_test_processed.shape
)
Training shape: (8, ...) Testing shape: (2, ...)
The exact number of columns after preprocessing depends on the number of categories produced by one-hot encoding.
๐ฏ Data Preprocessing Summary
1. Data Cleaning
Missing values, duplicate records and inconsistent data are identified and handled.
2. Outlier Handling
IQR, Z-score, MAD, percentile, capping, Winsorization and RobustScaler can be considered depending on the data.
3. Encoding
Categorical variables are converted into numerical representations.
4. Scaling
Min-Max, StandardScaler and RobustScaler bring numerical variables to suitable scales.
5. Correlation
Relationships between numerical variables are examined using correlation matrices and heatmaps.
6. Model Ready Data
After preprocessing, the dataset can be supplied to machine learning algorithms.
⚠️ Important Practical Notes
- Do not automatically delete every outlier. First determine whether it is an error or a legitimate observation.
- IQR is particularly useful for skewed numerical data.
- Z-score works naturally when the distribution is reasonably close to normal.
- MAD/Modified Z-score is more resistant to extreme values.
- Capping and Winsorization preserve the number of observations.
- RobustScaler reduces the influence of outliers without deleting them.
- Fit preprocessing transformations on training data and transform the test data afterward.
No comments:
Post a Comment