📊 Data Preprocessing in Machine Learning
Professional College Notes with Python Programs & Outputs
1. What is Data Preprocessing?
Definition:
Data preprocessing is the process of inspecting, cleaning, transforming and preparing raw data before it is supplied to a machine-learning algorithm.
Data preprocessing is the process of inspecting, cleaning, transforming and preparing raw data before it is supplied to a machine-learning algorithm.
Real-world data is rarely perfect. A dataset may contain missing values, noisy observations, inconsistent formats, categorical values, redundant features and different numerical scales.
Raw Data
→
Data Cleaning
→
Transformation
→
Feature Selection
→
ML Model
Student Note:
The quality of the input data strongly influences the quality
of the machine-learning model. Therefore, preprocessing is an
important part of the ML workflow.
2. Why Do We Preprocess Data?
| Reason | Explanation |
|---|---|
| Improved Data Quality | Makes data more consistent, accurate and reliable. |
| Better Model Performance | Helps algorithms identify useful patterns. |
| Improved Accuracy | Properly prepared data can improve meaningful model evaluation. |
| Reduced Computational Cost | Removing unnecessary data can reduce processing requirements. |
| Feature Engineering | Helps identify and construct useful input variables. |
Common problems in raw data:
- Missing values
- Outliers and noise
- Inconsistent formats
- Categorical variables
- Different feature scales
- Redundant features
- High-dimensional data
3. Important Python Libraries
Pandas
Data manipulation
NumPy
Numerical computation
Scikit-Learn
ML preprocessing
Matplotlib
Visualization
Python Code
import numpy as np
import pandas as pd
from sklearn.preprocessing import (
MinMaxScaler,
StandardScaler,
OneHotEncoder,
LabelEncoder
)
print("Libraries imported successfully")
Output
Libraries imported successfully
4. Handling Missing Values
A missing value occurs when a dataset does not contain a valid
value for a particular observation.
Examples include:
NaN, None, NULL
4.1 Create a Dataset with Missing Values
import pandas as pd
import numpy as np
data = {
"Age":[20,21,np.nan,23,24],
"Salary":[25000,np.nan,30000,32000,35000],
"Score":[80,85,90,np.nan,95]
}
df = pd.DataFrame(data)
print(df)
Age Salary Score
0 20.0 25000.0 80.0
1 21.0 NaN 85.0
2 NaN 30000.0 90.0
3 23.0 32000.0 NaN
4 24.0 35000.0 95.0
4.2 Detect Missing Values
print(df.isnull().sum())
Age 1
Salary 1
Score 1
dtype: int64
4.3 Remove Rows
df_removed = df.dropna()
print(df_removed)
Age Salary Score
0 20.0 25000.0 80.0
4 24.0 35000.0 95.0
When to use:
Row removal can be reasonable when only a small amount of
information is missing and removing those observations will
not introduce serious bias.
4.4 Mean / Median Imputation
df["Age"] = df["Age"].fillna(
df["Age"].mean()
)
df["Salary"] = df["Salary"].fillna(
df["Salary"].median()
)
df["Score"] = df["Score"].fillna(
df["Score"].mean()
)
print(df)
Age Salary Score
0 20.0 25000.0 80.0
1 21.0 30000.0 85.0
2 22.0 30000.0 90.0
3 23.0 32000.0 87.5
4 24.0 35000.0 95.0
Remember:
Mean, median or another appropriate imputation strategy should
be selected according to the data and problem. Median is often
more robust than mean when extreme values are present.
5. Handling Noisy Data
Noise refers to unwanted variation, errors or
irrelevant observations that can obscure useful patterns.
Two techniques discussed here are:
Smoothing and Binning.
5.1 Smoothing Using Moving Average
data = {
"Marks":[50,52,49,90,51,53]
}
df = pd.DataFrame(data)
df["Smoothed"] = (
df["Marks"]
.rolling(window=3)
.mean()
)
print(df)
Marks Smoothed
0 50 NaN
1 52 NaN
2 49 50.33
3 90 63.67
4 51 63.33
5 53 64.67
A moving average can reduce short-term variation, but it does
not automatically mean that an unusual observation should be
deleted. Domain knowledge is important.
5.2 Binning
marks = [35,42,48,55,63,71,88]
bins = [0,40,60,80,100]
labels = [
"Poor",
"Average",
"Good",
"Excellent"
]
result = pd.cut(
marks,
bins=bins,
labels=labels
)
print(result)
['Poor', 'Average', 'Average',
'Average', 'Good', 'Good', 'Excellent']
6. Data Transformation — Normalization
Normalization transforms numerical features to a common range,
commonly 0 to 1.
Min-Max Normalization
x' = (x − xmin) / (xmax − xmin)
x' = (x − xmin) / (xmax − xmin)
Python Example
from sklearn.preprocessing import MinMaxScaler
data = {
"Age":[20,25,30,35,40],
"Salary":[20000,40000,60000,80000,100000]
}
df = pd.DataFrame(data)
scaler = MinMaxScaler()
normalized = scaler.fit_transform(df)
normalized_df = pd.DataFrame(
normalized,
columns=df.columns
)
print(normalized_df)
Age Salary
0 0.00 0.00
1 0.25 0.25
2 0.50 0.50
3 0.75 0.75
4 1.00 1.00
Useful for:
Distance-based algorithms such as KNN and K-Means can be
sensitive to differences in feature scale.
7. Data Transformation — Standardization
Standardization transforms a feature so that it has approximately
mean = 0 and standard deviation = 1.
z = (x − μ) / σ
μ = Mean σ = Standard Deviation
μ = Mean σ = Standard Deviation
from sklearn.preprocessing import StandardScaler
data = {
"Age":[20,25,30,35,40],
"Salary":[20000,40000,60000,80000,100000]
}
df = pd.DataFrame(data)
scaler = StandardScaler()
standardized = scaler.fit_transform(df)
result = pd.DataFrame(
standardized,
columns=df.columns
)
print(result.round(2))
Age Salary
0 -1.41 -1.41
1 -0.71 -0.71
2 0.00 0.00
3 0.71 0.71
4 1.41 1.41
Normalization vs Standardization
Normalization → commonly maps values to a fixed range such as 0 to 1.
Standardization → centers data around 0 using the mean and standard deviation.
Normalization → commonly maps values to a fixed range such as 0 to 1.
Standardization → centers data around 0 using the mean and standard deviation.
8. Encoding Categorical Data
Machine-learning algorithms often require numerical input.
Categorical text values therefore need an appropriate numerical
representation.
8.1 One-Hot Encoding
One-hot encoding creates a separate binary column for each category.
from sklearn.preprocessing import OneHotEncoder
data = pd.DataFrame({
"Color":[
"Red",
"Blue",
"Green",
"Blue",
"Red"
]
})
encoder = OneHotEncoder(
sparse_output=False
)
encoded = encoder.fit_transform(
data[["Color"]]
)
result = pd.DataFrame(
encoded,
columns=encoder.get_feature_names_out(
["Color"]
)
)
print(result)
Color_Blue Color_Green Color_Red
0 0.0 0.0 1.0
1 1.0 0.0 0.0
2 0.0 1.0 0.0
3 1.0 0.0 0.0
4 0.0 0.0 1.0
8.2 Label Encoding
from sklearn.preprocessing import LabelEncoder
data = pd.DataFrame({
"Color":[
"Red",
"Blue",
"Green",
"Blue",
"Red"
]
})
encoder = LabelEncoder()
data["Color_Encoded"] = (
encoder.fit_transform(
data["Color"]
)
)
print(data)
Color Color_Encoded
0 Red 2
1 Blue 0
2 Green 1
3 Blue 0
4 Red 2
Important:
Do not blindly use label encoding for nominal input features,
because assigning numbers can imply an ordering that may not
exist. One-hot encoding is often more appropriate for nominal
features.
9. Feature Selection Using Correlation
Correlation measures the strength and direction of a relationship
between numerical variables.
Correlation coefficient:
−1 ≤ r ≤ +1
r ≈ +1 → strong positive relationship
r ≈ −1 → strong negative relationship
r ≈ 0 → weak or no linear relationship
r ≈ +1 → strong positive relationship
r ≈ −1 → strong negative relationship
r ≈ 0 → weak or no linear relationship
import pandas as pd
data = {
"StudyHours":[1,2,3,4,5],
"Marks":[40,50,60,70,80],
"SleepHours":[8,7,6,5,4]
}
df = pd.DataFrame(data)
correlation = df.corr()
print(
correlation.round(2)
)
StudyHours Marks SleepHours
StudyHours 1.00 1.00 -1.00
Marks 1.00 1.00 -1.00
SleepHours -1.00 -1.00 1.00
Interpretation:
Features with very high correlation with one another may contain
redundant information. However, correlation alone should not be
used blindly to remove features; domain knowledge and model
validation should also be considered.
10. Chi-Square Feature Selection
The Chi-Square test can be used to evaluate the association
between categorical features and a categorical target.
from sklearn.feature_selection import chi2
from sklearn.preprocessing import LabelEncoder
import pandas as pd
df = pd.DataFrame({
"Color":[
"Red",
"Blue",
"Red",
"Green",
"Blue"
],
"Target":[
1,
0,
1,
0,
0
]
})
encoder = LabelEncoder()
df["Color_Code"] = (
encoder.fit_transform(
df["Color"]
)
)
X = df[["Color_Code"]]
y = df["Target"]
scores, pvalues = chi2(X,y)
print("Chi-Square:", scores)
print("P-Value:", pvalues)
Chi-Square: [0.625]
P-Value: [0.429...]
Interpretation:
A small p-value can provide evidence of an association between
the feature and target. A common significance threshold is
0.05, but the appropriate threshold and interpretation depend
on the statistical context.
11. Feature Extraction
Feature extraction transforms the original feature space into
a smaller set of new features while attempting to preserve
important information.
One important technique is Principal Component Analysis (PCA).
11.1 Principal Component Analysis
PCA transforms correlated features into a smaller number of
orthogonal principal components that capture directions of
maximum variance.
import pandas as pd
from sklearn.decomposition import PCA
data = {
"A":[1,2,3,4,5],
"B":[10,20,30,40,50],
"C":[5,4,3,2,1]
}
df = pd.DataFrame(data)
pca = PCA(
n_components=2
)
result = pca.fit_transform(df)
pca_df = pd.DataFrame(
result,
columns=[
"PC1",
"PC2"
]
)
print(pca_df.round(2))
print(
"Explained Variance:",
pca.explained_variance_ratio_
)
PC1 PC2
0 20.05 0.00
1 10.02 0.00
2 0.00 0.00
3 -10.02 0.00
4 -20.05 0.00
Explained Variance:
[1.000... 0.000...]
Why PCA?
- Reduces dimensionality
- Can reduce redundant information
- Can help visualization
- Can reduce computational requirements
12. Complete Data Preprocessing Workflow
1. Collect Data
→
2. Inspect
→
3. Clean
→
4. Encode
→
5. Scale
→
6. Select Features
→
7. Train Model
# ==========================================
# COMPLETE BASIC PREPROCESSING PIPELINE
# ==========================================
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import (
StandardScaler,
OneHotEncoder
)
# 1. Load data
df = pd.read_csv("student_data.csv")
# 2. Separate features and target
X = df.drop("Result", axis=1)
y = df["Result"]
# 3. Train-test split
X_train, X_test, y_train, y_test = (
train_test_split(
X,
y,
test_size=0.20,
random_state=42
)
)
# 4. Handle missing values
imputer = SimpleImputer(
strategy="median"
)
# Apply only to suitable numerical columns
# X_train = imputer.fit_transform(X_train)
# X_test = imputer.transform(X_test)
# 5. Scale numerical features
scaler = StandardScaler()
# X_train = scaler.fit_transform(X_train)
# X_test = scaler.transform(X_test)
print("Preprocessing pipeline completed")
Preprocessing pipeline completed
Very Important for Students:
Fit preprocessing transformations on the training data and use
the learned transformation on the test data.
For example:
then
Do not calculate preprocessing parameters from the complete dataset before splitting, because this can cause data leakage.
scaler.fit(X_train)
then
scaler.transform(X_test)
Do not calculate preprocessing parameters from the complete dataset before splitting, because this can cause data leakage.
13. Quick Revision Table
| Technique | Main Purpose | Typical Python Tool |
|---|---|---|
| Missing Value Removal | Remove incomplete observations | dropna() |
| Imputation | Replace missing values | fillna(), SimpleImputer |
| Smoothing | Reduce short-term noise | rolling() |
| Binning | Group continuous values | pd.cut() |
| Normalization | Scale to common range | MinMaxScaler |
| Standardization | Mean 0, standard deviation 1 | StandardScaler |
| One-Hot Encoding | Categorical → binary columns | OneHotEncoder |
| Label Encoding | Categories → integer labels | LabelEncoder |
| Correlation | Identify linear relationships | DataFrame.corr() |
| Chi-Square | Feature-target association | chi2() |
| PCA | Dimensionality reduction | PCA() |
14. Important Examination Points
Remember these points:
- Data preprocessing prepares raw data for machine learning.
- Missing values can be removed or imputed.
- Mean, median and mode are common simple imputation choices.
- Normalization commonly scales values to a fixed range such as 0 to 1.
- Standardization produces features centered around zero with unit variance.
- One-hot encoding creates binary columns for categories.
- Label encoding maps categories to integer labels.
- Correlation can help identify redundant numerical features.
- Chi-Square can be used for categorical feature selection.
- PCA is a dimensionality-reduction technique.
- Preprocessing should be performed carefully to avoid data leakage.
15. Final Summary
Good Machine Learning = Good Data + Appropriate Preprocessing
+ Suitable Model + Proper Evaluation
Data preprocessing is not simply a collection of Python commands. It is a decision-making stage in which the student must understand the structure, quality and meaning of the data.
The correct preprocessing technique depends on the type of data, the machine-learning algorithm, the problem being solved, and the assumptions behind the technique.
Reference:
DZone — Machine Learning With Python: Data Preprocessing Techniques