📊 Data Preprocessing in dataset for Machine Learning using python

📊 Data Preprocessing in Machine Learning

Professional College Notes with Python Programs & Outputs

1. What is Data Preprocessing?

Definition:
Data preprocessing is the process of inspecting, cleaning, transforming and preparing raw data before it is supplied to a machine-learning algorithm.

Real-world data is rarely perfect. A dataset may contain missing values, noisy observations, inconsistent formats, categorical values, redundant features and different numerical scales.

Raw Data

→

Data Cleaning

→

Transformation

→

Feature Selection

→

ML Model

Student Note: The quality of the input data strongly influences the quality of the machine-learning model. Therefore, preprocessing is an important part of the ML workflow.

2. Why Do We Preprocess Data?

Reason	Explanation
Improved Data Quality	Makes data more consistent, accurate and reliable.
Better Model Performance	Helps algorithms identify useful patterns.
Improved Accuracy	Properly prepared data can improve meaningful model evaluation.
Reduced Computational Cost	Removing unnecessary data can reduce processing requirements.
Feature Engineering	Helps identify and construct useful input variables.

Common problems in raw data:

• Missing values
• Outliers and noise
• Inconsistent formats
• Categorical variables
• Different feature scales
• Redundant features
• High-dimensional data

3. Important Python Libraries

🐼

Pandas Data manipulation

🔢

NumPy Numerical computation

🤖

Scikit-Learn ML preprocessing

📈

Matplotlib Visualization

Python Code

import numpy as np import pandas as pd from sklearn.preprocessing import ( MinMaxScaler, StandardScaler, OneHotEncoder, LabelEncoder ) print("Libraries imported successfully")

Output

Libraries imported successfully

4. Handling Missing Values

A missing value occurs when a dataset does not contain a valid value for a particular observation. Examples include: NaN, None, NULL

4.1 Create a Dataset with Missing Values

import pandas as pd import numpy as np data = { "Age":[20,21,np.nan,23,24], "Salary":[25000,np.nan,30000,32000,35000], "Score":[80,85,90,np.nan,95] } df = pd.DataFrame(data) print(df)

Age Salary Score 0 20.0 25000.0 80.0 1 21.0 NaN 85.0 2 NaN 30000.0 90.0 3 23.0 32000.0 NaN 4 24.0 35000.0 95.0

4.2 Detect Missing Values

print(df.isnull().sum())

Age 1 Salary 1 Score 1 dtype: int64

4.3 Remove Rows

df_removed = df.dropna() print(df_removed)

Age Salary Score 0 20.0 25000.0 80.0 4 24.0 35000.0 95.0

When to use: Row removal can be reasonable when only a small amount of information is missing and removing those observations will not introduce serious bias.

4.4 Mean / Median Imputation

df["Age"] = df["Age"].fillna( df["Age"].mean() ) df["Salary"] = df["Salary"].fillna( df["Salary"].median() ) df["Score"] = df["Score"].fillna( df["Score"].mean() ) print(df)

Age Salary Score 0 20.0 25000.0 80.0 1 21.0 30000.0 85.0 2 22.0 30000.0 90.0 3 23.0 32000.0 87.5 4 24.0 35000.0 95.0

Remember: Mean, median or another appropriate imputation strategy should be selected according to the data and problem. Median is often more robust than mean when extreme values are present.

5. Handling Noisy Data

Noise refers to unwanted variation, errors or irrelevant observations that can obscure useful patterns. Two techniques discussed here are: Smoothing and Binning.

5.1 Smoothing Using Moving Average

data = { "Marks":[50,52,49,90,51,53] } df = pd.DataFrame(data) df["Smoothed"] = ( df["Marks"] .rolling(window=3) .mean() ) print(df)

Marks Smoothed 0 50 NaN 1 52 NaN 2 49 50.33 3 90 63.67 4 51 63.33 5 53 64.67

A moving average can reduce short-term variation, but it does not automatically mean that an unusual observation should be deleted. Domain knowledge is important.

5.2 Binning

marks = [35,42,48,55,63,71,88] bins = [0,40,60,80,100] labels = [ "Poor", "Average", "Good", "Excellent" ] result = pd.cut( marks, bins=bins, labels=labels ) print(result)

['Poor', 'Average', 'Average', 'Average', 'Good', 'Good', 'Excellent']

6. Data Transformation — Normalization

Normalization transforms numerical features to a common range, commonly 0 to 1.

Min-Max Normalization

x' = (x − x_min) / (x_max − x_min)

Python Example

from sklearn.preprocessing import MinMaxScaler data = { "Age":[20,25,30,35,40], "Salary":[20000,40000,60000,80000,100000] } df = pd.DataFrame(data) scaler = MinMaxScaler() normalized = scaler.fit_transform(df) normalized_df = pd.DataFrame( normalized, columns=df.columns ) print(normalized_df)

Age Salary 0 0.00 0.00 1 0.25 0.25 2 0.50 0.50 3 0.75 0.75 4 1.00 1.00

Useful for: Distance-based algorithms such as KNN and K-Means can be sensitive to differences in feature scale.

7. Data Transformation — Standardization

Standardization transforms a feature so that it has approximately mean = 0 and standard deviation = 1.

z = (x − μ) / σ

μ = Mean     σ = Standard Deviation

from sklearn.preprocessing import StandardScaler data = { "Age":[20,25,30,35,40], "Salary":[20000,40000,60000,80000,100000] } df = pd.DataFrame(data) scaler = StandardScaler() standardized = scaler.fit_transform(df) result = pd.DataFrame( standardized, columns=df.columns ) print(result.round(2))

Age Salary 0 -1.41 -1.41 1 -0.71 -0.71 2 0.00 0.00 3 0.71 0.71 4 1.41 1.41

Normalization vs Standardization

Normalization → commonly maps values to a fixed range such as 0 to 1.
Standardization → centers data around 0 using the mean and standard deviation.

8. Encoding Categorical Data

Machine-learning algorithms often require numerical input. Categorical text values therefore need an appropriate numerical representation.

8.1 One-Hot Encoding

One-hot encoding creates a separate binary column for each category.

from sklearn.preprocessing import OneHotEncoder data = pd.DataFrame({ "Color":[ "Red", "Blue", "Green", "Blue", "Red" ] }) encoder = OneHotEncoder( sparse_output=False ) encoded = encoder.fit_transform( data[["Color"]] ) result = pd.DataFrame( encoded, columns=encoder.get_feature_names_out( ["Color"] ) ) print(result)

Color_Blue Color_Green Color_Red 0 0.0 0.0 1.0 1 1.0 0.0 0.0 2 0.0 1.0 0.0 3 1.0 0.0 0.0 4 0.0 0.0 1.0

8.2 Label Encoding

from sklearn.preprocessing import LabelEncoder data = pd.DataFrame({ "Color":[ "Red", "Blue", "Green", "Blue", "Red" ] }) encoder = LabelEncoder() data["Color_Encoded"] = ( encoder.fit_transform( data["Color"] ) ) print(data)

Color Color_Encoded 0 Red 2 1 Blue 0 2 Green 1 3 Blue 0 4 Red 2

Important: Do not blindly use label encoding for nominal input features, because assigning numbers can imply an ordering that may not exist. One-hot encoding is often more appropriate for nominal features.

9. Feature Selection Using Correlation

Correlation measures the strength and direction of a relationship between numerical variables.

Correlation coefficient: −1 ≤ r ≤ +1

r ≈ +1 → strong positive relationship
r ≈ −1 → strong negative relationship
r ≈ 0 → weak or no linear relationship

import pandas as pd data = { "StudyHours":[1,2,3,4,5], "Marks":[40,50,60,70,80], "SleepHours":[8,7,6,5,4] } df = pd.DataFrame(data) correlation = df.corr() print( correlation.round(2) )

StudyHours Marks SleepHours StudyHours 1.00 1.00 -1.00 Marks 1.00 1.00 -1.00 SleepHours -1.00 -1.00 1.00

Interpretation: Features with very high correlation with one another may contain redundant information. However, correlation alone should not be used blindly to remove features; domain knowledge and model validation should also be considered.

10. Chi-Square Feature Selection

The Chi-Square test can be used to evaluate the association between categorical features and a categorical target.

from sklearn.feature_selection import chi2 from sklearn.preprocessing import LabelEncoder import pandas as pd df = pd.DataFrame({ "Color":[ "Red", "Blue", "Red", "Green", "Blue" ], "Target":[ 1, 0, 1, 0, 0 ] }) encoder = LabelEncoder() df["Color_Code"] = ( encoder.fit_transform( df["Color"] ) ) X = df[["Color_Code"]] y = df["Target"] scores, pvalues = chi2(X,y) print("Chi-Square:", scores) print("P-Value:", pvalues)

Chi-Square: [0.625] P-Value: [0.429...]

Interpretation: A small p-value can provide evidence of an association between the feature and target. A common significance threshold is 0.05, but the appropriate threshold and interpretation depend on the statistical context.

11. Feature Extraction

Feature extraction transforms the original feature space into a smaller set of new features while attempting to preserve important information.

One important technique is Principal Component Analysis (PCA).

11.1 Principal Component Analysis

PCA transforms correlated features into a smaller number of orthogonal principal components that capture directions of maximum variance.

import pandas as pd from sklearn.decomposition import PCA data = { "A":[1,2,3,4,5], "B":[10,20,30,40,50], "C":[5,4,3,2,1] } df = pd.DataFrame(data) pca = PCA( n_components=2 ) result = pca.fit_transform(df) pca_df = pd.DataFrame( result, columns=[ "PC1", "PC2" ] ) print(pca_df.round(2)) print( "Explained Variance:", pca.explained_variance_ratio_ )

PC1 PC2 0 20.05 0.00 1 10.02 0.00 2 0.00 0.00 3 -10.02 0.00 4 -20.05 0.00 Explained Variance: [1.000... 0.000...]

Why PCA?

• Reduces dimensionality
• Can reduce redundant information
• Can help visualization
• Can reduce computational requirements

12. Complete Data Preprocessing Workflow

1. Collect Data

→

2. Inspect

→

3. Clean

→

4. Encode

→

5. Scale

→

6. Select Features

→

7. Train Model

# ========================================== # COMPLETE BASIC PREPROCESSING PIPELINE # ========================================== import pandas as pd from sklearn.model_selection import train_test_split from sklearn.impute import SimpleImputer from sklearn.preprocessing import ( StandardScaler, OneHotEncoder ) # 1. Load data df = pd.read_csv("student_data.csv") # 2. Separate features and target X = df.drop("Result", axis=1) y = df["Result"] # 3. Train-test split X_train, X_test, y_train, y_test = ( train_test_split( X, y, test_size=0.20, random_state=42 ) ) # 4. Handle missing values imputer = SimpleImputer( strategy="median" ) # Apply only to suitable numerical columns # X_train = imputer.fit_transform(X_train) # X_test = imputer.transform(X_test) # 5. Scale numerical features scaler = StandardScaler() # X_train = scaler.fit_transform(X_train) # X_test = scaler.transform(X_test) print("Preprocessing pipeline completed")

Preprocessing pipeline completed

Very Important for Students: Fit preprocessing transformations on the training data and use the learned transformation on the test data. For example:

scaler.fit(X_train)
then
scaler.transform(X_test)

Do not calculate preprocessing parameters from the complete dataset before splitting, because this can cause data leakage.

13. Quick Revision Table

Technique	Main Purpose	Typical Python Tool
Missing Value Removal	Remove incomplete observations	dropna()
Imputation	Replace missing values	fillna(), SimpleImputer
Smoothing	Reduce short-term noise	rolling()
Binning	Group continuous values	pd.cut()
Normalization	Scale to common range	MinMaxScaler
Standardization	Mean 0, standard deviation 1	StandardScaler
One-Hot Encoding	Categorical → binary columns	OneHotEncoder
Label Encoding	Categories → integer labels	LabelEncoder
Correlation	Identify linear relationships	DataFrame.corr()
Chi-Square	Feature-target association	chi2()
PCA	Dimensionality reduction	PCA()

14. Important Examination Points

Remember these points:

1. Data preprocessing prepares raw data for machine learning.
2. Missing values can be removed or imputed.
3. Mean, median and mode are common simple imputation choices.
4. Normalization commonly scales values to a fixed range such as 0 to 1.
5. Standardization produces features centered around zero with unit variance.
6. One-hot encoding creates binary columns for categories.
7. Label encoding maps categories to integer labels.
8. Correlation can help identify redundant numerical features.
9. Chi-Square can be used for categorical feature selection.
10. PCA is a dimensionality-reduction technique.
11. Preprocessing should be performed carefully to avoid data leakage.

15. Final Summary

Good Machine Learning = Good Data + Appropriate Preprocessing + Suitable Model + Proper Evaluation

Data preprocessing is not simply a collection of Python commands. It is a decision-making stage in which the student must understand the structure, quality and meaning of the data.

The correct preprocessing technique depends on the type of data, the machine-learning algorithm, the problem being solved, and the assumptions behind the technique.

Reference: DZone — Machine Learning With Python: Data Preprocessing Techniques

Machine Learning • Data Preprocessing

College Study Notes • Python • Machine Learning

Created by Bijan Krishna Paul

```