Data Preprocessing

Data Preprocessing Core Framework

Data preprocessing is a foundational phase in data science that transforms raw, real-world data into a clean, integrated, and optimized format suitable for downstream mining algorithms.

3.2

Data Cleaning

Resolves data quality flaws by explicitly handling missing values and smoothing out noisy data structures to minimize system bias.

3.3

Data Integration

Consolidates multi-source schemas, eliminates entity redundancies, tracks value conflicts, and clears duplicate metadata profiles.

3.4

Data Reduction

Compresses volume and dimension footprints via mechanisms like Wavelets, PCA, Sampling, Histograms, and Data Cube Aggregation.

3.5

Transformation & Discretization

Standardizes ranges through data normalization, structural binning, and histogram cluster segmentations into actionable categorical intervals.

_________________

MEAN MEDIAN MODE

 

Statistical Analysis Summary

Central Tendency Metrics

I2.2 Suppose that the data for analysis includes the attribute age. The age values for the data tuples are (in increasing order) 13, 15, 16, 16, 19, 20, 20, 21, 22, 22, 25, 25, 25, 25, 30, 33, 33, 35, 35, 35, 35, 36, 40, 45, 46, 52, 70. (a) What is the mean of the data? What is the median? (b) What is the mode of the data?

Mean

29.96

Median

25

Mode

25, 35

Bimodal (Freq: 4)

Methodology & Calculations

Dataset Context

N = 27 observations (Sorted Age Attributes)

MEAN

Calculated by dividing the sum of all data elements by the total population size (N).

μ = 809 / 27 = 29.9629...

MEDIAN

Identified as the structural midpoint of the ordered dataset. For an odd population size (N = 27), the value aligns with position (N + 1) / 2.

Target Index: 14th position → 25

MODE

Determined by peak structural frequency. The values 25 and 35 share the highest recurring density within the sample profile.

Maximum Frequency: 4 occurrences each

import numpy as np

from scipy import stats

ages = [13, 15, 16, 16, 19, 20, 20, 21, 22, 22, 25, 25, 25, 25, 30, 33, 33, 35, 35, 35, 35, 36, 40, 45, 46, 52, 70]

# Total count

n = len(ages)

# Mean

mean_val = np.mean(ages)

# Median

median_val = np.median(ages)

# Mode

mode_result = stats.mode(ages, keepdims=True)

mode_val = mode_result.mode[0]

mode_count = mode_result.count[0]

# Find all modes just in case it is multimodal

from collections import Counter

counts = Counter(ages)

max_count = max(counts.values())

all_modes = [k for k, v in counts.items() if v == max_count]

print(f"Count: {n}")

print(f"Sum: {sum(ages)}")

print(f"Mean: {mean_val:.2f}")

print(f"Median: {median_val}")

print(f"Modes: {all_modes} with count {max_count}")

Basic Statistical Descriptions of Data

 

Basic Statistical Descriptions of Data

1 Measuring the Central Tendency: Mean, Median, and Mode

CLICK

2 Measuring the Dispersion of Data: Range, Quar tiles, Variance, Standard Deviation, and Interquar tile Range, Summary, Boxplots, and Outliers

68

3 Graphic Displays of Basic Statistical Descriptions of Data -  Quantile Plot, Histograms, catter Plots and Data Correlation

70

—

Measuring Data Similarity and Dissimilarity

 

Measuring Data Similarity and Dissimilarity

1 Data Matrix versus Dissimilarity Matrix

67

2 Proximity Measures for Nominal Attributes

68

3 Proximity Measures for Binary Attributes

70

4 Dissimilarity of Numeric Data: Minkowski Distance

72

5 Proximity Measures for Ordinal Attributes

74

6 Dissimilarity for Attributes of Mixed Types

75

7 Cosine Similarity

—

 Data Warehouses:

Definition of a Data Warehouse

• Core Concept: A centralized repository that collects information from multiple heterogeneous sources under a single, unified schema.
• Location: It typically resides at a single physical or cloud-based site to simplify access.

1. Clean: Erases errors and noise.
2. Integrate: Merges different data streams together.
3. Transform: Formats data into a unified structure.
4. Load: Populates the centralized warehouse.
5. Refresh: Periodically updates the repository to maintain accuracy.

• Subject-Oriented: Organized around major business subjects (e.g., customer, item, supplier) rather than day-to-day operations.
• Historical Perspective: Stored over longer time horizons (e.g., 6 to 12 months) to track trends.
• Summarized Data: Optimizes performance by pre-aggregating data rather than storing every atomic transaction.
• Multidimensional Modeling: Modeled using Data Cubes, where dimensions represent attributes and cells store aggregated values (like counts or sums) for quick querying.

---

Data Mining as a Step in the Process of Knowledge Discovery in Databases (KDD).

 1. Data Cleaning

• Objective: Remove noise and inconsistent data.
• Action: Erase errors, handle missing values, and smooth out discrepancies from the raw datasets.

2. Data Integration

• Objective: Combine multiple heterogeneous data sources.
• Action: Merge databases, data cubes, or flat files into a unified repository.

3. Data Selection

• Objective: Isolate target data.
• Action: Retrieve only the data relevant to the specific analysis task from the integrated database.

4. Data Transformation

• Objective: Consolidate data into appropriate formats.
• Action: Convert and aggregate data into mining-ready structures through operations like summary or normalization.

5. Data Mining

• Objective: Extract data patterns.
• Action: Apply intelligent statistical and algorithmic methods to uncover hidden trends or relationships.

6. Pattern Evaluation

• Objective: Identify truly valuable insights.
• Action: Evaluate discovered patterns against predefined interestingness measures to separate trivial findings from actionable knowledge.

7. Knowledge Presentation

• Objective: Deliver insights to decision-makers.
• Action: Use visualization tools and knowledge representation techniques to clearly communicate the final results to users.

---

MACHINE LEARNING • SCIKIT-LEARN

Scikit-Learn Models

A practical guide to major models, their purpose, when to use them, when not to use them, and quick model-selection rules.

Regression Classification Ensemble Clustering Dimensionality Reduction Neural Networks Quick Selection Memory Trick

How to Choose a Machine Learning Model

There is no single “best” algorithm for every dataset. Model selection depends on the type of target, dataset size, number and type of features, noise, nonlinearity, interpretability, computation time, and the evaluation metric.

Practical rule: Start with a simple baseline, validate correctly, then compare a few appropriate models. Do not select a model only because it is more powerful.

1

Regression Models

Model	What it does	When to use	When NOT to use
Linear Regression	Predicts a continuous value using a linear relationship.	Relationship is approximately linear; excellent baseline.	Strongly nonlinear relationships; severe outliers.
Ridge Regression	Linear regression with L2 regularization.	Many features, multicollinearity, or overfitting in linear models.	When automatic feature removal is specifically required.
Lasso Regression	Linear regression with L1 regularization.	High-dimensional data where automatic feature selection is useful.	When many correlated features are all important; Lasso may select only some.
ElasticNet	Combines L1 and L2 regularization.	Correlated features plus a need for feature selection.	Very simple problems where ordinary regression is sufficient.

Easy rule: Simple linear relationship → Linear Regression   |   Too many/correlated features → Ridge   |   Need feature selection → Lasso   |   Need both → ElasticNet

2

Classification Models

Model	What it does	When to use	When NOT to use
Logistic Regression	Predicts class probabilities.	Binary/multiclass classification; strong baseline; interpretable model.	Highly nonlinear boundaries without feature engineering.
Linear Discriminant Analysis (LDA)	Finds a linear boundary between classes using class distributions.	Small/medium datasets; classes reasonably Gaussian with similar covariance.	Strongly nonlinear boundaries or very different covariance structures.
Gaussian Naive Bayes	Uses Bayes' theorem with a Gaussian assumption for features.	Fast classification; continuous features; small datasets.	Features have complex dependencies that strongly violate the assumption.

Important: For text classification, Multinomial Naive Bayes is usually more natural for word counts or TF-IDF-like nonnegative features. GaussianNB is intended for continuous features that can be modeled approximately by Gaussian distributions.

3

Ensemble Methods

Model	What it does	When to use	When NOT to use
Random Forest	Combines many decision trees.	Tabular data; nonlinear relationships; robust general-purpose model.	Need a very small/interpretable model or strong extrapolation.
Gradient Boosting	Builds trees sequentially, correcting previous errors.	High predictive performance on tabular data.	Extremely large datasets when training time is a major concern.
AdaBoost	Gives greater emphasis to incorrectly classified observations.	Relatively clean classification data with simple weak learners.	Very noisy data or substantial outliers can cause problems.

Random Forest

Many trees are built independently and their predictions are combined.

Gradient Boosting

Trees are built sequentially; each new tree attempts to correct previous errors.

AdaBoost

Later weak learners focus more strongly on observations that earlier learners handled poorly.

Practical tip: Random Forest is often an excellent first choice for structured/tabular data. Gradient Boosting can be a strong choice when you want to push predictive performance.

4

Clustering

Model	What it does	When to use	When NOT to use
K-Means	Divides observations into K groups.	Compact, roughly spherical clusters; K can be specified.	Irregular shapes, substantial noise, or unknown number of clusters.
DBSCAN	Finds dense regions and identifies noise.	Arbitrary-shaped clusters; outlier detection; number of clusters unknown.	Clusters have very different densities or parameter selection is difficult.

Example: Customer Segmentation

Suppose customer data contains Age, Income, and Annual Spending, but no customer-type label.

Customer Data ↓ K-Means ↓ Cluster 1 → Low spending Cluster 2 → Medium spending Cluster 3 → High spending

5

Dimensionality Reduction

Model	What it does	When to use	When NOT to use
PCA	Converts many correlated features into fewer components.	Reduce dimensionality; preprocessing; visualization; remove redundancy.	When original feature interpretability is essential.
t-SNE	Creates a 2D/3D representation emphasizing local neighborhoods.	Visual exploration of high-dimensional data.	As a general preprocessing method or when reliable global distances are required.

PCA — Preprocessing

100 features ↓ PCA ↓ 10 components ↓ ML Model

PCA can be used as a preprocessing technique.

t-SNE — Visualization

100 features ↓ t-SNE ↓ 2D plot ↓ Explore patterns

t-SNE is primarily an exploratory visualization technique.

6

Neural Networks

Model	What it does	When to use	When NOT to use
MLP (Multi-Layer Perceptron)	Neural network for classification and regression.	Nonlinear relationships; medium-sized tabular datasets; neural-network approach.	Very small datasets; interpretability is important; specialized image/text deep-learning tasks.

MLP Structure

Input Features ↓ Input Layer ↓ Hidden Layer 1 ↓ Hidden Layer 2 ↓ Output Layer ↓ Prediction

MLPClassifier → Classification    |    MLPRegressor → Regression

7

Complete Quick Selection Guide

Your Problem	Start With
Simple numerical prediction	Linear Regression
Linear regression + overfitting	Ridge
Need automatic feature selection	Lasso
Need L1 + L2 regularization	ElasticNet
Binary classification	Logistic Regression
Linear classification with distribution assumptions	LDA
Fast probabilistic classification	Naive Bayes
General-purpose tabular ML	Random Forest
High-performance tabular ML	Gradient Boosting
Boosting with simple weak learners	AdaBoost
Known number of compact clusters	K-Means
Irregular clusters + noise	DBSCAN
Reduce features	PCA
Visualize high-dimensional data	t-SNE
Nonlinear neural-network prediction	MLP

8

One-Line Memory Trick

REGRESSION Linear → Ridge → Lasso → ElasticNet CLASSIFICATION Logistic → LDA → Naive Bayes ENSEMBLE Random Forest → Gradient Boosting → AdaBoost CLUSTERING K-Means → DBSCAN DIMENSION REDUCTION PCA → t-SNE NEURAL NETWORK MLP

Final Practical Advice

Model selection should be based on experimentation and validation. Consider dataset size, feature types, scaling requirements, class imbalance, missing values, noise, interpretability, training time, and the correct evaluation metric.

Start Simple

Build a baseline before moving to complex models.

Validate Correctly

Use appropriate train/test splits or cross-validation.

Compare Models

Evaluate suitable alternatives using the same validation strategy.

Avoid Leakage

Keep preprocessing inside a proper pipeline whenever appropriate.

↑

Scikit-Learn Models — Practical ML Reference
Designed for learning, revision and classroom reference.