Basic Statistical Descriptions of Data

 

Basic Statistical Descriptions of Data

1 Measuring the Central Tendency: Mean, Median, and Mode

CLICK

2 Measuring the Dispersion of Data: Range, Quar tiles, Variance, Standard Deviation, and Interquar tile Range, Summary, Boxplots, and Outliers

68

3 Graphic Displays of Basic Statistical Descriptions of Data -  Quantile Plot, Histograms, catter Plots and Data Correlation

70

—

Measuring Data Similarity and Dissimilarity

 

Measuring Data Similarity and Dissimilarity

1 Data Matrix versus Dissimilarity Matrix

67

2 Proximity Measures for Nominal Attributes

68

3 Proximity Measures for Binary Attributes

70

4 Dissimilarity of Numeric Data: Minkowski Distance

72

5 Proximity Measures for Ordinal Attributes

74

6 Dissimilarity for Attributes of Mixed Types

75

7 Cosine Similarity

—

 Data Warehouses:

Definition of a Data Warehouse

• Core Concept: A centralized repository that collects information from multiple heterogeneous sources under a single, unified schema.
• Location: It typically resides at a single physical or cloud-based site to simplify access.

1. Clean: Erases errors and noise.
2. Integrate: Merges different data streams together.
3. Transform: Formats data into a unified structure.
4. Load: Populates the centralized warehouse.
5. Refresh: Periodically updates the repository to maintain accuracy.

• Subject-Oriented: Organized around major business subjects (e.g., customer, item, supplier) rather than day-to-day operations.
• Historical Perspective: Stored over longer time horizons (e.g., 6 to 12 months) to track trends.
• Summarized Data: Optimizes performance by pre-aggregating data rather than storing every atomic transaction.
• Multidimensional Modeling: Modeled using Data Cubes, where dimensions represent attributes and cells store aggregated values (like counts or sums) for quick querying.

---

Data Mining as a Step in the Process of Knowledge Discovery in Databases (KDD).

 1. Data Cleaning

• Objective: Remove noise and inconsistent data.
• Action: Erase errors, handle missing values, and smooth out discrepancies from the raw datasets.

2. Data Integration

• Objective: Combine multiple heterogeneous data sources.
• Action: Merge databases, data cubes, or flat files into a unified repository.

3. Data Selection

• Objective: Isolate target data.
• Action: Retrieve only the data relevant to the specific analysis task from the integrated database.

4. Data Transformation

• Objective: Consolidate data into appropriate formats.
• Action: Convert and aggregate data into mining-ready structures through operations like summary or normalization.

5. Data Mining

• Objective: Extract data patterns.
• Action: Apply intelligent statistical and algorithmic methods to uncover hidden trends or relationships.

6. Pattern Evaluation

• Objective: Identify truly valuable insights.
• Action: Evaluate discovered patterns against predefined interestingness measures to separate trivial findings from actionable knowledge.

7. Knowledge Presentation

• Objective: Deliver insights to decision-makers.
• Action: Use visualization tools and knowledge representation techniques to clearly communicate the final results to users.

---

MACHINE LEARNING • SCIKIT-LEARN

Scikit-Learn Models

A practical guide to major models, their purpose, when to use them, when not to use them, and quick model-selection rules.

Regression Classification Ensemble Clustering Dimensionality Reduction Neural Networks Quick Selection Memory Trick

How to Choose a Machine Learning Model

There is no single “best” algorithm for every dataset. Model selection depends on the type of target, dataset size, number and type of features, noise, nonlinearity, interpretability, computation time, and the evaluation metric.

Practical rule: Start with a simple baseline, validate correctly, then compare a few appropriate models. Do not select a model only because it is more powerful.

1

Regression Models

Model	What it does	When to use	When NOT to use
Linear Regression	Predicts a continuous value using a linear relationship.	Relationship is approximately linear; excellent baseline.	Strongly nonlinear relationships; severe outliers.
Ridge Regression	Linear regression with L2 regularization.	Many features, multicollinearity, or overfitting in linear models.	When automatic feature removal is specifically required.
Lasso Regression	Linear regression with L1 regularization.	High-dimensional data where automatic feature selection is useful.	When many correlated features are all important; Lasso may select only some.
ElasticNet	Combines L1 and L2 regularization.	Correlated features plus a need for feature selection.	Very simple problems where ordinary regression is sufficient.

Easy rule: Simple linear relationship → Linear Regression   |   Too many/correlated features → Ridge   |   Need feature selection → Lasso   |   Need both → ElasticNet

2

Classification Models

Model	What it does	When to use	When NOT to use
Logistic Regression	Predicts class probabilities.	Binary/multiclass classification; strong baseline; interpretable model.	Highly nonlinear boundaries without feature engineering.
Linear Discriminant Analysis (LDA)	Finds a linear boundary between classes using class distributions.	Small/medium datasets; classes reasonably Gaussian with similar covariance.	Strongly nonlinear boundaries or very different covariance structures.
Gaussian Naive Bayes	Uses Bayes' theorem with a Gaussian assumption for features.	Fast classification; continuous features; small datasets.	Features have complex dependencies that strongly violate the assumption.

Important: For text classification, Multinomial Naive Bayes is usually more natural for word counts or TF-IDF-like nonnegative features. GaussianNB is intended for continuous features that can be modeled approximately by Gaussian distributions.

3

Ensemble Methods

Model	What it does	When to use	When NOT to use
Random Forest	Combines many decision trees.	Tabular data; nonlinear relationships; robust general-purpose model.	Need a very small/interpretable model or strong extrapolation.
Gradient Boosting	Builds trees sequentially, correcting previous errors.	High predictive performance on tabular data.	Extremely large datasets when training time is a major concern.
AdaBoost	Gives greater emphasis to incorrectly classified observations.	Relatively clean classification data with simple weak learners.	Very noisy data or substantial outliers can cause problems.

Random Forest

Many trees are built independently and their predictions are combined.

Gradient Boosting

Trees are built sequentially; each new tree attempts to correct previous errors.

AdaBoost

Later weak learners focus more strongly on observations that earlier learners handled poorly.

Practical tip: Random Forest is often an excellent first choice for structured/tabular data. Gradient Boosting can be a strong choice when you want to push predictive performance.

4

Clustering

Model	What it does	When to use	When NOT to use
K-Means	Divides observations into K groups.	Compact, roughly spherical clusters; K can be specified.	Irregular shapes, substantial noise, or unknown number of clusters.
DBSCAN	Finds dense regions and identifies noise.	Arbitrary-shaped clusters; outlier detection; number of clusters unknown.	Clusters have very different densities or parameter selection is difficult.

Example: Customer Segmentation

Suppose customer data contains Age, Income, and Annual Spending, but no customer-type label.

Customer Data ↓ K-Means ↓ Cluster 1 → Low spending Cluster 2 → Medium spending Cluster 3 → High spending

5

Dimensionality Reduction

Model	What it does	When to use	When NOT to use
PCA	Converts many correlated features into fewer components.	Reduce dimensionality; preprocessing; visualization; remove redundancy.	When original feature interpretability is essential.
t-SNE	Creates a 2D/3D representation emphasizing local neighborhoods.	Visual exploration of high-dimensional data.	As a general preprocessing method or when reliable global distances are required.

PCA — Preprocessing

100 features ↓ PCA ↓ 10 components ↓ ML Model

PCA can be used as a preprocessing technique.

t-SNE — Visualization

100 features ↓ t-SNE ↓ 2D plot ↓ Explore patterns

t-SNE is primarily an exploratory visualization technique.

6

Neural Networks

Model	What it does	When to use	When NOT to use
MLP (Multi-Layer Perceptron)	Neural network for classification and regression.	Nonlinear relationships; medium-sized tabular datasets; neural-network approach.	Very small datasets; interpretability is important; specialized image/text deep-learning tasks.

MLP Structure

Input Features ↓ Input Layer ↓ Hidden Layer 1 ↓ Hidden Layer 2 ↓ Output Layer ↓ Prediction

MLPClassifier → Classification    |    MLPRegressor → Regression

7

Complete Quick Selection Guide

Your Problem	Start With
Simple numerical prediction	Linear Regression
Linear regression + overfitting	Ridge
Need automatic feature selection	Lasso
Need L1 + L2 regularization	ElasticNet
Binary classification	Logistic Regression
Linear classification with distribution assumptions	LDA
Fast probabilistic classification	Naive Bayes
General-purpose tabular ML	Random Forest
High-performance tabular ML	Gradient Boosting
Boosting with simple weak learners	AdaBoost
Known number of compact clusters	K-Means
Irregular clusters + noise	DBSCAN
Reduce features	PCA
Visualize high-dimensional data	t-SNE
Nonlinear neural-network prediction	MLP

8

One-Line Memory Trick

REGRESSION Linear → Ridge → Lasso → ElasticNet CLASSIFICATION Logistic → LDA → Naive Bayes ENSEMBLE Random Forest → Gradient Boosting → AdaBoost CLUSTERING K-Means → DBSCAN DIMENSION REDUCTION PCA → t-SNE NEURAL NETWORK MLP

Final Practical Advice

Model selection should be based on experimentation and validation. Consider dataset size, feature types, scaling requirements, class imbalance, missing values, noise, interpretability, training time, and the correct evaluation metric.

Start Simple

Build a baseline before moving to complex models.

Validate Correctly

Use appropriate train/test splits or cross-validation.

Compare Models

Evaluate suitable alternatives using the same validation strategy.

Avoid Leakage

Keep preprocessing inside a proper pipeline whenever appropriate.

↑

Scikit-Learn Models — Practical ML Reference
Designed for learning, revision and classroom reference.

Deep Learning (Neural Networks)

 Deep Learning (Neural Networks): Multi-layered artificial neural networks capable of parsing complex unstructured inputs like images, speech, and text.

Reinforcement Learning

 Reinforcement Learning: Uses an agent that takes actions in an environment and optimizes behavior based on cumulative rewards or penalties rather than static training datasets. 

Semi-Supervised Learning:

 Semi-Supervised Learning: Trains on a small amount of labeled data combined with a large pool of unlabeled data to save labeling costs. 

Unsupervised Learning Models

 Unsupervised Learning Models

Unsupervised models process data without predefined labels, allowing the system to discover structural properties on its own. 

• Clustering: Groups similar data points together based on shared traits (e.g., K-Means, Hierarchical clustering).
• Dimensionality Reduction: Simplifies complex datasets while retaining vital patterns (e.g., Principal Component Analysis). Specialized and Advanced Models

 

Supervised Learning Models

 Supervised Learning Models

Supervised models use input data paired with known output labels to learn the relationship between them.

• Regression models: Predict continuous numeric values like prices or temperatures (e.g., Linear Regression, Ridge, Lasso).
• Classification models: Predict discrete categories or classes (e.g., Logistic Regression, Support Vector Machines, Naive Bayes).
• Decision Trees & Ensembles: Combine multiple decision rules or trees to boost accuracy (e.g., Random Forest).

 

 Different type of models used in Machine       Learning in Python 

Scikit-Learn (Traditional Machine Learning)

LinearRegression: Basic regression for continuous data.
LogisticRegression: Standard baseline for binary classification.
RandomForestClassifier / RandomForestRegressor: Powerful ensemble tree models.
GradientBoostingClassifier: Sequential tree building for high accuracy.
SVC: Support Vector Classifier for complex boundaries.
KMeans: Unsupervised clustering algorithm.

XGBoost & LightGBM (Gradient Boosting)

XGBClassifier / XGBRegressor: High-performance, scalable gradient boosting.
LGBMClassifier / LGBMRegressor: Fast, leaf-wise tree growth models.

PyTorch (Deep Learning Ecosystem)

torchvision.models.resnet50: Industry standard for image classification.
torchvision.models.vit_b_16: Vision Transformer for advanced image tasks.
torchaudio.models.wave2vec2_model: Architecture for speech processing.
torch.nn.Transformer: Raw building block for sequence-to-sequence tasks.

TensorFlow / Keras (Deep Learning Ecosystem)

keras.applications.ResNet50: Pre-trained deep residual network for vision.
keras.applications.MobileNetV3Large: Lightweight, mobile-optimized vision model.
keras.applications.EfficientNetB0: State-of-the-art scaling for image tasks.
keras.layers.LSTM: Recurrent layer stringed together for text/time-series.

Hugging Face Transformers (NLP & GenAI)

bert-base-uncased: Baseline for text classification and extraction.
roberta-base: Improved, harder-trained version of BERT.
gpt2: Standard starting point for causal text generation.
meta-llama/Meta-Llama-3-8B: State-of-the-art large language model for fine-tuning.
google/vit-base-patch16-224: Vision transformer adapted for Hugging Face pipelines. 

Statsmodels (Statistical & Time Series)

OLS: Ordinary Least Squares regression with deep statistical summaries.
ARIMA: Autoregressive Integrated Moving Average for time-series forecasting.
Logit: Logistic regression specialized for statistical inference.