PRINCIPAL COMPONENT ANALYSIS (PCA)

PRINCIPAL COMPONENT ANALYSIS (PCA)

Four-Feature Numerical Example of Dimensionality Reduction

1. Problem Statement

We have marks of five students in four subjects: Mathematics, Physics, Chemistry and Biology. PCA is used to transform the four original features into principal components while preserving maximum variance.

2. Original Dataset

Student	Mathematics	Physics	Chemistry	Biology
A	2	1	5	7
B	4	3	4	6
C	6	5	8	9
D	8	7	6	8
E	10	9	10	11

3. PCA Calculation Process

PCA is performed through the following steps:
1. Calculate the mean
2. Center the data
3. Calculate the covariance matrix
4. Calculate eigenvalues
5. Calculate eigenvectors
6. Sort principal components
7. Calculate explained variance
8. Project data onto the principal components
9. Reduce the dimensionality

STEP 1 Calculate the Mean

Mean of Mathematics

Mean = (2 + 4 + 6 + 8 + 10) / 5 = 6

Mean of Physics

Mean = (1 + 3 + 5 + 7 + 9) / 5 = 5

Mean of Chemistry

Mean = (5 + 4 + 8 + 6 + 10) / 5 = 6.6

Mean of Biology

Mean = (7 + 6 + 9 + 8 + 11) / 5 = 8.2

Mean Vector:
Mathematics = 6   |   Physics = 5   |   Chemistry = 6.6   |   Biology = 8.2

STEP 2 Center the Data

The mean of each feature is subtracted from every corresponding value.

X_centered = X − Mean

Student	Math − 6	Physics − 5	Chemistry − 6.6	Biology − 8.2
A	-4	-4	-1.6	-1.2
B	-2	-2	-2.6	-2.2
C	0	0	1.4	0.8
D	2	2	-0.6	-0.2
E	4	4	3.4	2.8

X_c = [ -4   -4   -1.6   -1.2 ]
[ -2   -2   -2.6   -2.2 ]
[ 0    0    1.4    0.8 ]
[ 2    2   -0.6   -0.2 ]
[ 4    4    3.4    2.8 ]

STEP 3 Calculate the Covariance Matrix

C = (1 / (n − 1)) X_c^TX_c

Number of observations: n = 5
Therefore: n − 1 = 4

Variance and Covariance Values

	Mathematics	Physics	Chemistry	Biology
Mathematics	10	10	6	5
Physics	10	10	6	5
Chemistry	6	6	5.8	4.6
Biology	5	5	4.6	3.7

Final 4 × 4 Covariance Matrix:

C = [ 10   10   6   5 ]
[ 10   10   6   5 ]
[ 6    6   5.8   4.6 ]
[ 5    5   4.6   3.7 ]

STEP 4 Calculate the Eigenvalues

Eigenvalues are obtained by solving the characteristic equation:

|C − λI| = 0

| 10−λ   10    6    5 |
| 10   10−λ   6    5 |
| 6     6   5.8−λ   4.6 |
| 5     5    4.6   3.7−λ |

λ₁ ≈ 26.970

λ₂ ≈ 2.507

λ₃ ≈ 0.024

λ₄ = 0

Eigenvalues:
λ₁ ≈ 26.970   |   λ₂ ≈ 2.507   |   λ₃ ≈ 0.024   |   λ₄ = 0

STEP 5 Calculate the Eigenvectors

Each eigenvector represents the direction of a principal component. The eigenvectors are normalized to have unit length.

Eigenvector for PC1

PC1 ≈ [ 0.598 ]
[ 0.598 ]
[ 0.413 ]
[ 0.339 ]

Eigenvector for PC2

PC2 ≈ [ 0.377 ]
[ 0.377 ]
[-0.692 ]
[-0.487 ]

Eigenvector for PC3

PC3 ≈ [ 0.024 ]
[ 0.024 ]
[-0.592 ]
[ 0.805 ]

Eigenvector for PC4

PC4 ≈ [ 0.707 ]
[-0.707]
[ 0.000 ]
[ 0.000 ]

The signs of eigenvectors can be reversed without changing the PCA. Therefore, an eigenvector and its negative represent the same principal component direction.

STEP 6 Sort the Principal Components

Principal Component	Eigenvalue	Rank
PC1	26.970	1
PC2	2.507	2
PC3	0.024	3
PC4	0	4

STEP 7 Calculate Explained Variance

Total Variance = 26.970 + 2.507 + 0.024 + 0 ≈ 29.500

Explained Variance of PC1 = (26.970 / 29.500) × 100 ≈ 91.42%

Explained Variance of PC2 = (2.507 / 29.500) × 100 ≈ 8.50%

Explained Variance of PC3 = (0.024 / 29.500) × 100 ≈ 0.08%

Explained Variance of PC4 = (0 / 29.500) × 100 = 0%

Component	Eigenvalue	Explained Variance	Cumulative Variance
PC1	26.970	91.42%	91.42%
PC2	2.507	8.50%	99.92%
PC3	0.024	0.08%	100.00%
PC4	0	0%	100.00%

Important Result:
PC1 alone preserves approximately 91.42% of the total variance.
PC1 + PC2 together preserve approximately 99.92% of the total variance.

STEP 8 Project Data onto PC1

To reduce the four original features to one dimension, we multiply each centered observation by the PC1 eigenvector.

PC1 Score = X_centered × PC1 Eigenvector

Student A

PC1 = (-4 × 0.598) + (-4 × 0.598) + (-1.6 × 0.413) + (-1.2 × 0.339) ≈ -5.850

Student B

PC1 = (-2 × 0.598) + (-2 × 0.598) + (-2.6 × 0.413) + (-2.2 × 0.339) ≈ -4.209

Student C

PC1 = (0 × 0.598) + (0 × 0.598) + (1.4 × 0.413) + (0.8 × 0.339) ≈ 0.848

Student D

PC1 = (2 × 0.598) + (2 × 0.598) + (-0.6 × 0.413) + (-0.2 × 0.339) ≈ 2.077

Student E

PC1 = (4 × 0.598) + (4 × 0.598) + (3.4 × 0.413) + (2.8 × 0.339) ≈ 7.135

STEP 9 Final Dataset with PCA Value

The original four subject columns are retained for comparison, and the calculated PC1 value is added as the reduced feature.

Student	Mathematics	Physics	Chemistry	Biology	PC1
A	2	1	5	7	-5.850
B	4	3	4	6	-4.209
C	6	5	8	9	0.848
D	8	7	6	8	2.077
E	10	9	10	11	7.135

10. Original Dataset vs PCA Dataset

Original Dataset	After PCA
Mathematics	PC1
Physics
Chemistry
Biology

Dimensionality Reduction:

Original Dataset = 4 features
Reduced Dataset = 1 principal component

PC1 preserves approximately 91.42% of the total variance.

11. Alternative Reduction Using PC1 + PC2

If we require approximately 99% of the information instead of 91%, we can retain the first two principal components.

Components Retained	Variance Preserved	Dimensions
PC1	91.42%	1
PC1 + PC2	99.92%	2

Therefore, PC1 + PC2 can reduce the original 4-dimensional dataset to only 2 dimensions while preserving approximately 99.92% of the information.

12. Final PCA Result

PCA DIMENSIONALITY REDUCTION

4 Original Features → 1 Principal Component

PC1 ≈ 91.42%

PC1 contains most of the variation present in Mathematics, Physics, Chemistry and Biology.

13. Conclusion

PCA transforms the original four subject features into four new orthogonal principal components.

PC1 captures approximately 91.42% of the total variance.

PC2 captures approximately 8.50% of the total variance.

PC3 captures approximately 0.08% of the total variance.

PC4 captures approximately 0% of the total variance.

If approximately 91% information is sufficient, the original four-dimensional dataset can be reduced to one dimension using PC1.

If approximately 99% information is required, retain PC1 and PC2.

14. PCA Summary

Step	Result
Number of Original Features	4
Number of Students	5
Covariance Matrix	4 × 4
PC1 Eigenvalue	26.970
PC2 Eigenvalue	2.507
PC3 Eigenvalue	0.024
PC4 Eigenvalue	0
PC1 Variance	91.42%
PC2 Variance	8.50%
PC3 Variance	0.08%
PC4 Variance	0%
PC1 + PC2 Variance	99.92%
Reduced Dimension using PC1	1
Reduced Dimension using PC1 + PC2	2

```