🧠 SVM Kernel Mathematics & Kernel Matrices

A Support Vector Machine (SVM) uses a kernel function to calculate the similarity between two observations. Instead of explicitly transforming the data into a higher-dimensional feature space, the kernel computes:

K(x,z) = φ(x)ᵀφ(z)

For n observations, the kernel calculations produce an n × n Gram matrix.

1️⃣ Five-Point Dataset

Point	x₁	x₂	Class y
A	1	2	-1
B	2	1	-1
C	4	5	+1
D	5	4	+1
E	6	6	+1

Example Points

We will frequently calculate the kernel between:

A = (1,2)
C = (4,5)

Dot product:

A · C = (1)(4) + (2)(5)
= 4 + 10
= 14

Squared Euclidean distance:

||A-C||² = (1-4)² + (2-5)²
= 9 + 9
= 18

L1 distance:

||A-C||₁ = |1-4| + |2-5|
= 3 + 3
= 6

2️⃣ Linear Kernel

K(x,z) = xᵀz

For A and C:

K(A,C) = (1)(4) + (2)(5)
= 4 + 10
= 14

5 × 5 Linear Kernel Matrix

[

5	4	14	13	18
4	5	13	14	18
14	13	41	40	54
13	14	40	41	54
18	18	54	54	72

]

3️⃣ Polynomial Kernel

K(x,z) = (γxᵀz + r)ᵈ

Take:

γ = 1    r = 1    d = 2

K(A,C)
= (1 × 14 + 1)²
= 15²
= 225

Polynomial Degree 2 Matrix

[

36	25	225	196	361
25	36	196	225	361
225	196	1764	1681	3025
196	225	1681	1764	3025
361	361	3025	3025	5329

]

4️⃣ RBF / Gaussian Kernel

K(x,z) = exp(-γ||x-z||²)

Take:

γ = 0.1

For A and C:

||A-C||² = 18
K(A,C) = e^-0.1(18)
= e^-1.8
≈ 0.1653

RBF Kernel Matrix

[

1.0000	0.8187	0.1653	0.1353	0.0166
0.8187	1.0000	0.1353	0.1653	0.0166
0.1653	0.1353	1.0000	0.8187	0.6065
0.1353	0.1653	0.8187	1.0000	0.6065
0.0166	0.0166	0.6065	0.6065	1.0000

]

5️⃣ Sigmoid Kernel

K(x,z) = tanh(γxᵀz + r)

Take:

γ = 0.1    r = 0

K(A,C)
= tanh(0.1 × 14)
= tanh(1.4)
≈ 0.8854

Sigmoid Kernel Matrix

[

0.4621	0.3799	0.8854	0.8617	0.9468
0.3799	0.4621	0.8617	0.8854	0.9468
0.8854	0.8617	0.9995	0.9993	1.0000
0.8617	0.8854	0.9993	0.9995	1.0000
0.9468	0.9468	1.0000	1.0000	1.0000

]

6️⃣ Laplacian Kernel

K(x,z) = exp(-γ||x-z||₁)

For A and C:

||A-C||₁ = |1-4| + |2-5|
= 3 + 3
= 6

K(A,C) = e^-0.1(6)
= e^-0.6
≈ 0.5488

Laplacian Kernel Matrix

[

1.0000	0.8187	0.5488	0.5488	0.4066
0.8187	1.0000	0.5488	0.5488	0.4066
0.5488	0.5488	1.0000	0.8187	0.7408
0.5488	0.5488	0.8187	1.0000	0.7408
0.4066	0.4066	0.7408	0.7408	1.0000

]

7️⃣ ANOVA Kernel

K(x,z) = [ Σᵢ exp(-γ(xᵢ-zᵢ)²) ]ᵈ

Take:

γ = 0.1    d = 2

For A = (1,2) and C = (4,5):

K(A,C) = [e^-0.1(1-4)² + e^-0.1(2-5)²]²
= [e^-0.9 + e^-0.9]²
= [2(0.40657)]²
≈ 0.6612

ANOVA Kernel Matrix

[

4.0000	3.2749	0.6612	0.7608	0.0806
3.2749	4.0000	0.7608	0.6612	0.0806
0.6612	0.7608	4.0000	3.2749	2.4811
0.7608	0.6612	3.2749	4.0000	2.4811
0.0806	0.0806	2.4811	2.4811	4.0000

]

8️⃣ Chi-Square Kernel

K(x,z) = exp[ -γ Σᵢ (xᵢ-zᵢ)²/(xᵢ+zᵢ) ]

For A and C:

Term 1:
(1-4)²/(1+4) = 9/5 = 1.8

Term 2:
(2-5)²/(2+5) = 9/7 ≈ 1.2857

Total:
1.8 + 1.2857 = 3.0857

K(A,C) = e^-0.1(3.0857)
≈ 0.7345

Chi-Square Kernel Matrix

[

1.0000	0.9355	0.7345	0.7165	0.5728
0.9355	1.0000	0.7165	0.7345	0.5728
0.7345	0.7165	1.0000	0.9780	0.9521
0.7165	0.7345	0.9780	1.0000	0.9521
0.5728	0.5728	0.9521	0.9521	1.0000

]

9️⃣ Histogram Intersection Kernel

K(x,z) = Σᵢ min(xᵢ,zᵢ)

For A and C:

K(A,C) = min(1,4) + min(2,5)
= 1 + 2
= 3

Histogram Intersection Matrix

[

3	2	3	3	3
2	3	3	3	3
3	3	9	8	9
3	3	8	9	9
3	3	9	9	12

]

🔟 Kernel Matrix Concept

For five observations:

X = {x₁,x₂,x₃,x₄,x₅}

the kernel matrix is:

K = [ K(x₁,x₁)   K(x₁,x₂)   ...   K(x₁,x₅)
K(x₂,x₁)   K(x₂,x₂)   ...   K(x₂,x₅)
⋮            ⋱          ⋮
K(x₅,x₁)   K(x₅,x₂)   ...   K(x₅,x₅) ]

Each entry represents the similarity between two observations. For example:

K₂,₄ = K(B,D)

The diagonal contains self-similarity:

K(xᵢ,xᵢ)

📊 Kernel Comparison

Kernel	Formula	K(A,C)
Linear	xᵀz	14
Polynomial d=2	(xᵀz+1)²	225
Polynomial d=3	(xᵀz+1)³	3375
RBF	e-0.1||x-z||²	0.1653
Sigmoid	tanh(0.1xᵀz)	0.8854
Laplacian	e-0.1||x-z||₁	0.5488
ANOVA	[Σe-0.1(xᵢ-zᵢ)²]²	0.6612
Chi-Square	exp[-γΣ((xᵢ-zᵢ)²/(xᵢ+zᵢ))]	0.7345
Histogram Intersection	Σmin(xᵢ,zᵢ)	3

📌 Important:

The kernel matrix is also called the Gram matrix. For five observations, every kernel produces a 5 × 5 matrix.

For the SVM, these kernel values are used inside the dual optimization formulation:

max α: Σᵢ αᵢ − 1/2 ΣᵢΣⱼ αᵢαⱼ yᵢyⱼ K(xᵢ,xⱼ)

subject to:

αᵢ ≥ 0

Σᵢ αᵢyᵢ = 0

CREATED BY BIJAN KRISHNA PAUL