Support Vector Machine (SVM) Using Python

 

Support Vector Machine (SVM) 

---

🟦 Program Aim

Aim:

To implement the Support Vector Machine (SVM) algorithm using Python and classify objects into different categories.

---

🟩 Algorithm Used

Support Vector Machine (SVM) Classifier

---

🟨 Problem Statement

A fruit shop wants to classify fruits into two categories:

• 🍎 Small Fruit
• 🍉 Large Fruit

The classification is based on the weight of the fruit.

---

🟪 Step 1: Import the Required Library

First, import the SVC (Support Vector Classifier) class from the sklearn.svm module.

from sklearn.svm import SVC

Explanation

• sklearn is the Scikit-learn machine learning library.
• svm is the module that contains Support Vector Machine algorithms.
• SVC() is used for classification problems.

---

🟦 Step 2: Create the Training Dataset

X = [
    [2],
    [3],
    [4],
    [5]
]

Explanation

X represents the input feature (Independent Variable).

Here, each value represents the weight of a fruit (in kg).

Fruit	Weight (kg)
Fruit 1	2
Fruit 2	3
Fruit 3	4
Fruit 4	5

The SVM algorithm learns from these weight values.

---

🟩 Step 3: Create the Output Labels

y = [
    "Small",
    "Small",
    "Large",
    "Large"
]

Explanation

y represents the target labels (Dependent Variable).

Weight	Category
2	Small
3	Small
4	Large
5	Large

The model learns which weight belongs to which category.

---

🟨 Step 4: Create the SVM Model

model = SVC(kernel="linear")

Explanation

• SVC() creates the Support Vector Machine model.
• kernel="linear" tells the model to use a Linear Kernel.
• The model will find the best straight-line boundary (hyperplane) between the two categories.

---

🟪 Step 5: Train the Model

model.fit(X, y)

Explanation

The fit() function trains the SVM model.

Syntax

model.fit(X, y)

Where:

• X = Input data
• y = Output labels

During training, the algorithm:

• Reads the training data.
• Finds the support vectors.
• Calculates the maximum margin.
• Draws the optimal hyperplane.

---

🟦 Step 6: Predict New Data

Suppose a new fruit has a weight of 4 kg.

prediction = model.predict([[4]])

Explanation

predict() is used to classify new data.

Syntax

model.predict([[value]])

Here,

[[4]]

means the weight of the new fruit is 4 kg.

The model predicts whether it is Small or Large.

---

🟩 Step 7: Display the Result

print("Prediction =", prediction[0])

Explanation

prediction is returned as a list.

Example:

['Large']

To print only the predicted class, use:

prediction[0]

Output

Prediction = Large

---

🟨 Complete Python Program

# Import Support Vector Machine
from sklearn.svm import SVC

# Training Data (Fruit Weight)
X = [
    [2],
    [3],
    [4],
    [5]
]

# Output Labels
y = [
    "Small",
    "Small",
    "Large",
    "Large"
]

# Create SVM Model
model = SVC(kernel="linear")

# Train the Model
model.fit(X, y)

# Predict New Fruit
prediction = model.predict([[4]])

# Display Result
print("Prediction =", prediction[0])

---

🟥 Expected Output

Prediction = Large

---

🟦 Step-by-Step Working of the Program

Step 1
Import SVC Class
        │
        ▼
Step 2
Create Training Dataset (X)
        │
        ▼
Step 3
Create Output Labels (y)
        │
        ▼
Step 4
Create SVM Model
(kernel = "linear")
        │
        ▼
Step 5
Train Model
(model.fit)
        │
        ▼
Step 6
Predict New Data
(model.predict)
        │
        ▼
Step 7
Display Prediction

---

🟩 How SVM Makes the Decision

Suppose the training data is:

Weight	Category
2	Small
3	Small
4	Large
5	Large

The SVM finds the best boundary:

Small Fruits           Large Fruits

2      3      |      4      5
○------○------|------●------●
               ↑
         Best Hyperplane

When a new fruit with weight = 4 kg is given:

• It lies on the Large side of the hyperplane.
• Therefore, the model predicts Large.

---

🟪 Advantages of SVM

• ✔ High accuracy
• ✔ Effective for classification problems
• ✔ Works well with high-dimensional data
• ✔ Handles both linear and non-linear data (using kernels)
• ✔ Less prone to overfitting

---

🟥 Limitations of SVM

• ❌ Training is slower for very large datasets
• ❌ Choosing the correct kernel can be difficult
• ❌ Sensitive to noisy data
• ❌ Requires careful parameter tuning

---

🌍 Real-Life Applications

• 🏥 Disease Diagnosis
• 📧 Spam Email Detection
• 😊 Face Recognition
• ✍️ Handwriting Recognition
• 💳 Credit Card Fraud Detection
• 🚗 Traffic Sign Recognition
• 📱 Image Classification

---

📝 Viva Questions

1. What is Support Vector Machine (SVM)?
2. What is a hyperplane in SVM?
3. What are support vectors?
4. What is the role of the kernel in SVM?
5. What is the difference between Linear SVM and Non-Linear SVM?
6. Why is SVM considered a powerful classification algorithm?

---

⭐ One-Line Revision

Support Vector Machine (SVM) is a supervised machine learning algorithm that classifies data by finding the optimal hyperplane with the maximum margin between different classes.

Decision Tree USING PYTHON

 

Decision Tree in Machine Learning

---

🟦 What is a Decision Tree?

A Decision Tree is a Supervised Machine Learning algorithm used for both:

• ✅ Classification (Predict Categories)
• ✅ Regression (Predict Numerical Values)

It works like a flowchart, where every question divides the dataset into smaller groups until a final decision is reached.

---

🌟 Definition

Decision Tree is a supervised machine learning algorithm that predicts the output by asking a sequence of questions. Each question splits the data into smaller subsets until a final prediction (leaf node) is obtained.

---

🌳 Real-Life Example

🎓 Student Scholarship Prediction

A university wants to decide whether a student is eligible for a scholarship.

Conditions

• CGPA
• Attendance

Decision Tree

               CGPA ≥ 8?
              /        \
           Yes          No
          /              \
 Attendance ≥ 85?     Not Eligible
      /      \
    Yes      No
    |         |
Eligible   Not Eligible

Suppose a student has:

• CGPA = 8.5
• Attendance = 90%

Decision:

CGPA ≥ 8 → Yes

Attendance ≥ 85 → Yes

➡ Scholarship Eligible

---

🌳 Step-by-Step Working of Decision Tree

---

🟩 Step 1: Import Required Libraries

First, import the required libraries.

from sklearn.tree import DecisionTreeClassifier

Explanation

• sklearn.tree contains the Decision Tree algorithm.
• DecisionTreeClassifier() is used for classification problems.

---

🟩 Step 2: Prepare the Dataset

Suppose we have the following training data.

Age	Loan Approved
22	No
25	No
35	Yes
40	Yes
28	No
45	Yes

Python Code

X = [
    [22],
    [25],
    [35],
    [40],
    [28],
    [45]
]

y = [
    "No",
    "No",
    "Yes",
    "Yes",
    "No",
    "Yes"
]

Explanation

• X = Input Feature (Age)
• y = Output Label (Loan Approved)

---

🟩 Step 3: Create the Model

model = DecisionTreeClassifier()

Explanation

This creates an empty Decision Tree model.

Nothing is learned yet.

---

🟩 Step 4: Train the Model

model.fit(X, y)

Explanation

The fit() function trains the model using historical data.

During training, the Decision Tree learns patterns such as:

• Age ≤ 30 → Mostly "No"
• Age > 30 → Mostly "Yes"

The model builds a tree automatically.

---

🟩 Step 5: Predict New Data

Suppose a new applicant is 30 years old.

prediction = model.predict([[30]])

Explanation

The model follows the decision rules learned during training and predicts the class.

---

🟩 Step 6: Display the Result

print("Loan Approval =", prediction[0])

Sample Output

Loan Approval = No

---

🟩 Complete Program

from sklearn.tree import DecisionTreeClassifier

# Training Data
X = [
    [22],
    [25],
    [35],
    [40],
    [28],
    [45]
]

y = [
    "No",
    "No",
    "Yes",
    "Yes",
    "No",
    "Yes"
]

# Create Model
model = DecisionTreeClassifier()

# Train Model
model.fit(X, y)

# Test Data
prediction = model.predict([[30]])

# Output
print("Loan Approval =", prediction[0])

---

🌳 What Happens Internally?

Training Data

Age     Loan
22      No
25      No
28      No
35      Yes
40      Yes
45      Yes

The algorithm searches for the best splitting point.

Possible split:

Age < 30 ?

---

If Yes

22 → No

25 → No

28 → No

---

If No

35 → Yes

40 → Yes

45 → Yes

The tree becomes

             Age < 30?
            /         \
         Yes          No
         |             |
        No            Yes

---

🌳 Decision Process

Suppose the input is

Age = 30

Is Age < 30?

        No

        ↓

Loan Approved = Yes

Suppose

Age = 25

Is Age < 30?

       Yes

        ↓

Loan Approved = No

---

🌳 Visual Representation

             Root Node

            Age < 30?
            /       \
         Yes         No

         No         Yes

---

🌳 Important Functions

Create Model

model = DecisionTreeClassifier()

Creates a Decision Tree model.

---

Train Model

model.fit(X,y)

Learns patterns from data.

---

Predict

model.predict([[30]])

Predicts output for new data.

---

Accuracy

model.score(X,y)

Returns model accuracy.

Example

accuracy = model.score(X,y)

print("Accuracy =", accuracy)

Output

Accuracy = 1.0

---

🌳 Decision Tree Workflow

Training Data
      │
      ▼
Create DecisionTreeClassifier
      │
      ▼
Train Model using fit()
      │
      ▼
Decision Tree is Built
      │
      ▼
New Input Data
      │
      ▼
predict()
      │
      ▼
Final Prediction

---

🌳 Advantages

✔ Easy to understand

✔ Easy to visualize

✔ No feature scaling required

✔ Handles numerical and categorical data

✔ Works for Classification and Regression

---

🌳 Limitations

❌ Can overfit

❌ Sensitive to noisy data

❌ Large trees become complex

---

🌳 Applications

🏦 Loan Approval

🏥 Disease Prediction

📧 Spam Detection

🎓 Student Performance Prediction

🌾 Crop Classification

🚗 Insurance Risk Prediction

---

⭐ Interview / Viva Questions

Q1. What is Decision Tree?

A supervised machine learning algorithm that predicts outputs by splitting data into smaller subsets using decision rules.

---

Q2. Why is it called a Decision Tree?

Because it resembles a tree structure where each node represents a decision, each branch represents an outcome, and each leaf node represents the final prediction.

---

Q3. What does fit() do?

It trains the Decision Tree using the training dataset.

---

Q4. What does predict() do?

It predicts the output for new, unseen data based on the trained model.

---

Q5. Can Decision Trees solve both Classification and Regression problems?

Yes.

• DecisionTreeClassifier → Classification
• DecisionTreeRegressor → Regression

---

⭐ One-Line Revision

Decision Tree = Training Data → Split into Decision Rules → Build Tree → Predict Output

 

🟦🟩🟨 Logistic Regression Using Python

---

🎯 Aim

To implement the Logistic Regression algorithm using Python to classify whether a patient has a disease based on their age.

---

🧠 Theory

Logistic Regression is a Supervised Machine Learning algorithm used for classification problems. Unlike Linear Regression, which predicts continuous values, Logistic Regression predicts categories or classes.

Examples

• 📧 Spam / Not Spam
• 🏥 Disease / Healthy
• 🎓 Pass / Fail
• 💳 Fraud / Not Fraud

The output is usually:

• 0 → No
• 1 → Yes

Logistic Regression uses the Sigmoid Function to convert predictions into probabilities between 0 and 1.

---

🌍 Real-Life Example

A hospital wants to predict whether a patient has diabetes based only on Age.

Training Data

Age	Disease
20	0
25	0
30	0
35	0
40	1
45	1
50	1
55	1

Here,

• 0 = Healthy
• 1 = Disease

---

📝 Step 1: Import Required Libraries

from sklearn.linear_model import LogisticRegression

Explanation

• sklearn → Machine Learning library.
• linear_model → Contains regression algorithms.
• LogisticRegression → Used for classification.

---

📝 Step 2: Create Input Data

X = [
    [20],
    [25],
    [30],
    [35],
    [40],
    [45],
    [50],
    [55]
]

Explanation

X stores the input feature (Age).

Each value is written inside another list because Scikit-learn expects 2D input.

X

20
25
30
35
40
45
50
55

---

📝 Step 3: Create Output Data

y = [
    0,
    0,
    0,
    0,
    1,
    1,
    1,
    1
]

Explanation

y stores the output labels.

Value	Meaning
0	Healthy
1	Disease

---

📝 Step 4: Create the Model

model = LogisticRegression()

Explanation

This creates an empty Logistic Regression model.

At this stage:

✔ No learning

✔ No prediction

✔ Just an empty model

---

📝 Step 5: Train the Model

model.fit(X, y)

Explanation

fit() teaches the model using the training data.

The model learns:

• Relationship between Age and Disease
• Probability of Disease

This is called the Training Phase.

Training Data
      │
      ▼
Model Learning
      │
      ▼
Trained Model

---

📝 Step 6: Predict New Data

Suppose a new patient is 42 years old.

prediction = model.predict([[42]])

Explanation

The model compares age 42 with the learned pattern and predicts:

Either

0

or

1

---

📝 Step 7: Display Prediction

print(prediction)

Output

[1]

Meaning

Disease

---

📝 Step 8: Display User-Friendly Output

if prediction[0] == 1:
    print("Patient has Disease")
else:
    print("Patient is Healthy")

Output

Patient has Disease

---

✅ Complete Python Program

# Logistic Regression Example

from sklearn.linear_model import LogisticRegression

# Input Data (Age)
X = [
    [20],
    [25],
    [30],
    [35],
    [40],
    [45],
    [50],
    [55]
]

# Output Data
# 0 = Healthy
# 1 = Disease

y = [
    0,
    0,
    0,
    0,
    1,
    1,
    1,
    1
]

# Create Model
model = LogisticRegression()

# Train Model
model.fit(X, y)

# Predict
prediction = model.predict([[42]])

# Display Prediction
if prediction[0] == 1:
    print("Patient has Disease")
else:
    print("Patient is Healthy")

---

💻 Expected Output

Patient has Disease

---

🔍 Step-by-Step Flow

Import Library
      │
      ▼
Create Dataset
      │
      ▼
Create Logistic Regression Model
      │
      ▼
Train Model using fit()
      │
      ▼
Predict using predict()
      │
      ▼
Display Result

---

📌 Understanding fit()

model.fit(X, y)

This is the most important line.

It means:

Teach the computer using
X → Input

y → Output

After this line, the model becomes trained.

---

📌 Understanding predict()

model.predict([[42]])

Meaning:

Predict the output for a new patient whose age is 42 years.

---

📌 Understanding prediction[0]

predict() returns a list (or array).

Example:

prediction = [1]

To access the first value:

prediction[0]

Result

1

---

📊 Training Data Visualization

Age	Disease
20	Healthy
25	Healthy
30	Healthy
35	Healthy
40	Disease
45	Disease
50	Disease
55	Disease

The model learns that higher ages in this small example are associated with the "Disease" class.

---

📈 Why Logistic Regression?

Linear Regression	Logistic Regression
Predicts numbers	Predicts categories
Output can be any value	Output is a probability (0–1) and a class
Used for Regression	Used for Classification

---

⭐ Advantages

• ✔ Easy to implement
• ✔ Fast training
• ✔ Works well for binary classification
• ✔ Produces probability estimates
• ✔ Easy to interpret

---

❌ Limitations

• ❌ Assumes a linear relationship between features and the log-odds
• ❌ Less effective for highly complex, non-linear data
• ❌ Sensitive to outliers in some situations

---

🌍 Applications

• 🏥 Disease Prediction
• 📧 Spam Detection
• 💳 Credit Card Fraud Detection
• 🎓 Student Pass/Fail Prediction
• 🛒 Customer Purchase Prediction
• 🏦 Loan Approval Prediction

---

🎯 Viva Questions

1. What is Logistic Regression?
2. Why is it called "Regression" if it is used for classification?
3. What is the role of the Sigmoid Function?
4. What does fit() do?
5. What does predict() do?
6. Why is the input written as [[42]] instead of [42]?
7. What is the meaning of prediction[0]?
8. What types of problems can Logistic Regression solve?

---

📝 One-Line Revision

Logistic Regression is a supervised machine learning algorithm that predicts the probability of a data point belonging to a particular class and is mainly used for binary classification problems.