xbe.at

Machine learning / Classification and clustering

Introduction to Support Vector Machines (SVM) in Python

3 min read, 12 sections

Contents
  1. Introduction to Support Vector Machines (SVM)
  2. Importing Libraries
  3. Loading Data
  4. Creating SVM Model
  5. Training the SVM Model
  6. Visualizing Decision Boundaries
  7. Plotting Training Data
  8. Using Non-linear Kernels
  9. Visualizing Non-linear Decision Boundaries
  10. Tuning SVM Hyperparameters
  11. Making Predictions
  12. Evaluating SVM Performance

Introduction to Support Vector Machines (SVM)

Support Vector Machines (SVM) are powerful supervised machine learning algorithms used for classification and regression tasks. They work by finding the optimal hyperplane that maximally separates classes in high-dimensional feature spaces. SVMs are particularly effective for handling non-linear problems and high-dimensional data.

# No code for this introductory slide

Importing Libraries

To work with SVMs in Python, we need to import the necessary libraries. The scikit-learn library provides a user-friendly implementation of SVMs.

from sklearn import datasets
from sklearn import svm
import numpy as np
import matplotlib.pyplot as plt

Loading Data

Let's load the iris dataset, a popular example for classification tasks. This dataset contains measurements of iris flowers from three different species.

iris = datasets.load_iris()
X = iris.data[:, :2]  # We only take the first two features
y = iris.target

Creating SVM Model

We create an instance of the SVM classifier from scikit-learn. The kernel parameter specifies the type of kernel function used for mapping the data into a higher-dimensional space.

clf = svm.SVC(kernel='linear')

Training the SVM Model

The SVM model is trained using the fit() method, which takes the feature matrix (X) and target labels (y) as input.

clf.fit(X, y)

Visualizing Decision Boundaries

We can visualize the decision boundaries learned by the SVM model using a mesh grid and the predict() method to get the predictions for each point in the grid.

x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
h = (x_max / x_min)/100
xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])
Z = Z.reshape(xx.shape)
plt.contourf(xx, yy, Z, cmap=plt.cm.Paired, alpha=0.8)

Plotting Training Data

We can plot the training data points on top of the decision boundaries to better visualize the classification.

plt.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.Paired)
plt.xlabel('Sepal length')
plt.ylabel('Sepal width')
plt.xlim(xx.min(), xx.max())
plt.title('SVM with linear kernel')
plt.show()

Using Non-linear Kernels

SVMs can handle non-linear decision boundaries by using different kernel functions. The rbf (Radial Basis Function) kernel is a popular choice for non-linear problems.

clf = svm.SVC(kernel='rbf')
clf.fit(X, y)

Visualizing Non-linear Decision Boundaries

We can visualize the non-linear decision boundaries learned by the SVM model with an RBF kernel using the same visualization code from earlier slides.

# Same visualization code as before

Tuning SVM Hyperparameters

SVMs have several hyperparameters that can be tuned to improve performance, such as the regularization parameter C and the kernel coefficient gamma for the RBF kernel.

from sklearn.model_selection import GridSearchCV

param_grid = {'C': [0.1, 1, 10, 100], 'gamma': [1, 0.1, 0.01, 0.001]}
grid = GridSearchCV(svm.SVC(kernel='rbf'), param_grid, verbose=3)
grid.fit(X, y)
print("Best parameters: ", grid.best_params_)

Making Predictions

Once the SVM model is trained, we can use it to make predictions on new, unseen data using the predict() method.

new_data = [[5.1, 3.5], [4.9, 3.0], [6.7, 3.1]]
predictions = clf.predict(new_data)
print("Predictions: ", predictions)

Evaluating SVM Performance

To evaluate the performance of the SVM model, we can use metrics like accuracy, precision, recall, and F1-score from scikit-learn.

from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score

y_true = [0, 1, 2]  # True labels
y_pred = clf.predict(new_data)

accuracy = accuracy_score(y_true, y_pred)
precision = precision_score(y_true, y_pred, average='macro')
recall = recall_score(y_true, y_pred, average='macro')
f1 = f1_score(y_true, y_pred, average='macro')

print("Accuracy: ", accuracy)
print("Precision: ", precision)
print("Recall: ", recall)
print("F1-score: ", f1)

More in Classification and clustering

  1. K-means & K-modes for numerical data9 min

    The K-means algorithm partitions n observations into k clusters by minimizing within-cluster variances. Each cluster is represented by its centroid, which is...

  2. In-Depth Exploration of SVM Polynomial Kernel11 min

    Support Vector Machines (SVMs) are powerful classification algorithms that can be enhanced through the use of kernels. Kernels allow SVMs to handle...

  3. K-Means Clustering Algorithm Explained14 min

    The fundamental k-means clustering algorithm implements iterative refinement to partition n observations into k clusters. Each cluster is represented by the...

  4. Impact of Min Samples Leaf on Tree-Based Model Performance11 min

    Min Samples Leaf is a crucial hyperparameter in tree-based models that significantly impacts model performance. It defines the minimum number of samples...

  5. K-Means Clustering Algorithm in Python5 min

    Clustering is an unsupervised machine learning technique used to group similar data points together. While there are various clustering algorithms, this...

  6. Hierarchical Clustering in Python8 min

    Hierarchical clustering is an unsupervised machine learning technique used to group similar data points into clusters. Unlike other clustering methods, it...