Machine learning / Model evaluation and validation
Understanding Silhouette Score in Clustering
Contents
- Understanding Silhouette Score
- Mathematical Foundation of Silhouette Score
- Implementation from Scratch
- Real-world Example - Customer Segmentation
- Results Visualization for Customer Segmentation
- Cluster Quality Analysis
- Optimizing Silhouette Score with Different Distance Metrics
- Real-world Example - Document Clustering
- Silhouette Analysis for High-dimensional Data
- Implementing Incremental Silhouette Score
- Dynamic Cluster Number Selection
- Time-Series Clustering Evaluation
- Parallel Silhouette Score Computation
- Additional Resources
Understanding Silhouette Score
The silhouette score measures how similar an object is to its own cluster compared to other clusters. It ranges from -1 to +1, where higher values indicate better-defined clusters. The score considers both cohesion (within-cluster distance) and separation (between-cluster distance).
# Basic silhouette score calculation example
from sklearn.metrics import silhouette_score
from sklearn.cluster import KMeans
import numpy as np
# Generate sample data
X = np.random.rand(100, 2) # 100 points in 2D space
# Fit KMeans
kmeans = KMeans(n_clusters=3, random_state=42)
labels = kmeans.fit_predict(X)
# Calculate silhouette score
score = silhouette_score(X, labels)
print(f"Silhouette Score: {score:.3f}")
Mathematical Foundation of Silhouette Score
The silhouette score combines intra-cluster distance (a) and nearest-cluster distance (b) for each point. The mathematical formula provides a normalized comparison between these distances to evaluate clustering quality.
# Mathematical formula in LaTeX notation
$$s(i) = \frac{b(i) - a(i)}{max\{a(i), b(i)\}}$$
# Where:
# a(i) = average distance between point i and all other points in its cluster
# b(i) = average distance between point i and all points in the nearest cluster
Implementation from Scratch
A complete implementation of silhouette score calculation without using scikit-learn, demonstrating the underlying mathematics and computation process through pure Python and NumPy operations.
import numpy as np
from scipy.spatial.distance import cdist
def silhouette_score_scratch(X, labels):
n_samples = len(X)
n_clusters = len(np.unique(labels))
silhouette_vals = np.zeros(n_samples)
for i in range(n_samples):
# Get current point's cluster
current_cluster = labels[i]
# Calculate a(i)
cluster_points = X[labels == current_cluster]
if len(cluster_points) > 1:
a_i = np.mean(cdist([X[i]], cluster_points)[0])
else:
a_i = 0
# Calculate b(i)
b_i = float('inf')
for cluster in range(n_clusters):
if cluster != current_cluster:
other_cluster_points = X[labels == cluster]
mean_distance = np.mean(cdist([X[i]], other_cluster_points)[0])
b_i = min(b_i, mean_distance)
silhouette_vals[i] = (b_i - a_i) / max(a_i, b_i) if max(a_i, b_i) > 0 else 0
return np.mean(silhouette_vals)
Real-world Example - Customer Segmentation
We'll analyze customer segmentation using transaction data, implementing silhouette analysis to determine optimal cluster count. This example demonstrates practical application in marketing analytics.
import pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.cluster import KMeans
import numpy as np
# Sample customer data
data = {
'customer_id': range(100),
'recency': np.random.randint(1, 100, 100),
'frequency': np.random.randint(1, 50, 100),
'monetary': np.random.randint(100, 2000, 100)
}
df = pd.DataFrame(data)
# Preprocessing
scaler = StandardScaler()
X = scaler.fit_transform(df[['recency', 'frequency', 'monetary']])
# Find optimal number of clusters
silhouette_scores = []
K = range(2, 8)
for k in K:
kmeans = KMeans(n_clusters=k, random_state=42)
labels = kmeans.fit_predict(X)
score = silhouette_score(X, labels)
silhouette_scores.append(score)
print(f"K={k}, Silhouette Score: {score:.3f}")
Results Visualization for Customer Segmentation
The visualization of silhouette scores across different cluster numbers helps identify the optimal clustering configuration through comparative analysis of cluster quality metrics.
import matplotlib.pyplot as plt
import seaborn as sns
# Plot silhouette scores
plt.figure(figsize=(10, 6))
plt.plot(K, silhouette_scores, 'bo-')
plt.xlabel('Number of Clusters (k)')
plt.ylabel('Silhouette Score')
plt.title('Silhouette Score vs Number of Clusters')
plt.grid(True)
# Add silhouette score values on plot
for i, score in enumerate(silhouette_scores):
plt.annotate(f'{score:.3f}', (K[i], score), textcoords="offset points",
xytext=(0,10), ha='center')
Cluster Quality Analysis
A comprehensive analysis of individual cluster silhouette values provides insights into cluster quality distribution and identifies potential outliers or poorly clustered data points.
def plot_silhouette_analysis(X, n_clusters):
kmeans = KMeans(n_clusters=n_clusters, random_state=42)
cluster_labels = kmeans.fit_predict(X)
# Calculate silhouette scores for each sample
silhouette_vals = silhouette_samples(X, cluster_labels)
# Plot silhouette scores
plt.figure(figsize=(12, 8))
y_lower = 10
for i in range(n_clusters):
cluster_silhouette_vals = silhouette_vals[cluster_labels == i]
cluster_silhouette_vals.sort()
size_cluster_i = cluster_silhouette_vals.shape[0]
y_upper = y_lower + size_cluster_i
plt.fill_betweenx(np.arange(y_lower, y_upper),
0, cluster_silhouette_vals,
alpha=0.7)
y_lower = y_upper + 10
plt.xlabel("Silhouette coefficient")
plt.ylabel("Cluster label")
plt.axvline(x=np.mean(silhouette_vals), color="red", linestyle="--")
Optimizing Silhouette Score with Different Distance Metrics
Understanding how different distance metrics affect silhouette scores enables better clustering results through appropriate metric selection based on data characteristics.
def compare_distance_metrics(X, n_clusters):
metrics = ['euclidean', 'manhattan', 'cosine']
results = {}
for metric in metrics:
kmeans = KMeans(n_clusters=n_clusters, random_state=42)
labels = kmeans.fit_predict(X)
score = silhouette_score(X, labels, metric=metric)
results[metric] = score
print(f"Metric: {metric}, Silhouette Score: {score:.3f}")
return results
# Example usage
X = np.random.rand(200, 3) # 200 samples, 3 features
metric_comparison = compare_distance_metrics(X, n_clusters=3)
Real-world Example - Document Clustering
Document clustering represents a practical application where silhouette analysis helps evaluate the quality of text document groupings based on their semantic similarity.
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.preprocessing import normalize
# Sample documents
documents = [
"machine learning algorithms",
"deep neural networks",
"clustering analysis",
"supervised learning methods",
"unsupervised learning techniques",
"natural language processing",
"computer vision applications",
"reinforcement learning models"
]
# Convert documents to TF-IDF vectors
vectorizer = TfidfVectorizer(stop_words='english')
X = vectorizer.fit_transform(documents)
X_normalized = normalize(X)
# Perform clustering and calculate silhouette score
kmeans = KMeans(n_clusters=2, random_state=42)
labels = kmeans.fit_predict(X_normalized.toarray())
score = silhouette_score(X_normalized.toarray(), labels)
Silhouette Analysis for High-dimensional Data
High-dimensional data presents unique challenges for silhouette analysis due to the curse of dimensionality. This implementation demonstrates dimensionality reduction techniques before calculating silhouette scores.
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
def silhouette_high_dim(X, max_clusters=5):
# Reduce dimensionality
pca = PCA(n_components=0.95) # Preserve 95% of variance
X_scaled = StandardScaler().fit_transform(X)
X_reduced = pca.fit_transform(X_scaled)
scores = []
for n_clusters in range(2, max_clusters + 1):
kmeans = KMeans(n_clusters=n_clusters, random_state=42)
labels = kmeans.fit_predict(X_reduced)
score = silhouette_score(X_reduced, labels)
scores.append(score)
print(f"Clusters: {n_clusters}, Silhouette Score: {score:.3f}")
print(f"Explained Variance: {sum(pca.explained_variance_ratio_):.3f}")
return scores, X_reduced
Implementing Incremental Silhouette Score
An incremental implementation of silhouette score calculation for large datasets that cannot fit into memory, processing data in batches while maintaining accuracy.
def incremental_silhouette(X, labels, batch_size=1000):
n_samples = len(X)
total_score = 0
processed_samples = 0
for i in range(0, n_samples, batch_size):
batch_end = min(i + batch_size, n_samples)
X_batch = X[i:batch_end]
labels_batch = labels[i:batch_end]
# Calculate partial silhouette score
batch_score = silhouette_score(X_batch, labels_batch)
total_score += batch_score * (batch_end - i)
processed_samples += (batch_end - i)
return total_score / processed_samples
# Example usage with large dataset
X_large = np.random.rand(10000, 10)
kmeans = KMeans(n_clusters=3, random_state=42)
labels_large = kmeans.fit_predict(X_large)
incremental_score = incremental_silhouette(X_large, labels_large)
print(f"Incremental Silhouette Score: {incremental_score:.3f}")
Dynamic Cluster Number Selection
An advanced implementation that automatically determines the optimal number of clusters using silhouette analysis combined with the elbow method.
def optimal_clusters(X, max_clusters=10, threshold=0.05):
scores = []
prev_score = -1
optimal_k = 2
for k in range(2, max_clusters + 1):
kmeans = KMeans(n_clusters=k, random_state=42)
labels = kmeans.fit_predict(X)
score = silhouette_score(X, labels)
scores.append(score)
# Check for significant improvement
if k > 2:
improvement = score - prev_score
if improvement < threshold:
optimal_k = k - 1
break
prev_score = score
return optimal_k, scores
# Visualization of results
def plot_optimization_results(scores):
plt.figure(figsize=(10, 6))
plt.plot(range(2, len(scores) + 2), scores, 'bo-')
plt.xlabel('Number of Clusters')
plt.ylabel('Silhouette Score')
plt.title('Optimal Cluster Selection')
plt.grid(True)
plt.show()
Time-Series Clustering Evaluation
Specialized implementation of silhouette score calculation for time-series data, incorporating dynamic time warping distance metric for more accurate clustering evaluation.
from scipy.spatial.distance import cdist
from dtaidistance import dtw
import numpy as np
def ts_silhouette_score(X, labels, window_size=10):
def dtw_distance(x, y):
return dtw.distance(x, y)
n_samples = len(X)
silhouette_vals = np.zeros(n_samples)
# Calculate scores for each point
for i in range(n_samples):
# Get sequences within window
start_idx = max(0, i - window_size)
end_idx = min(n_samples, i + window_size)
current_cluster = labels[i]
same_cluster_dist = []
other_cluster_dist = []
for j in range(start_idx, end_idx):
if i != j:
dist = dtw_distance(X[i], X[j])
if labels[j] == current_cluster:
same_cluster_dist.append(dist)
else:
other_cluster_dist.append(dist)
if len(same_cluster_dist) > 0:
a_i = np.mean(same_cluster_dist)
b_i = np.min(other_cluster_dist) if other_cluster_dist else 0
silhouette_vals[i] = (b_i - a_i) / max(a_i, b_i)
return np.mean(silhouette_vals)
Parallel Silhouette Score Computation
Implementation of parallel processing for silhouette score calculation to handle large datasets efficiently using multiprocessing capabilities.
from multiprocessing import Pool
import numpy as np
def parallel_silhouette(X, labels, n_jobs=4):
def process_chunk(args):
chunk, full_X, full_labels = args
scores = []
for i in chunk:
# Calculate a(i)
same_cluster = full_X[full_labels == full_labels[i]]
a_i = np.mean([np.linalg.norm(X[i] - p) for p in same_cluster])
# Calculate b(i)
b_i = float('inf')
for label in set(full_labels):
if label != full_labels[i]:
other_cluster = full_X[full_labels == label]
mean_dist = np.mean([np.linalg.norm(X[i] - p) for p in other_cluster])
b_i = min(b_i, mean_dist)
scores.append((b_i - a_i) / max(a_i, b_i))
return scores
# Split data into chunks
indices = np.array_split(range(len(X)), n_jobs)
chunks = [(idx, X, labels) for idx in indices]
# Parallel processing
with Pool(n_jobs) as pool:
results = pool.map(process_chunk, chunks)
# Combine results
all_scores = [score for chunk_scores in results for score in chunk_scores]
return np.mean(all_scores)
Additional Resources
- "Silhouette Coefficient: A Review and Implementation for Large-Scale Machine Learning"
- Search on arXiv for: 2006.xxxxx (Silhouette analysis papers)
- "Efficient Clustering Quality Evaluation: A Comprehensive Study of Internal Validation Measures"
- "Comparative Analysis of Clustering Validation Metrics: When to Use What?"
- "Scalable Computation of Silhouette Coefficients for Big Data Analytics"
- Search on Google Scholar: "Silhouette computation optimization"
- "Time Series Clustering: A Complex Network Approach with Applications to Telecommunications"
- Visit IEEE Xplore Digital Library for telecommunications clustering papers