xbe.at

Machine learning / Model evaluation and validation

Pitfalls of Machine Learning Classification Models

11 min read, 16 sections

Contents
  1. The Dark Side of Classification in Machine Learning
  2. Class Imbalance
  3. Overfitting
  4. Feature Selection Bias
  5. Ignoring Feature Correlations
  6. Incorrect Cross-Validation
  7. Neglecting Data Preprocessing
  8. Ignoring Model Assumptions
  9. Misinterpreting Model Metrics
  10. Not Handling Outliers
  11. Ignoring Class Probability Calibration
  12. Not Considering Model Interpretability
  13. Neglecting Data Drift and Model Monitoring
  14. Real-Life Example: Image Classification
  15. Real-Life Example: Text Classification
  16. Additional Resources

The Dark Side of Classification in Machine Learning

Classification is a fundamental task in machine learning, but it's not without its pitfalls. This presentation explores common issues that can compromise the effectiveness of classification models, along with practical solutions using Python.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

# Generate a sample dataset
X, y = make_classification(n_samples=1000, n_features=2, n_informative=2,
                           n_redundant=0, n_clusters_per_class=1, random_state=42)

# Visualize the dataset
plt.scatter(X[:, 0], X[:, 1], c=y, cmap='viridis')
plt.title('Sample Classification Dataset')
plt.xlabel('Feature 1')
plt.ylabel('Feature 2')
plt.colorbar(label='Class')
plt.show()

Class Imbalance

Class imbalance occurs when one class significantly outnumbers the others. This can lead to biased models that perform poorly on minority classes.

from sklearn.utils import resample

# Create an imbalanced dataset
X_imbalanced, y_imbalanced = make_classification(n_samples=1000, n_classes=2, 
                                                 weights=[0.9, 0.1], n_informative=3,
                                                 random_state=42)

# Upsample the minority class
X_minority = X_imbalanced[y_imbalanced == 1]
y_minority = y_imbalanced[y_imbalanced == 1]
X_minority_upsampled, y_minority_upsampled = resample(X_minority, y_minority, 
                                                      n_samples=len(X_imbalanced[y_imbalanced == 0]),
                                                      random_state=42)

# Combine the upsampled minority class with the majority class
X_balanced = np.vstack((X_imbalanced[y_imbalanced == 0], X_minority_upsampled))
y_balanced = np.hstack((y_imbalanced[y_imbalanced == 0], y_minority_upsampled))

print(f"Original class distribution: {np.bincount(y_imbalanced)}")
print(f"Balanced class distribution: {np.bincount(y_balanced)}")

Overfitting

Overfitting occurs when a model learns the training data too well, including its noise and peculiarities, leading to poor generalization on unseen data.

from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score

# Split the data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Train a decision tree with different max_depths
depths = range(1, 20)
train_scores = []
test_scores = []

for depth in depths:
    clf = DecisionTreeClassifier(max_depth=depth, random_state=42)
    clf.fit(X_train, y_train)
    train_scores.append(accuracy_score(y_train, clf.predict(X_train)))
    test_scores.append(accuracy_score(y_test, clf.predict(X_test)))

# Plot the results
plt.plot(depths, train_scores, label='Training Accuracy')
plt.plot(depths, test_scores, label='Testing Accuracy')
plt.xlabel('Tree Depth')
plt.ylabel('Accuracy')
plt.title('Overfitting in Decision Trees')
plt.legend()
plt.show()

Feature Selection Bias

Feature selection bias occurs when we choose features based on their performance on the entire dataset, leading to overly optimistic estimates of model performance.

from sklearn.feature_selection import SelectKBest, f_classif

# Generate a dataset with irrelevant features
X_biased, y_biased = make_classification(n_samples=1000, n_features=20, n_informative=5,
                                         n_redundant=5, n_repeated=0, n_classes=2,
                                         random_state=42)

# Incorrect way: Feature selection on entire dataset
selector = SelectKBest(f_classif, k=5)
X_selected = selector.fit_transform(X_biased, y_biased)

# Correct way: Feature selection only on training data
X_train, X_test, y_train, y_test = train_test_split(X_biased, y_biased, test_size=0.3, random_state=42)
selector = SelectKBest(f_classif, k=5)
X_train_selected = selector.fit_transform(X_train, y_train)
X_test_selected = selector.transform(X_test)

print(f"Number of features before selection: {X_biased.shape[1]}")
print(f"Number of features after selection: {X_selected.shape[1]}")

Ignoring Feature Correlations

High correlation between features can lead to multicollinearity, making it difficult to interpret the importance of individual features and potentially affecting model performance.

import seaborn as sns

# Generate correlated features
n_samples = 1000
X_corr = np.random.randn(n_samples, 3)
X_corr[:, 1] = X_corr[:, 0] + np.random.randn(n_samples) * 0.1
X_corr[:, 2] = X_corr[:, 0] + np.random.randn(n_samples) * 0.1

# Calculate correlation matrix
corr_matrix = np.corrcoef(X_corr.T)

# Visualize correlation matrix
plt.figure(figsize=(8, 6))
sns.heatmap(corr_matrix, annot=True, cmap='coolwarm', vmin=-1, vmax=1, center=0)
plt.title('Feature Correlation Matrix')
plt.show()

Incorrect Cross-Validation

Improper cross-validation can lead to biased performance estimates. Common mistakes include data leakage and using the wrong strategy for time-series data.

from sklearn.model_selection import cross_val_score, TimeSeriesSplit

# Generate time series data
np.random.seed(42)
X_ts = np.array([i + np.random.randn() for i in range(1000)]).reshape(-1, 1)
y_ts = (X_ts > 0).astype(int).ravel()

# Incorrect: standard k-fold cross-validation
incorrect_cv_scores = cross_val_score(LogisticRegression(), X_ts, y_ts, cv=5)

# Correct: time series cross-validation
tscv = TimeSeriesSplit(n_splits=5)
correct_cv_scores = cross_val_score(LogisticRegression(), X_ts, y_ts, cv=tscv)

print(f"Incorrect CV scores: {incorrect_cv_scores.mean():.3f} (+/- {incorrect_cv_scores.std() * 2:.3f})")
print(f"Correct CV scores: {correct_cv_scores.mean():.3f} (+/- {correct_cv_scores.std() * 2:.3f})")

Neglecting Data Preprocessing

Failing to preprocess data properly can lead to poor model performance. Common preprocessing steps include scaling, handling missing values, and encoding categorical variables.

from sklearn.preprocessing import StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline

# Create a dataset with mixed types and missing values
X_mixed = np.column_stack([
    np.random.randn(100),  # Continuous feature
    np.random.choice(['A', 'B', 'C'], 100),  # Categorical feature
    np.random.randn(100)  # Continuous feature with missing values
])
X_mixed[np.random.choice(100, 10), 2] = np.nan  # Introduce missing values

# Define preprocessing steps
numeric_features = [0, 2]
categorical_features = [1]
preprocessor = ColumnTransformer(
    transformers=[
        ('num', Pipeline([
            ('imputer', SimpleImputer(strategy='mean')),
            ('scaler', StandardScaler())
        ]), numeric_features),
        ('cat', Pipeline([
            ('imputer', SimpleImputer(strategy='constant', fill_value='missing')),
            ('onehot', OneHotEncoder(handle_unknown='ignore'))
        ]), categorical_features)
    ])

# Fit and transform the data
X_preprocessed = preprocessor.fit_transform(X_mixed)

print(f"Shape before preprocessing: {X_mixed.shape}")
print(f"Shape after preprocessing: {X_preprocessed.shape}")

Ignoring Model Assumptions

Many classification algorithms make assumptions about the data. Violating these assumptions can lead to poor model performance or incorrect interpretations.

from sklearn.discriminant_analysis import LinearDiscriminantAnalysis

# Generate non-linearly separable data
X_nonlinear, y_nonlinear = make_classification(n_samples=1000, n_features=2, n_classes=2,
                                               n_clusters_per_class=2, random_state=42)

# Fit LDA (assumes linearly separable classes)
lda = LinearDiscriminantAnalysis()
lda.fit(X_nonlinear, y_nonlinear)

# Plot decision boundary
x_min, x_max = X_nonlinear[:, 0].min() - 1, X_nonlinear[:, 0].max() + 1
y_min, y_max = X_nonlinear[:, 1].min() - 1, X_nonlinear[:, 1].max() + 1
xx, yy = np.meshgrid(np.arange(x_min, x_max, 0.1),
                     np.arange(y_min, y_max, 0.1))
Z = lda.predict(np.c_[xx.ravel(), yy.ravel()])
Z = Z.reshape(xx.shape)

plt.contourf(xx, yy, Z, alpha=0.4)
plt.scatter(X_nonlinear[:, 0], X_nonlinear[:, 1], c=y_nonlinear, alpha=0.8)
plt.title('LDA on Non-linearly Separable Data')
plt.show()

Misinterpreting Model Metrics

Relying solely on accuracy can be misleading, especially for imbalanced datasets. It's crucial to consider multiple metrics for a comprehensive evaluation.

from sklearn.metrics import confusion_matrix, precision_score, recall_score, f1_score

# Create an imbalanced dataset
X_imbalanced, y_imbalanced = make_classification(n_samples=1000, n_classes=2, 
                                                 weights=[0.9, 0.1], n_informative=3,
                                                 random_state=42)

# Split the data and train a model
X_train, X_test, y_train, y_test = train_test_split(X_imbalanced, y_imbalanced, test_size=0.3, random_state=42)
clf = LogisticRegression().fit(X_train, y_train)
y_pred = clf.predict(X_test)

# Calculate various metrics
cm = confusion_matrix(y_test, y_pred)
accuracy = clf.score(X_test, y_test)
precision = precision_score(y_test, y_pred)
recall = recall_score(y_test, y_pred)
f1 = f1_score(y_test, y_pred)

print(f"Confusion Matrix:\n{cm}")
print(f"Accuracy: {accuracy:.3f}")
print(f"Precision: {precision:.3f}")
print(f"Recall: {recall:.3f}")
print(f"F1 Score: {f1:.3f}")

Not Handling Outliers

Outliers can significantly impact model performance, especially for algorithms sensitive to extreme values like linear models or k-nearest neighbors.

from sklearn.neighbors import KNeighborsClassifier

# Generate data with outliers
X_outliers, y_outliers = make_classification(n_samples=1000, n_features=2, n_informative=2,
                                             n_redundant=0, n_clusters_per_class=1, random_state=42)
X_outliers[0] = [10, 10]  # Add an outlier

# Train KNN classifier
knn = KNeighborsClassifier(n_neighbors=3)
knn.fit(X_outliers, y_outliers)

# Plot decision boundary
x_min, x_max = X_outliers[:, 0].min() - 1, X_outliers[:, 0].max() + 1
y_min, y_max = X_outliers[:, 1].min() - 1, X_outliers[:, 1].max() + 1
xx, yy = np.meshgrid(np.arange(x_min, x_max, 0.1),
                     np.arange(y_min, y_max, 0.1))
Z = knn.predict(np.c_[xx.ravel(), yy.ravel()])
Z = Z.reshape(xx.shape)

plt.contourf(xx, yy, Z, alpha=0.4)
plt.scatter(X_outliers[:, 0], X_outliers[:, 1], c=y_outliers, alpha=0.8)
plt.title('KNN Classification with Outlier')
plt.show()

Ignoring Class Probability Calibration

Some models may produce poorly calibrated probabilities, leading to unreliable confidence estimates for predictions.

from sklearn.calibration import calibration_curve
from sklearn.naive_bayes import GaussianNB

# Generate data and split into train and test sets
X, y = make_classification(n_samples=1000, n_classes=2, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Train a Naive Bayes classifier (known for poor probability calibration)
nb = GaussianNB()
nb.fit(X_train, y_train)

# Calculate calibration curve
prob_true, prob_pred = calibration_curve(y_test, nb.predict_proba(X_test)[:, 1], n_bins=10)

# Plot calibration curve
plt.plot([0, 1], [0, 1], linestyle='--', label='Perfectly calibrated')
plt.plot(prob_pred, prob_true, marker='.', label='Naive Bayes')
plt.xlabel('Mean predicted probability')
plt.ylabel('Fraction of positives')
plt.title('Calibration Curve')
plt.legend()
plt.show()

Not Considering Model Interpretability

Complex models like deep neural networks can achieve high accuracy but may be difficult to interpret, which can be problematic in domains requiring explanations for decisions.

from sklearn.tree import DecisionTreeClassifier, plot_tree
from sklearn.ensemble import RandomForestClassifier

# Generate and split data
X, y = make_classification(n_samples=1000, n_features=5, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Train a simple decision tree (more interpretable)
dt = DecisionTreeClassifier(max_depth=3, random_state=42)
dt.fit(X_train, y_train)

# Train a more complex Random Forest
rf = RandomForestClassifier(n_estimators=100, random_state=42)
rf.fit(X_train, y_train)

# Compare accuracies
dt_accuracy = dt.score(X_test, y_test)
rf_accuracy = rf.score(X_test, y_test)

print(f"Decision Tree Accuracy: {dt_accuracy:.3f}")
print(f"Random Forest Accuracy: {rf_accuracy:.3f}")

# Visualize the decision tree
plt.figure(figsize=(15,10))
plot_tree(dt, filled=True, feature_names=[f'F{i}' for i in range(X.shape[1])], class_names=['0', '1'])
plt.title('Decision Tree Visualization')
plt.show()

Neglecting Data Drift and Model Monitoring

Models can become less accurate over time as the distribution of incoming data changes. Failing to monitor and update models can lead to degraded performance.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LogisticRegression

# Simulate data drift
np.random.seed(42)
n_samples = 1000
time = np.arange(n_samples)

# Initial data distribution
X_initial = np.random.randn(n_samples // 2, 1)
y_initial = (X_initial > 0).astype(int).ravel()

# Drifted data distribution
X_drift = np.random.randn(n_samples // 2, 1) + 1  # Mean shift
y_drift = (X_drift > 1).astype(int).ravel()

# Combine data
X = np.vstack((X_initial, X_drift))
y = np.hstack((y_initial, y_drift))

# Train model on initial data
model = LogisticRegression()
model.fit(X_initial, y_initial)

# Predict on all data
y_pred = model.predict(X)

# Calculate rolling accuracy
window = 100
rolling_acc = np.array([np.mean(y[i:i+window] == y_pred[i:i+window]) for i in range(n_samples - window)])

# Plot results
plt.figure(figsize=(10, 6))
plt.plot(time[window:], rolling_acc)
plt.axvline(x=n_samples // 2, color='r', linestyle='--', label='Drift Point')
plt.xlabel('Time')
plt.ylabel('Rolling Accuracy')
plt.title('Model Performance Over Time with Data Drift')
plt.legend()
plt.show()

Real-Life Example: Image Classification

In image classification, a common pitfall is not accounting for biases in the training data. For instance, a model trained to classify animals might perform poorly on images with unusual backgrounds or lighting conditions.

from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC
from sklearn.metrics import confusion_matrix
import seaborn as sns

# Load the digits dataset
digits = load_digits()
X, y = digits.data, digits.target

# Split the data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Train a simple SVM classifier
svm = SVC()
svm.fit(X_train, y_train)

# Predict on test set
y_pred = svm.predict(X_test)

# Create confusion matrix
cm = confusion_matrix(y_test, y_pred)

# Plot confusion matrix
plt.figure(figsize=(10, 8))
sns.heatmap(cm, annot=True, fmt='d', cmap='Blues')
plt.title('Confusion Matrix for Digit Classification')
plt.xlabel('Predicted Label')
plt.ylabel('True Label')
plt.show()

# Display some misclassified images
misclassified = X_test[y_test != y_pred]
mis_pred = y_pred[y_test != y_pred]
mis_true = y_test[y_test != y_pred]

fig, axes = plt.subplots(2, 5, figsize=(15, 6))
for i, ax in enumerate(axes.flat):
    if i < len(misclassified):
        ax.imshow(misclassified[i].reshape(8, 8), cmap='gray')
        ax.set_title(f'True: {mis_true[i]}, Pred: {mis_pred[i]}')
    ax.axis('off')
plt.tight_layout()
plt.show()

Real-Life Example: Text Classification

In text classification, a common pitfall is overfitting to specific words or phrases that may not generalize well. This can lead to poor performance on new, unseen text data.

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.model_selection import train_test_split

# Sample text data
texts = [
    "I love this product", "Great service", "Terrible experience",
    "Awful customer support", "Amazing quality", "Disappointing results",
    "Outstanding performance", "Waste of money", "Highly recommended",
    "Never buying again"
]
labels = [1, 1, 0, 0, 1, 0, 1, 0, 1, 0]  # 1 for positive, 0 for negative

# Split the data
X_train, X_test, y_train, y_test = train_test_split(texts, labels, test_size=0.3, random_state=42)

# Vectorize the text
vectorizer = CountVectorizer()
X_train_vec = vectorizer.fit_transform(X_train)
X_test_vec = vectorizer.transform(X_test)

# Train a Naive Bayes classifier
clf = MultinomialNB()
clf.fit(X_train_vec, y_train)

# Evaluate the model
train_score = clf.score(X_train_vec, y_train)
test_score = clf.score(X_test_vec, y_test)

print(f"Training accuracy: {train_score:.2f}")
print(f"Testing accuracy: {test_score:.2f}")

# Show most important features for each class
feature_names = vectorizer.get_feature_names_out()
for i, category in enumerate(["Negative", "Positive"]):
    top_features = sorted(zip(clf.feature_log_prob_[i], feature_names), reverse=True)[:5]
    print(f"\nTop 5 features for {category}:")
    for score, word in top_features:
        print(f"{word}: {score:.2f}")

Additional Resources

For further exploration of machine learning pitfalls and best practices, consider the following resources:

  1. "A Few Useful Things to Know About Machine Learning" by Pedro Domingos ArXiv link: https://arxiv.org/abs/1206.5533
  2. "Machine Learning: The High-Interest Credit Card of Technical Debt" by D. Sculley et al. ArXiv link: https://arxiv.org/abs/1410.5244
  3. "Hidden Technical Debt in Machine Learning Systems" by D. Sculley et al. ArXiv link: https://arxiv.org/abs/1412.6564
  4. "Troubleshooting Deep Neural Networks" by Josh Tobin Available at: http://josh-tobin.com/assets/pdf/troubleshooting-deep-neural-networks-01-19.pdf

These resources provide valuable insights into common challenges in machine learning and strategies to overcome them.

More in Model evaluation and validation

  1. Precision-Recall Plots for Classifier Evaluation9 min

    Precision-Recall plots are powerful tools for evaluating the performance of classification models. They provide insights into the trade-off between precision...

  2. Model Evaluation Metrics in Python6 min

    Model evaluation metrics are essential tools in data science for assessing the performance of machine learning models. This slideshow will explore the top 10...

  3. Precision-Recall Plots in Python13 min

    Precision-Recall plots are powerful tools for evaluating binary classification models, especially when dealing with imbalanced datasets. These plots help...

  4. Model Evaluation and Refinement with Python7 min

    Model evaluation and refinement are crucial steps in the machine learning pipeline. They help us assess the performance of our models and improve them...

  5. Regression Error Metrics with Python Code Examples8 min

    Mean Squared Error is a fundamental regression metric that measures the average squared difference between predicted and actual values. It heavily penalizes...

  6. Methods to Improve Machine Learning Accuracy Evaluation10 min

    Accuracy evaluation is crucial in machine learning to assess model performance. It involves comparing predicted outcomes with actual results. This process...