Evaluating Machine Learning Models with Python

Understanding Model Evaluation Metrics for Machine Learning

Model evaluation metrics are essential for assessing the performance of machine learning algorithms. These metrics help us understand how well our models are performing and guide us in making improvements. In this presentation, we'll explore various evaluation metrics and implement them using Python.

import numpy as np
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score

# Example predictions and true labels
y_true = np.array([1, 0, 1, 1, 0, 1])
y_pred = np.array([1, 0, 1, 0, 0, 1])

# Calculate basic metrics
accuracy = accuracy_score(y_true, y_pred)
precision = precision_score(y_true, y_pred)
recall = recall_score(y_true, y_pred)
f1 = f1_score(y_true, y_pred)

print(f"Accuracy: {accuracy:.2f}")
print(f"Precision: {precision:.2f}")
print(f"Recall: {recall:.2f}")
print(f"F1 Score: {f1:.2f}")

Accuracy: The Simplest Metric

Accuracy is the ratio of correct predictions to the total number of predictions. While it's easy to understand, it can be misleading for imbalanced datasets. Accuracy is best used when classes are evenly distributed and all misclassifications have equal cost.

from sklearn.metrics import accuracy_score

def calculate_accuracy(y_true, y_pred):
    return accuracy_score(y_true, y_pred)

# Example usage
y_true = [0, 1, 1, 0, 1, 0]
y_pred = [0, 1, 0, 0, 1, 1]

accuracy = calculate_accuracy(y_true, y_pred)
print(f"Accuracy: {accuracy:.2f}")

Precision: Measuring Positive Predictive Value

Precision is the ratio of true positive predictions to the total number of positive predictions. It answers the question: "Of all instances predicted as positive, how many are actually positive?" Precision is crucial in scenarios where false positives are costly.

from sklearn.metrics import precision_score

def calculate_precision(y_true, y_pred):
    return precision_score(y_true, y_pred)

# Example usage
y_true = [0, 1, 1, 0, 1, 0]
y_pred = [0, 1, 0, 0, 1, 1]

precision = calculate_precision(y_true, y_pred)
print(f"Precision: {precision:.2f}")

Recall: Measuring Sensitivity

Recall, also known as sensitivity or true positive rate, is the ratio of true positive predictions to the total number of actual positive instances. It answers the question: "Of all actual positive instances, how many did we correctly identify?" Recall is important when missing positive instances is costly.

from sklearn.metrics import recall_score

def calculate_recall(y_true, y_pred):
    return recall_score(y_true, y_pred)

# Example usage
y_true = [0, 1, 1, 0, 1, 0]
y_pred = [0, 1, 0, 0, 1, 1]

recall = calculate_recall(y_true, y_pred)
print(f"Recall: {recall:.2f}")

F1 Score: Balancing Precision and Recall

The F1 score is the harmonic mean of precision and recall, providing a single score that balances both metrics. It's particularly useful when you have an uneven class distribution and want to find an optimal balance between precision and recall.

from sklearn.metrics import f1_score

def calculate_f1(y_true, y_pred):
    return f1_score(y_true, y_pred)

# Example usage
y_true = [0, 1, 1, 0, 1, 0]
y_pred = [0, 1, 0, 0, 1, 1]

f1 = calculate_f1(y_true, y_pred)
print(f"F1 Score: {f1:.2f}")

Confusion Matrix: A Comprehensive View

A confusion matrix provides a tabular summary of a classifier's performance, showing the counts of true positives, true negatives, false positives, and false negatives. It's a powerful tool for understanding the types of errors your model is making.

import numpy as np
from sklearn.metrics import confusion_matrix
import matplotlib.pyplot as plt
import seaborn as sns

def plot_confusion_matrix(y_true, y_pred):
    cm = confusion_matrix(y_true, y_pred)
    plt.figure(figsize=(8, 6))
    sns.heatmap(cm, annot=True, fmt='d', cmap='Blues')
    plt.xlabel('Predicted')
    plt.ylabel('Actual')
    plt.title('Confusion Matrix')
    plt.show()

# Example usage
y_true = [0, 1, 1, 0, 1, 0, 1, 1, 0, 0]
y_pred = [0, 1, 0, 0, 1, 1, 1, 1, 0, 1]

plot_confusion_matrix(y_true, y_pred)

ROC Curve and AUC: Evaluating Binary Classifiers

The Receiver Operating Characteristic (ROC) curve and the Area Under the Curve (AUC) are powerful tools for evaluating binary classifiers. The ROC curve plots the True Positive Rate against the False Positive Rate at various threshold settings, while the AUC provides a single score summarizing the classifier's performance.

from sklearn.metrics import roc_curve, auc
import matplotlib.pyplot as plt

def plot_roc_curve(y_true, y_scores):
    fpr, tpr, _ = roc_curve(y_true, y_scores)
    roc_auc = auc(fpr, tpr)
    
    plt.figure(figsize=(8, 6))
    plt.plot(fpr, tpr, color='darkorange', lw=2, label=f'ROC curve (AUC = {roc_auc:.2f})')
    plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--')
    plt.xlim([0.0, 1.0])
    plt.ylim([0.0, 1.05])
    plt.xlabel('False Positive Rate')
    plt.ylabel('True Positive Rate')
    plt.title('Receiver Operating Characteristic (ROC) Curve')
    plt.legend(loc="lower right")
    plt.show()

# Example usage
y_true = [0, 1, 1, 0, 1, 0, 1, 1, 0, 0]
y_scores = [0.1, 0.9, 0.8, 0.3, 0.7, 0.6, 0.9, 0.8, 0.2, 0.5]

plot_roc_curve(y_true, y_scores)

Mean Squared Error and Root Mean Squared Error

Mean Squared Error (MSE) and Root Mean Squared Error (RMSE) are common metrics for regression problems. MSE measures the average squared difference between predicted and actual values, while RMSE is the square root of MSE, providing a measure in the same units as the target variable.

import numpy as np
from sklearn.metrics import mean_squared_error

def calculate_mse_rmse(y_true, y_pred):
    mse = mean_squared_error(y_true, y_pred)
    rmse = np.sqrt(mse)
    return mse, rmse

# Example usage
y_true = [3, 2, 5, 1, 7]
y_pred = [2.5, 3.0, 4.5, 1.5, 6.5]

mse, rmse = calculate_mse_rmse(y_true, y_pred)
print(f"Mean Squared Error: {mse:.2f}")
print(f"Root Mean Squared Error: {rmse:.2f}")

R-squared (Coefficient of Determination)

R-squared, also known as the coefficient of determination, measures the proportion of variance in the dependent variable that is predictable from the independent variable(s). It provides an indication of how well the model fits the data, with values ranging from 0 to 1.

from sklearn.metrics import r2_score

def calculate_r2(y_true, y_pred):
    return r2_score(y_true, y_pred)

# Example usage
y_true = [3, 2, 5, 1, 7]
y_pred = [2.5, 3.0, 4.5, 1.5, 6.5]

r2 = calculate_r2(y_true, y_pred)
print(f"R-squared: {r2:.2f}")

Cross-Validation: Assessing Model Generalization

Cross-validation is a technique used to assess how well a model generalizes to unseen data. It involves partitioning the data into subsets, training the model on a subset, and validating it on the remaining data. This process is repeated multiple times to get a robust estimate of the model's performance.

from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification

# Generate a random classification dataset
X, y = make_classification(n_samples=1000, n_features=20, n_classes=2, random_state=42)

# Create a logistic regression model
model = LogisticRegression()

# Perform 5-fold cross-validation
scores = cross_val_score(model, X, y, cv=5, scoring='accuracy')

print("Cross-validation scores:", scores)
print(f"Mean accuracy: {scores.mean():.2f} (+/- {scores.std() * 2:.2f})")

Learning Curves: Diagnosing Bias and Variance

Learning curves are a useful tool for diagnosing whether a model is suffering from high bias (underfitting) or high variance (overfitting). By plotting the model's performance on both training and validation sets as a function of the training set size, we can gain insights into the model's learning behavior.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import learning_curve
from sklearn.svm import SVC
from sklearn.datasets import load_digits

# Load the digits dataset
digits = load_digits()
X, y = digits.data, digits.target

# Create an SVM classifier
model = SVC(kernel='rbf', gamma=0.001)

# Calculate learning curves
train_sizes, train_scores, valid_scores = learning_curve(
    model, X, y, train_sizes=np.linspace(0.1, 1.0, 5), cv=5)

# Calculate mean and standard deviation
train_scores_mean = np.mean(train_scores, axis=1)
train_scores_std = np.std(train_scores, axis=1)
valid_scores_mean = np.mean(valid_scores, axis=1)
valid_scores_std = np.std(valid_scores, axis=1)

# Plot learning curves
plt.figure(figsize=(10, 6))
plt.fill_between(train_sizes, train_scores_mean - train_scores_std,
                 train_scores_mean + train_scores_std, alpha=0.1, color="r")
plt.fill_between(train_sizes, valid_scores_mean - valid_scores_std,
                 valid_scores_mean + valid_scores_std, alpha=0.1, color="g")
plt.plot(train_sizes, train_scores_mean, 'o-', color="r", label="Training score")
plt.plot(train_sizes, valid_scores_mean, 'o-', color="g", label="Cross-validation score")
plt.xlabel("Training examples")
plt.ylabel("Score")
plt.legend(loc="best")
plt.title("Learning Curves")
plt.show()

Real-Life Example: Spam Email Classification

Let's consider a real-life example of evaluating a spam email classifier. We'll use various metrics to assess the model's performance in distinguishing between spam and non-spam emails.

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report, confusion_matrix
import seaborn as sns
import matplotlib.pyplot as plt

# Sample email data (content, label)
emails = [
    ("Get rich quick! Buy now!", "spam"),
    ("Meeting at 3pm tomorrow", "ham"),
    ("You've won a free iPhone!", "spam"),
    ("Project report due next week", "ham"),
    ("Discount on luxury watches", "spam"),
    ("Reminder: dentist appointment", "ham")
]

# Split data into features (X) and labels (y)
X, y = zip(*emails)

# Convert text to numerical features
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(X)

# Split data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Train a Naive Bayes classifier
model = MultinomialNB()
model.fit(X_train, y_train)

# Make predictions
y_pred = model.predict(X_test)

# Print classification report
print(classification_report(y_test, y_pred))

# Plot confusion matrix
cm = confusion_matrix(y_test, y_pred)
plt.figure(figsize=(8, 6))
sns.heatmap(cm, annot=True, fmt='d', cmap='Blues')
plt.xlabel('Predicted')
plt.ylabel('Actual')
plt.title('Confusion Matrix: Spam Email Classification')
plt.show()

Real-Life Example: Image Classification Performance

In this example, we'll evaluate the performance of an image classification model trained on a subset of the CIFAR-10 dataset. We'll use various metrics to assess how well the model distinguishes between different object categories.

import numpy as np
from tensorflow.keras.datasets import cifar10
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense
from tensorflow.keras.utils import to_categorical
from sklearn.metrics import classification_report, confusion_matrix
import matplotlib.pyplot as plt
import seaborn as sns

# Load and preprocess CIFAR-10 data
(X_train, y_train), (X_test, y_test) = cifar10.load_data()
X_train, X_test = X_train / 255.0, X_test / 255.0
y_train, y_test = to_categorical(y_train, 10), to_categorical(y_test, 10)

# Build and train a simple CNN model (simplified for brevity)
model = Sequential([
    Conv2D(32, (3, 3), activation='relu', input_shape=(32, 32, 3)),
    MaxPooling2D((2, 2)),
    Flatten(),
    Dense(64, activation='relu'),
    Dense(10, activation='softmax')
])
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
model.fit(X_train[:5000], y_train[:5000], epochs=5, validation_split=0.2, verbose=0)

# Evaluate the model
y_pred = model.predict(X_test[:1000])
y_pred_classes = np.argmax(y_pred, axis=1)
y_true_classes = np.argmax(y_test[:1000], axis=1)

# Print classification report
print(classification_report(y_true_classes, y_pred_classes))

# Plot confusion matrix
cm = confusion_matrix(y_true_classes, y_pred_classes)
plt.figure(figsize=(10, 8))
sns.heatmap(cm, annot=True, fmt='d', cmap='Blues')
plt.xlabel('Predicted')
plt.ylabel('Actual')
plt.title('Confusion Matrix: CIFAR-10 Image Classification')
plt.show()

Interpreting Evaluation Metrics

When interpreting evaluation metrics, it's crucial to consider the context of your problem. Different scenarios may prioritize different metrics:

1. In medical diagnosis, high recall might be crucial to avoid missing potential cases.
2. In content recommendation systems, precision could be more important to ensure relevance.
3. For balanced problems, accuracy and F1 score provide good overall performance indicators.
4. ROC AUC is useful for comparing different models, especially with imbalanced datasets.

Remember that no single metric tells the whole story. It's often beneficial to consider multiple metrics together for a comprehensive evaluation.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score

def plot_metric_comparison(y_true, y_pred):
    metrics = [
        ('Accuracy', accuracy_score(y_true, y_pred)),
        ('Precision', precision_score(y_true, y_pred, average='weighted')),
        ('Recall', recall_score(y_true, y_pred, average='weighted')),
        ('F1 Score', f1_score(y_true, y_pred, average='weighted'))
    ]
    
    names, values = zip(*metrics)
    
    plt.figure(figsize=(10, 6))
    plt.bar(names, values)
    plt.ylim(0, 1)
    plt.title('Comparison of Evaluation Metrics')
    plt.ylabel('Score')
    for i, v in enumerate(values):
        plt.text(i, v + 0.01, f'{v:.2f}', ha='center')
    plt.show()

# Example usage
y_true = np.array([0, 1, 2, 0, 1, 2, 0, 2, 1, 0])
y_pred = np.array([0, 2, 1, 0, 1, 2, 0, 1, 1, 0])

plot_metric_comparison(y_true, y_pred)

Additional Resources

For those interested in diving deeper into model evaluation metrics and techniques, here are some valuable resources:

1. "A Survey of Evaluation Metrics Used for NLP Tasks" by S. Sharma et al. (2021) ArXiv: https://arxiv.org/abs/2108.12197
2. "Metrics to Evaluate Machine Learning Models" by S. Raschka (2020) ArXiv: https://arxiv.org/abs/2002.06325
3. "Beyond Accuracy: Precision and Recall" by J. Brownlee Available at: https://machinelearningmastery.com/classification-accuracy-is-not-enough-more-performance-measures-you-can-use/

These resources provide in-depth discussions on various evaluation metrics, their strengths, limitations, and appropriate use cases in different machine learning scenarios.