xbe.at

Machine learning / Model evaluation and validation

Evaluation Metrics for ML and AI Models in Python

7 min read, 14 sections

Contents
  1. Introduction to Evaluation Metrics
  2. Accuracy
  3. Precision
  4. Recall
  5. F1 Score
  6. Confusion Matrix
  7. ROC Curve and AUC
  8. Mean Squared Error (MSE)
  9. R-squared (Coefficient of Determination)
  10. Cross-Entropy Loss
  11. Real-Life Example: Spam Detection
  12. Real-Life Example: Image Classification
  13. Choosing the Right Metric
  14. Additional Resources

Introduction to Evaluation Metrics

Evaluation metrics are crucial for assessing the performance of machine learning and AI models. They provide quantitative measures to compare different models and guide the improvement process. In this presentation, we'll explore various evaluation metrics and how to implement them using Python.

import numpy as np
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score

# Example predictions and true labels
y_true = np.array([0, 1, 1, 0, 1, 0])
y_pred = np.array([0, 1, 0, 0, 1, 1])

# Calculate basic metrics
accuracy = accuracy_score(y_true, y_pred)
precision = precision_score(y_true, y_pred)
recall = recall_score(y_true, y_pred)
f1 = f1_score(y_true, y_pred)

print(f"Accuracy: {accuracy:.2f}")
print(f"Precision: {precision:.2f}")
print(f"Recall: {recall:.2f}")
print(f"F1 Score: {f1:.2f}")

Accuracy

Accuracy is the most straightforward metric, measuring the proportion of correct predictions among the total number of cases examined. It's suitable for balanced datasets but can be misleading for imbalanced ones.

from sklearn.metrics import accuracy_score

y_true = [0, 1, 1, 0, 1, 0, 1, 1, 0, 1]
y_pred = [0, 1, 1, 0, 1, 1, 0, 1, 0, 1]

accuracy = accuracy_score(y_true, y_pred)
print(f"Accuracy: {accuracy:.2f}")

# Output: Accuracy: 0.80

Precision

Precision measures the proportion of true positive predictions among all positive predictions. It's particularly useful when the cost of false positives is high.

from sklearn.metrics import precision_score

y_true = [0, 1, 1, 0, 1, 0, 1, 1, 0, 1]
y_pred = [1, 1, 1, 0, 1, 1, 0, 1, 1, 1]

precision = precision_score(y_true, y_pred)
print(f"Precision: {precision:.2f}")

# Output: Precision: 0.75

Recall

Recall, also known as sensitivity or true positive rate, measures the proportion of actual positive cases that were correctly identified. It's crucial when the cost of false negatives is high.

from sklearn.metrics import recall_score

y_true = [0, 1, 1, 0, 1, 0, 1, 1, 0, 1]
y_pred = [1, 1, 1, 0, 1, 1, 0, 1, 1, 1]

recall = recall_score(y_true, y_pred)
print(f"Recall: {recall:.2f}")

# Output: Recall: 0.86

F1 Score

The F1 score is the harmonic mean of precision and recall, providing a single score that balances both metrics. It's particularly useful for imbalanced datasets.

from sklearn.metrics import f1_score

y_true = [0, 1, 1, 0, 1, 0, 1, 1, 0, 1]
y_pred = [1, 1, 1, 0, 1, 1, 0, 1, 1, 1]

f1 = f1_score(y_true, y_pred)
print(f"F1 Score: {f1:.2f}")

# Output: F1 Score: 0.80

Confusion Matrix

A confusion matrix provides a tabular summary of a classifier's performance, showing the counts of true positives, true negatives, false positives, and false negatives.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.metrics import confusion_matrix
import seaborn as sns

y_true = [0, 1, 1, 0, 1, 0, 1, 1, 0, 1]
y_pred = [1, 1, 1, 0, 1, 1, 0, 1, 1, 1]

cm = confusion_matrix(y_true, y_pred)

plt.figure(figsize=(8, 6))
sns.heatmap(cm, annot=True, fmt='d', cmap='Blues')
plt.xlabel('Predicted')
plt.ylabel('True')
plt.title('Confusion Matrix')
plt.show()

ROC Curve and AUC

The Receiver Operating Characteristic (ROC) curve and the Area Under the Curve (AUC) are used to evaluate the performance of a binary classifier across various thresholds.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.metrics import roc_curve, auc

y_true = np.array([0, 1, 1, 0, 1, 0, 1, 1, 0, 1])
y_scores = np.array([0.1, 0.9, 0.8, 0.3, 0.7, 0.6, 0.2, 0.9, 0.5, 0.8])

fpr, tpr, thresholds = roc_curve(y_true, y_scores)
roc_auc = auc(fpr, tpr)

plt.figure()
plt.plot(fpr, tpr, color='darkorange', lw=2, label=f'ROC curve (AUC = {roc_auc:.2f})')
plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver Operating Characteristic (ROC) Curve')
plt.legend(loc="lower right")
plt.show()

Mean Squared Error (MSE)

Mean Squared Error is a common metric for regression problems, measuring the average squared difference between predicted and actual values.

import numpy as np
from sklearn.metrics import mean_squared_error

y_true = np.array([3, -0.5, 2, 7])
y_pred = np.array([2.5, 0.0, 2, 8])

mse = mean_squared_error(y_true, y_pred)
print(f"Mean Squared Error: {mse:.2f}")

# Output: Mean Squared Error: 0.375

R-squared (Coefficient of Determination)

R-squared measures the proportion of variance in the dependent variable that is predictable from the independent variable(s) in a regression model.

import numpy as np
from sklearn.metrics import r2_score

y_true = np.array([3, -0.5, 2, 7, 4.2])
y_pred = np.array([2.5, 0.0, 2, 8, 4.5])

r2 = r2_score(y_true, y_pred)
print(f"R-squared: {r2:.2f}")

# Output: R-squared: 0.97

Cross-Entropy Loss

Cross-entropy loss, also known as log loss, measures the performance of a classification model whose output is a probability value between 0 and 1.

import numpy as np

def cross_entropy_loss(y_true, y_pred):
    epsilon = 1e-15
    y_pred = np.clip(y_pred, epsilon, 1 - epsilon)
    return -np.mean(y_true * np.log(y_pred) + (1 - y_true) * np.log(1 - y_pred))

y_true = np.array([1, 0, 1, 1, 0])
y_pred = np.array([0.9, 0.1, 0.8, 0.9, 0.2])

loss = cross_entropy_loss(y_true, y_pred)
print(f"Cross-Entropy Loss: {loss:.4f}")

# Output: Cross-Entropy Loss: 0.2150

Real-Life Example: Spam Detection

In a spam detection system, precision and recall are crucial metrics. High precision ensures that legitimate emails aren't marked as spam, while high recall ensures that most spam is caught.

from sklearn.metrics import precision_recall_fscore_support

# Simulated results from a spam detection model
y_true = [1, 0, 1, 1, 0, 1, 0, 1, 1, 0]  # 1: spam, 0: not spam
y_pred = [1, 0, 1, 1, 1, 0, 0, 1, 1, 0]

precision, recall, f1, _ = precision_recall_fscore_support(y_true, y_pred, average='binary')

print(f"Precision: {precision:.2f}")
print(f"Recall: {recall:.2f}")
print(f"F1 Score: {f1:.2f}")

# Output:
# Precision: 0.83
# Recall: 0.83
# F1 Score: 0.83

Real-Life Example: Image Classification

In image classification tasks, such as identifying objects in photographs, accuracy and confusion matrices are commonly used to evaluate model performance.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.metrics import confusion_matrix, accuracy_score
import seaborn as sns

# Simulated results from an image classification model
classes = ['dog', 'cat', 'bird']
y_true = ['dog', 'cat', 'bird', 'dog', 'cat', 'bird', 'dog', 'cat', 'bird']
y_pred = ['dog', 'cat', 'dog', 'dog', 'cat', 'bird', 'cat', 'cat', 'bird']

accuracy = accuracy_score(y_true, y_pred)
cm = confusion_matrix(y_true, y_pred, labels=classes)

plt.figure(figsize=(8, 6))
sns.heatmap(cm, annot=True, fmt='d', cmap='Blues', xticklabels=classes, yticklabels=classes)
plt.xlabel('Predicted')
plt.ylabel('True')
plt.title(f'Confusion Matrix (Accuracy: {accuracy:.2f})')
plt.show()

Choosing the Right Metric

Selecting the appropriate evaluation metric depends on the specific problem, dataset characteristics, and business objectives. Consider the following factors:

  1. Class balance: For imbalanced datasets, prefer metrics like F1 score or AUC over accuracy.
  2. Cost of errors: If false positives are more costly, focus on precision; if false negatives are more critical, prioritize recall.
  3. Problem type: Use classification metrics for categorical outcomes and regression metrics for continuous outcomes.
  4. Interpretability: Some metrics, like accuracy, are easier to explain to non-technical stakeholders.
  5. Model comparison: Consistent use of metrics allows for fair comparison between different models.
import numpy as np
import matplotlib.pyplot as plt
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score

# Generate random predictions and true labels
np.random.seed(42)
y_true = np.random.randint(2, size=1000)
y_pred = np.random.randint(2, size=1000)

# Calculate metrics
metrics = {
    'Accuracy': accuracy_score(y_true, y_pred),
    'Precision': precision_score(y_true, y_pred),
    'Recall': recall_score(y_true, y_pred),
    'F1 Score': f1_score(y_true, y_pred)
}

# Plot results
plt.figure(figsize=(10, 6))
plt.bar(metrics.keys(), metrics.values())
plt.title('Comparison of Different Evaluation Metrics')
plt.ylabel('Score')
plt.ylim(0, 1)
for i, v in enumerate(metrics.values()):
    plt.text(i, v + 0.01, f'{v:.2f}', ha='center')
plt.show()

Additional Resources

For those interested in diving deeper into evaluation metrics for ML and AI models, here are some recommended resources:

  1. "A Survey of Evaluation Metrics Used for NLP Tasks" by S. Sharma et al. (2021) - ArXiv:2108.03302
  2. "Evaluation Metrics for Language Models" by S. Merity (2019) - ArXiv:1912.00607
  3. "Beyond Accuracy: Behavioral Testing of NLP Models with CheckList" by M. T. Ribeiro et al. (2020) - ArXiv:2005.04118
  4. "A Unified View of Performance Metrics: Translating Threshold Choice into Expected Classification Loss" by J. Hernández-Orallo et al. (2012) - ArXiv:1202.5597

These papers provide in-depth discussions on various evaluation metrics, their applications, and limitations in different ML and AI domains.

More in Model evaluation and validation

  1. Explaining Leave-One-Out Cross-Validation with Python14 min

    Leave-One-Out Cross-Validation is a technique used in machine learning to assess model performance and generalization. It's a special case of k-fold...

  2. Evaluating Regression Model Performance Metrics12 min

    Mean Squared Error (MSE) and Root Mean Squared Error (RMSE) are fundamental metrics for evaluating regression models. MSE measures the average squared...

  3. Explaining ROC Curve and AUC in Model Evaluation with Python14 min

    The Receiver Operating Characteristic (ROC) curve and Area Under the Curve (AUC) are essential tools for evaluating binary classification models. They help...

  4. Evaluating Regression Model Metrics in Python7 min

    Regression models are essential tools in predictive analytics. To ensure their effectiveness, we need reliable metrics to assess their performance. This...

  5. Explaining the F1 Score for Binary Classification in Python7 min

    The F1 score is a powerful metric for evaluating binary classification models. It combines precision and recall into a single value, providing a balanced...

  6. Evaluating Model Generalization with Cross-Validation9 min

    Cross-validation is a statistical method used to assess model performance by partitioning data into training and testing sets. It helps evaluate how well a...