xbe.at

Machine learning / Model evaluation and validation

Understanding ROC Curve AUC

10 min read, 15 sections

Contents
  1. Understanding ROC AUC
  2. Mathematical Foundation of AUC
  3. Implementing a Random Classifier
  4. Visualizing AUC Distribution
  5. Comparison with Perfect Classifier
  6. Practical Implementation with Real Data
  7. Statistical Significance Testing
  8. Calibration Analysis
  9. Cross-Validation Analysis
  10. ROC Curve Decomposition
  11. Real-world Example: Credit Card Fraud Detection
  12. Precision-Recall Analysis for AUC=0.5
  13. Cost Analysis of Random Performance
  14. Density Analysis of Predictions
  15. Additional Resources

Understanding ROC AUC

An Area Under the Curve (AUC) of 0.5 in a Receiver Operating Characteristic (ROC) curve indicates that the classifier performs no better than random guessing. This represents a diagonal line from (0,0) to (1,1) in the ROC space, demonstrating zero discriminative ability.

import numpy as np
from sklearn.metrics import roc_curve, auc
import matplotlib.pyplot as plt

# Generate random predictions (random classifier)
np.random.seed(42)
y_true = np.random.binomial(1, 0.5, 1000)
y_pred_random = np.random.rand(1000)

# Calculate ROC curve and AUC
fpr, tpr, _ = roc_curve(y_true, y_pred_random)
roc_auc = auc(fpr, tpr)

# Plot ROC curve
plt.figure(figsize=(8, 6))
plt.plot(fpr, tpr, color='darkorange', lw=2, label=f'ROC curve (AUC = {roc_auc:.2f})')
plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--', label='Random (AUC = 0.5)')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC Curve: Random Classifier')
plt.legend(loc='lower right')
plt.show()

Mathematical Foundation of AUC

The AUC represents the probability that a randomly chosen positive instance is ranked higher than a randomly chosen negative instance. For a random classifier, this probability is 0.5, mathematically expressed through integration.

# Mathematical representation of AUC calculation
'''
The AUC can be expressed mathematically as:

$$
AUC = \int_{0}^{1} TPR(FPR^{-1}(x)) dx
$$

For a random classifier:

$$
AUC = \int_{0}^{1} x dx = 0.5
$$
'''

# Numerical approximation of AUC
def calculate_auc_numerical(y_true, y_scores, num_points=1000):
    fpr, tpr, _ = roc_curve(y_true, y_scores)
    return np.trapz(tpr, fpr)  # Numerical integration using trapezoidal rule

Implementing a Random Classifier

A random classifier assigns random probabilities to instances, demonstrating the baseline performance level. This implementation shows how random predictions consistently achieve an AUC score around 0.5 across multiple runs.

import numpy as np
from sklearn.metrics import roc_auc_score

def random_classifier_experiment(n_samples=1000, n_experiments=100):
    auc_scores = []
    
    for _ in range(n_experiments):
        # Generate random true labels and predictions
        y_true = np.random.binomial(1, 0.5, n_samples)
        y_pred = np.random.rand(n_samples)
        
        # Calculate AUC score
        auc = roc_auc_score(y_true, y_pred)
        auc_scores.append(auc)
    
    print(f"Mean AUC: {np.mean(auc_scores):.3f}")
    print(f"Std AUC: {np.std(auc_scores):.3f}")
    return auc_scores

# Run experiment
results = random_classifier_experiment()

Visualizing AUC Distribution

The distribution of AUC scores for random classifiers follows a normal distribution centered around 0.5. This visualization demonstrates the statistical nature of random classification and its inherent limitations.

import seaborn as sns

def plot_auc_distribution(auc_scores):
    plt.figure(figsize=(10, 6))
    sns.histplot(auc_scores, bins=30, kde=True)
    plt.axvline(x=0.5, color='r', linestyle='--', label='Perfect Random (AUC=0.5)')
    plt.xlabel('AUC Score')
    plt.ylabel('Frequency')
    plt.title('Distribution of AUC Scores for Random Classifier')
    plt.legend()
    
    # Calculate confidence intervals
    ci_lower = np.percentile(auc_scores, 2.5)
    ci_upper = np.percentile(auc_scores, 97.5)
    print(f"95% Confidence Interval: [{ci_lower:.3f}, {ci_upper:.3f}]")
    
    plt.show()

# Plot distribution using previous results
plot_auc_distribution(results)

Comparison with Perfect Classifier

To understand the significance of AUC=0.5, we compare random classification with perfect classification (AUC=1.0). This demonstrates the full spectrum of classifier performance and proper separation capabilities.

def compare_classifiers(n_samples=1000):
    # Generate true labels
    y_true = np.random.binomial(1, 0.5, n_samples)
    
    # Generate predictions for different classifiers
    y_pred_random = np.random.rand(n_samples)
    y_pred_perfect = y_true  # Perfect predictions
    
    # Calculate ROC curves
    fpr_random, tpr_random, _ = roc_curve(y_true, y_pred_random)
    fpr_perfect, tpr_perfect, _ = roc_curve(y_true, y_pred_perfect)
    
    # Calculate AUC scores
    auc_random = auc(fpr_random, tpr_random)
    auc_perfect = auc(fpr_perfect, tpr_perfect)
    
    # Plotting
    plt.figure(figsize=(10, 8))
    plt.plot(fpr_random, tpr_random, label=f'Random (AUC = {auc_random:.2f})')
    plt.plot(fpr_perfect, tpr_perfect, label=f'Perfect (AUC = {auc_perfect:.2f})')
    plt.plot([0, 1], [0, 1], 'k--')
    plt.xlabel('False Positive Rate')
    plt.ylabel('True Positive Rate')
    plt.title('ROC Curves: Random vs Perfect Classifier')
    plt.legend()
    plt.show()

compare_classifiers()

Practical Implementation with Real Data

Using the popular breast cancer dataset, we demonstrate how a poorly trained model can approach random classification performance with AUC near 0.5, highlighting the importance of proper model configuration.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler

# Load and prepare data
data = load_breast_cancer()
X, y = data.data, data.target

# Split and scale data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Create deliberately poor model (high regularization)
poor_model = LogisticRegression(C=0.0001, random_state=42)
poor_model.fit(X_train_scaled, y_train)

# Calculate predictions and AUC
y_pred_proba = poor_model.predict_proba(X_test_scaled)[:, 1]
poor_auc = roc_auc_score(y_test, y_pred_proba)
print(f"Poor Model AUC: {poor_auc:.3f}")

Statistical Significance Testing

When AUC is close to 0.5, it's crucial to perform statistical testing to determine if the classifier is truly random. This implementation uses bootstrap resampling to establish confidence intervals.

def bootstrap_auc_test(y_true, y_pred, n_iterations=1000, ci=0.95):
    auc_scores = []
    n_samples = len(y_true)
    
    for _ in range(n_iterations):
        # Bootstrap sampling
        indices = np.random.randint(0, n_samples, n_samples)
        sample_true = y_true[indices]
        sample_pred = y_pred[indices]
        
        # Calculate AUC for bootstrap sample
        auc_scores.append(roc_auc_score(sample_true, sample_pred))
    
    # Calculate confidence intervals
    ci_lower = np.percentile(auc_scores, ((1-ci)/2)*100)
    ci_upper = np.percentile(auc_scores, (1-(1-ci)/2)*100)
    
    return {
        'mean_auc': np.mean(auc_scores),
        'ci_lower': ci_lower,
        'ci_upper': ci_upper,
        'p_value': np.mean(np.array(auc_scores) <= 0.5)
    }

# Test the poor model
test_results = bootstrap_auc_test(y_test, y_pred_proba)
print(f"Mean AUC: {test_results['mean_auc']:.3f}")
print(f"95% CI: [{test_results['ci_lower']:.3f}, {test_results['ci_upper']:.3f}]")
print(f"p-value: {test_results['p_value']:.3f}")

Calibration Analysis

A classifier with AUC=0.5 might still have useful probability estimates. This implementation analyzes probability calibration to assess the reliability of predictions despite poor discrimination.

from sklearn.calibration import calibration_curve

def analyze_calibration(y_true, y_prob, n_bins=10):
    # Calculate calibration curve
    prob_true, prob_pred = calibration_curve(y_true, y_prob, n_bins=n_bins)
    
    # Plot calibration curve
    plt.figure(figsize=(8, 6))
    plt.plot([0, 1], [0, 1], 'k--', label='Perfectly calibrated')
    plt.plot(prob_pred, prob_true, 'ro-', label='Model calibration')
    plt.xlabel('Mean predicted probability')
    plt.ylabel('Fraction of positives')
    plt.title('Calibration Curve Analysis')
    plt.legend()
    
    # Calculate calibration metrics
    expected_accuracy = np.mean(np.abs(prob_pred - prob_true))
    print(f"Mean calibration error: {expected_accuracy:.3f}")
    
    return expected_accuracy

# Analyze calibration of the poor model
calibration_error = analyze_calibration(y_test, y_pred_proba)

Cross-Validation Analysis

To ensure that an AUC of 0.5 is not due to data splitting artifacts, we implement stratified k-fold cross-validation with multiple metrics to validate the random performance.

from sklearn.model_selection import StratifiedKFold
from sklearn.metrics import precision_score, recall_score

def comprehensive_cv_analysis(X, y, model, n_splits=5):
    skf = StratifiedKFold(n_splits=n_splits, shuffle=True, random_state=42)
    metrics = {
        'auc': [], 'precision': [], 'recall': []
    }
    
    for train_idx, val_idx in skf.split(X, y):
        # Split data
        X_train, X_val = X[train_idx], X[val_idx]
        y_train, y_val = y[train_idx], y[val_idx]
        
        # Train and predict
        model.fit(X_train, y_train)
        y_pred_proba = model.predict_proba(X_val)[:, 1]
        y_pred = model.predict(X_val)
        
        # Calculate metrics
        metrics['auc'].append(roc_auc_score(y_val, y_pred_proba))
        metrics['precision'].append(precision_score(y_val, y_pred))
        metrics['recall'].append(recall_score(y_val, y_pred))
    
    # Print results
    for metric, values in metrics.items():
        print(f"{metric.upper()}: {np.mean(values):.3f} ± {np.std(values):.3f}")
    
    return metrics

# Run cross-validation analysis
X_scaled = scaler.fit_transform(X)
cv_results = comprehensive_cv_analysis(X_scaled, y, poor_model)

ROC Curve Decomposition

Understanding how an AUC of 0.5 emerges requires analyzing the relationship between True Positive Rate and False Positive Rate across different classification thresholds. This implementation provides detailed threshold analysis.

def analyze_roc_thresholds(y_true, y_pred_proba, n_thresholds=50):
    thresholds = np.linspace(0, 1, n_thresholds)
    metrics = {
        'threshold': [], 'tpr': [], 'fpr': [], 
        'ratio': []  # TPR/FPR ratio
    }
    
    for threshold in thresholds:
        y_pred = (y_pred_proba >= threshold).astype(int)
        tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
        
        tpr = tp / (tp + fn) if (tp + fn) > 0 else 0
        fpr = fp / (fp + tn) if (fp + tn) > 0 else 0
        
        metrics['threshold'].append(threshold)
        metrics['tpr'].append(tpr)
        metrics['fpr'].append(fpr)
        metrics['ratio'].append(tpr/fpr if fpr > 0 else np.inf)
    
    # Plotting
    fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15, 6))
    
    ax1.plot(metrics['threshold'], metrics['tpr'], label='TPR')
    ax1.plot(metrics['threshold'], metrics['fpr'], label='FPR')
    ax1.set_xlabel('Threshold')
    ax1.set_ylabel('Rate')
    ax1.legend()
    ax1.set_title('TPR and FPR vs Threshold')
    
    valid_ratios = [r for r, f in zip(metrics['ratio'], metrics['fpr']) 
                   if f > 0 and r != np.inf]
    ax2.plot(metrics['threshold'][:len(valid_ratios)], valid_ratios)
    ax2.set_xlabel('Threshold')
    ax2.set_ylabel('TPR/FPR Ratio')
    ax2.set_title('TPR/FPR Ratio vs Threshold')
    
    plt.tight_layout()
    return metrics

Real-world Example: Credit Card Fraud Detection

Implementing a deliberately randomized model for credit card fraud detection to demonstrate how AUC=0.5 manifests in a critical real-world scenario with imbalanced classes.

def simulate_credit_fraud_detection(n_samples=10000):
    # Generate synthetic imbalanced dataset
    n_frauds = int(n_samples * 0.001)  # 0.1% fraud rate
    
    # Generate normal transactions
    normal_features = np.random.normal(0, 1, (n_samples - n_frauds, 10))
    normal_labels = np.zeros(n_samples - n_frauds)
    
    # Generate fraudulent transactions
    fraud_features = np.random.normal(0, 1, (n_frauds, 10))
    fraud_labels = np.ones(n_frauds)
    
    # Combine datasets
    X = np.vstack([normal_features, fraud_features])
    y = np.hstack([normal_labels, fraud_labels])
    
    # Random shuffling
    shuffle_idx = np.random.permutation(len(X))
    X, y = X[shuffle_idx], y[shuffle_idx]
    
    # Split and evaluate
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
    
    # Train a deliberately poor model
    model = LogisticRegression(class_weight='balanced', C=0.0001)
    model.fit(X_train, y_train)
    
    y_pred_proba = model.predict_proba(X_test)[:, 1]
    auc_score = roc_auc_score(y_test, y_pred_proba)
    
    print(f"AUC Score: {auc_score:.3f}")
    return X_test, y_test, y_pred_proba

# Run simulation
X_test, y_test, y_pred_proba = simulate_credit_fraud_detection()

Precision-Recall Analysis for AUC=0.5

For imbalanced datasets, Precision-Recall curves provide additional insight into the implications of random classification beyond ROC AUC=0.5.

from sklearn.metrics import precision_recall_curve, average_precision_score

def analyze_pr_curve(y_true, y_pred_proba):
    # Calculate PR curve
    precision, recall, thresholds = precision_recall_curve(y_true, y_pred_proba)
    ap_score = average_precision_score(y_true, y_pred_proba)
    
    # Calculate baseline
    baseline = np.mean(y_true)
    
    # Plotting
    plt.figure(figsize=(8, 6))
    plt.plot(recall, precision, label=f'PR curve (AP = {ap_score:.3f})')
    plt.axhline(y=baseline, color='r', linestyle='--', 
                label=f'Random (AP = {baseline:.3f})')
    plt.xlabel('Recall')
    plt.ylabel('Precision')
    plt.title('Precision-Recall Curve Analysis')
    plt.legend()
    
    # Calculate metrics at different operating points
    operating_points = [0.1, 0.3, 0.5, 0.7, 0.9]
    metrics = []
    
    for threshold in operating_points:
        y_pred = (y_pred_proba >= threshold).astype(int)
        prec = precision_score(y_true, y_pred)
        rec = recall_score(y_true, y_pred)
        metrics.append({
            'threshold': threshold,
            'precision': prec,
            'recall': rec
        })
    
    return metrics

# Analyze PR curve
pr_metrics = analyze_pr_curve(y_test, y_pred_proba)

Cost Analysis of Random Performance

Understanding the economic implications of a classifier with AUC=0.5 through cost matrix analysis and expected value calculations in real-world scenarios.

def cost_analysis(y_true, y_pred_proba, cost_matrix=None):
    if cost_matrix is None:
        # Default cost matrix for fraud detection
        cost_matrix = {
            'tn': 0,      # Correct rejection
            'fp': 100,    # False alarm cost
            'fn': 1000,   # Missed fraud cost
            'tp': -500    # Fraud prevention savings
        }
    
    thresholds = np.linspace(0, 1, 100)
    results = {
        'threshold': [],
        'total_cost': [],
        'cost_per_transaction': []
    }
    
    for threshold in thresholds:
        y_pred = (y_pred_proba >= threshold).astype(int)
        tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
        
        total_cost = (
            tn * cost_matrix['tn'] +
            fp * cost_matrix['fp'] +
            fn * cost_matrix['fn'] +
            tp * cost_matrix['tp']
        )
        
        results['threshold'].append(threshold)
        results['total_cost'].append(total_cost)
        results['cost_per_transaction'].append(total_cost / len(y_true))
    
    # Plot cost analysis
    plt.figure(figsize=(10, 6))
    plt.plot(results['threshold'], results['cost_per_transaction'])
    plt.xlabel('Classification Threshold')
    plt.ylabel('Cost per Transaction')
    plt.title('Cost Analysis of Random Classification')
    
    # Find optimal threshold
    optimal_idx = np.argmin(results['cost_per_transaction'])
    optimal_threshold = results['threshold'][optimal_idx]
    optimal_cost = results['cost_per_transaction'][optimal_idx]
    
    plt.axvline(x=optimal_threshold, color='r', linestyle='--',
                label=f'Optimal Threshold: {optimal_threshold:.2f}')
    plt.axhline(y=optimal_cost, color='g', linestyle='--',
                label=f'Minimum Cost: {optimal_cost:.2f}')
    plt.legend()
    
    return results, optimal_threshold, optimal_cost

# Perform cost analysis
results, opt_threshold, opt_cost = cost_analysis(y_test, y_pred_proba)

Density Analysis of Predictions

Examining the distribution of predicted probabilities helps understand why a classifier achieves AUC=0.5 through visualization of class separation (or lack thereof).

def analyze_prediction_density(y_true, y_pred_proba):
    # Separate predictions by true class
    pos_preds = y_pred_proba[y_true == 1]
    neg_preds = y_pred_proba[y_true == 0]
    
    plt.figure(figsize=(10, 6))
    
    # Plot density for each class
    sns.kdeplot(pos_preds, label='Positive Class', color='blue', fill=True, alpha=0.3)
    sns.kdeplot(neg_preds, label='Negative Class', color='red', fill=True, alpha=0.3)
    
    # Calculate and plot means
    plt.axvline(np.mean(pos_preds), color='blue', linestyle='--',
                label=f'Positive Mean: {np.mean(pos_preds):.3f}')
    plt.axvline(np.mean(neg_preds), color='red', linestyle='--',
                label=f'Negative Mean: {np.mean(neg_preds):.3f}')
    
    # Calculate overlap coefficient
    bins = np.linspace(0, 1, 100)
    hist_pos = np.histogram(pos_preds, bins=bins, density=True)[0]
    hist_neg = np.histogram(neg_preds, bins=bins, density=True)[0]
    overlap = np.sum(np.minimum(hist_pos, hist_neg)) * (bins[1] - bins[0])
    
    plt.title(f'Prediction Density Analysis\nOverlap Coefficient: {overlap:.3f}')
    plt.xlabel('Predicted Probability')
    plt.ylabel('Density')
    plt.legend()
    
    return overlap

# Analyze prediction density
overlap = analyze_prediction_density(y_test, y_pred_proba)

Additional Resources

  1. "Understanding ROC curves and Area Under the Curve (AUC)" https://arxiv.org/abs/2006.11278
  2. "The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets" https://arxiv.org/abs/1504.06823
  3. "On the Relationship between Class Probability Estimates and ROC AUC" https://arxiv.org/abs/1805.11736
  4. "Cost-sensitive Learning Methods for Imbalanced Data" https://arxiv.org/abs/1901.09337
  5. "The Central Role of the Area Under the Curve (AUC) in Radiological Assessment" https://arxiv.org/abs/2008.09773

More in Model evaluation and validation

  1. Understanding Silhouette Score in Clustering7 min

    The silhouette score measures how similar an object is to its own cluster compared to other clusters. It ranges from -1 to +1, where higher values indicate...

  2. Understanding Confusion Matrices in Machine Learning8 min

    A confusion matrix is a fundamental evaluation metric in machine learning that tabulates the performance of a classification model by comparing predicted...

  3. Understanding the Silhouette Score in Clustering9 min

    The silhouette score is a metric used to evaluate the quality of clustering results. It measures how similar an object is to its own cluster compared to...

  4. The Dark Side of Cross-Validation in Python11 min

    Cross-validation is a widely used technique in machine learning for model evaluation and selection. However, it's not without its pitfalls. This presentation...

  5. Unlocking the Secrets of Model Performance with the Confusion Matrix8 min

    A confusion matrix is a fundamental evaluation tool in classification tasks that presents the relationships between predicted and actual class labels through...

  6. Stratified K-Fold Cross-Validation in Python11 min

    Stratified K-Fold cross-validation is an essential technique in machine learning for evaluating model performance. It addresses the limitations of simple...