xbe.at

Machine learning / Model evaluation and validation

The Dark Side of Cross-Validation in Python

11 min read, 15 sections

Contents
  1. The Dark Side of Cross-Validation
  2. Overfitting to the Validation Set
  3. Data Leakage
  4. Temporal Dependence
  5. Small Sample Sizes
  6. Computational Cost
  7. Bias-Variance Tradeoff in CV
  8. Inappropriate Performance Metrics
  9. Real-life Example: Image Classification
  10. Real-life Example: Time Series Forecasting
  11. Nested Cross-Validation
  12. Stratified Cross-Validation
  13. Cross-Validation with Grouped Data
  14. Cross-Validation in Small Datasets
  15. Additional Resources

The Dark Side of Cross-Validation

Cross-validation is a widely used technique in machine learning for model evaluation and selection. However, it's not without its pitfalls. This presentation will explore the potential drawbacks and limitations of cross-validation, providing insights into when and how it might lead to incorrect conclusions or suboptimal model choices.

import numpy as np
from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LinearRegression

# Generate sample data
np.random.seed(42)
X = np.random.rand(100, 1)
y = 2 * X + 1 + np.random.randn(100, 1) * 0.1

# Perform cross-validation
model = LinearRegression()
scores = cross_val_score(model, X, y, cv=5)

print(f"Cross-validation scores: {scores}")
print(f"Mean score: {scores.mean():.2f}")

Overfitting to the Validation Set

One of the primary issues with cross-validation is the risk of overfitting to the validation set. When we use cross-validation to select hyperparameters or model architectures, we may inadvertently tune our model to perform well on the validation set, leading to optimistic performance estimates.

from sklearn.model_selection import KFold
from sklearn.metrics import mean_squared_error

def cv_tune_model(X, y, n_splits=5):
    kf = KFold(n_splits=n_splits)
    best_score = float('inf')
    best_degree = 0
    
    for degree in range(1, 10):
        scores = []
        for train_index, val_index in kf.split(X):
            X_train, X_val = X[train_index], X[val_index]
            y_train, y_val = y[train_index], y[val_index]
            
            model = np.poly1d(np.polyfit(X_train.ravel(), y_train.ravel(), degree))
            score = mean_squared_error(y_val, model(X_val))
            scores.append(score)
        
        mean_score = np.mean(scores)
        if mean_score < best_score:
            best_score = mean_score
            best_degree = degree
    
    return best_degree

best_degree = cv_tune_model(X, y)
print(f"Best polynomial degree: {best_degree}")

Data Leakage

Data leakage occurs when information from the test set inadvertently influences the training process. In cross-validation, this can happen if preprocessing steps are performed on the entire dataset before splitting, leading to unrealistically good performance estimates.

from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split

# Incorrect approach (data leakage)
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.2)

# Correct approach
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

print("Incorrect approach: Test set influenced by training data")
print("Correct approach: Test set remains truly unseen")

Temporal Dependence

In time series data, standard cross-validation can lead to incorrect results due to temporal dependence. Using future data to predict past events violates the assumption of independence and can result in overly optimistic performance estimates.

import pandas as pd
from sklearn.model_selection import TimeSeriesSplit

# Generate time series data
dates = pd.date_range(start='2023-01-01', end='2023-12-31', freq='D')
values = np.cumsum(np.random.randn(len(dates))) + 100
ts_data = pd.DataFrame({'date': dates, 'value': values})

# Incorrect: standard K-Fold
kf = KFold(n_splits=5)
for train_index, test_index in kf.split(ts_data):
    print(f"Train: {ts_data.iloc[train_index]['date'].min()} to {ts_data.iloc[train_index]['date'].max()}")
    print(f"Test: {ts_data.iloc[test_index]['date'].min()} to {ts_data.iloc[test_index]['date'].max()}")
    print()

print("Note: Test data mixes with training data, violating temporal order")

# Correct: Time Series Split
tscv = TimeSeriesSplit(n_splits=5)
for train_index, test_index in tscv.split(ts_data):
    print(f"Train: {ts_data.iloc[train_index]['date'].min()} to {ts_data.iloc[train_index]['date'].max()}")
    print(f"Test: {ts_data.iloc[test_index]['date'].min()} to {ts_data.iloc[test_index]['date'].max()}")
    print()

print("Note: Test data always comes after training data, preserving temporal order")

Small Sample Sizes

Cross-validation can be unreliable with small datasets. The high variance in performance estimates can lead to incorrect model selection or overly optimistic/pessimistic evaluations.

from sklearn.datasets import make_classification
from sklearn.svm import SVC

def cv_small_sample(n_samples, n_splits=5):
    X, y = make_classification(n_samples=n_samples, n_features=20, random_state=42)
    model = SVC()
    scores = cross_val_score(model, X, y, cv=n_splits)
    return scores

sample_sizes = [20, 50, 100, 500, 1000]
for size in sample_sizes:
    scores = cv_small_sample(size)
    print(f"Sample size: {size}")
    print(f"CV scores: {scores}")
    print(f"Mean: {scores.mean():.2f}, Std: {scores.std():.2f}")
    print()

print("Note: Smaller sample sizes lead to higher variance in CV scores")

Computational Cost

Cross-validation can be computationally expensive, especially with large datasets or complex models. This can limit its applicability in scenarios where quick iterations are necessary.

import time
from sklearn.ensemble import RandomForestClassifier

def measure_cv_time(n_samples, n_estimators, n_splits=5):
    X, y = make_classification(n_samples=n_samples, n_features=20, random_state=42)
    model = RandomForestClassifier(n_estimators=n_estimators)
    
    start_time = time.time()
    scores = cross_val_score(model, X, y, cv=n_splits)
    end_time = time.time()
    
    return end_time - start_time

sample_sizes = [1000, 10000, 100000]
n_estimators_list = [10, 100, 1000]

for size in sample_sizes:
    for n_estimators in n_estimators_list:
        cv_time = measure_cv_time(size, n_estimators)
        print(f"Samples: {size}, Estimators: {n_estimators}")
        print(f"CV Time: {cv_time:.2f} seconds")
        print()

print("Note: CV time increases with dataset size and model complexity")

Bias-Variance Tradeoff in CV

Cross-validation involves a bias-variance tradeoff. Using more folds reduces bias but increases variance, while fewer folds do the opposite. This can lead to different conclusions depending on the chosen number of folds.

from sklearn.model_selection import KFold, cross_val_score
from sklearn.tree import DecisionTreeClassifier

def cv_bias_variance(n_samples, max_depth, n_splits_list):
    X, y = make_classification(n_samples=n_samples, n_features=20, random_state=42)
    model = DecisionTreeClassifier(max_depth=max_depth)
    
    results = []
    for n_splits in n_splits_list:
        scores = cross_val_score(model, X, y, cv=n_splits)
        results.append((n_splits, scores.mean(), scores.std()))
    
    return results

n_splits_list = [2, 5, 10, 20]
results = cv_bias_variance(1000, max_depth=5, n_splits_list=n_splits_list)

for n_splits, mean_score, std_score in results:
    print(f"Folds: {n_splits}")
    print(f"Mean Score: {mean_score:.3f}")
    print(f"Std Dev: {std_score:.3f}")
    print()

print("Note: More folds generally reduce bias but increase variance")

Inappropriate Performance Metrics

Choosing inappropriate performance metrics in cross-validation can lead to misleading results, especially in imbalanced datasets or when the cost of different types of errors varies.

from sklearn.metrics import accuracy_score, f1_score, precision_score, recall_score
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

# Create an imbalanced dataset
X, y = make_classification(n_samples=1000, n_classes=2, weights=[0.9, 0.1], random_state=42)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model = LogisticRegression()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

print("Accuracy:", accuracy_score(y_test, y_pred))
print("F1 Score:", f1_score(y_test, y_pred))
print("Precision:", precision_score(y_test, y_pred))
print("Recall:", recall_score(y_test, y_pred))

print("\nNote: Accuracy can be misleading for imbalanced datasets")
print("Consider using F1 score, precision, or recall instead")

Real-life Example: Image Classification

In image classification tasks, cross-validation can lead to overfitting when data augmentation is applied incorrectly. Here's an example of how this might occur:

import numpy as np
from sklearn.model_selection import KFold
from sklearn.metrics import accuracy_score
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense
from tensorflow.keras.preprocessing.image import ImageDataGenerator

# Simulating image data
X = np.random.rand(1000, 32, 32, 3)
y = np.random.randint(0, 2, 1000)

# Incorrect approach: Data augmentation before splitting
datagen = ImageDataGenerator(rotation_range=20, width_shift_range=0.2, height_shift_range=0.2)
X_augmented = np.array([datagen.random_transform(img) for img in X])

kf = KFold(n_splits=5)
for train_index, val_index in kf.split(X_augmented):
    X_train, X_val = X_augmented[train_index], X_augmented[val_index]
    y_train, y_val = y[train_index], y[val_index]
    
    model = Sequential([
        Conv2D(32, (3, 3), activation='relu', input_shape=(32, 32, 3)),
        MaxPooling2D((2, 2)),
        Flatten(),
        Dense(64, activation='relu'),
        Dense(1, activation='sigmoid')
    ])
    model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
    model.fit(X_train, y_train, epochs=5, validation_data=(X_val, y_val), verbose=0)
    
    y_pred = model.predict(X_val).round()
    print(f"Incorrect CV Accuracy: {accuracy_score(y_val, y_pred):.4f}")

print("\nNote: Augmenting data before splitting can lead to data leakage and overfitting")

# Correct approach: Data augmentation after splitting
for train_index, val_index in kf.split(X):
    X_train, X_val = X[train_index], X[val_index]
    y_train, y_val = y[train_index], y[val_index]
    
    X_train_augmented = np.array([datagen.random_transform(img) for img in X_train])
    
    model = Sequential([
        Conv2D(32, (3, 3), activation='relu', input_shape=(32, 32, 3)),
        MaxPooling2D((2, 2)),
        Flatten(),
        Dense(64, activation='relu'),
        Dense(1, activation='sigmoid')
    ])
    model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
    model.fit(X_train_augmented, y_train, epochs=5, validation_data=(X_val, y_val), verbose=0)
    
    y_pred = model.predict(X_val).round()
    print(f"Correct CV Accuracy: {accuracy_score(y_val, y_pred):.4f}")

print("\nNote: Augmenting only training data prevents leakage and provides more reliable estimates")

Real-life Example: Time Series Forecasting

In time series forecasting, using standard cross-validation can lead to look-ahead bias. Here's an example of how to correctly implement time series cross-validation:

import pandas as pd
import numpy as np
from sklearn.model_selection import TimeSeriesSplit
from sklearn.metrics import mean_squared_error
from statsmodels.tsa.arima.model import ARIMA

# Generate sample time series data
np.random.seed(42)
date_rng = pd.date_range(start='2020-01-01', end='2022-12-31', freq='D')
ts_data = pd.Series(np.cumsum(np.random.randn(len(date_rng))), index=date_rng)

# Time Series Cross-Validation
tscv = TimeSeriesSplit(n_splits=5)

for train_index, test_index in tscv.split(ts_data):
    train_data = ts_data.iloc[train_index]
    test_data = ts_data.iloc[test_index]
    
    # Fit ARIMA model
    model = ARIMA(train_data, order=(1,1,1))
    model_fit = model.fit()
    
    # Make predictions
    predictions = model_fit.forecast(steps=len(test_data))
    
    # Calculate MSE
    mse = mean_squared_error(test_data, predictions)
    print(f"Train period: {train_data.index[0]} to {train_data.index[-1]}")
    print(f"Test period: {test_data.index[0]} to {test_data.index[-1]}")
    print(f"MSE: {mse:.4f}\n")

print("Note: This approach ensures that we always predict future values using past data")
print("It prevents look-ahead bias and provides more realistic performance estimates")

Nested Cross-Validation

Nested cross-validation addresses some of the issues with standard cross-validation by separating model selection from model evaluation. This approach provides a more reliable estimate of model performance and helps mitigate overfitting to the validation set.

from sklearn.model_selection import KFold, GridSearchCV, cross_val_score
from sklearn.svm import SVC
import numpy as np

def nested_cv(X, y, inner_cv=5, outer_cv=5):
    outer_scores = []
    
    outer_cv = KFold(n_splits=outer_cv, shuffle=True, random_state=42)
    for train_index, test_index in outer_cv.split(X):
        X_train, X_test = X[train_index], X[test_index]
        y_train, y_test = y[train_index], y[test_index]
        
        # Inner cross-validation for hyperparameter tuning
        param_grid = {'C': [0.1, 1, 10], 'kernel': ['rbf', 'linear']}
        inner_cv = KFold(n_splits=inner_cv, shuffle=True, random_state=42)
        grid_search = GridSearchCV(SVC(), param_grid, cv=inner_cv)
        grid_search.fit(X_train, y_train)
        
        # Evaluate the best model on the test set
        best_model = grid_search.best_estimator_
        score = best_model.score(X_test, y_test)
        outer_scores.append(score)
    
    return np.mean(outer_scores), np.std(outer_scores)

# Generate sample data
X, y = np.random.rand(100, 5), np.random.randint(0, 2, 100)

mean_score, std_score = nested_cv(X, y)
print(f"Nested CV Score: {mean_score:.3f} (+/- {std_score:.3f})")

Stratified Cross-Validation

Stratified cross-validation is crucial for maintaining class distribution across folds, especially in imbalanced datasets. It helps ensure that each fold is representative of the overall dataset.

from sklearn.model_selection import StratifiedKFold
from sklearn.datasets import make_classification
from sklearn.svm import SVC
from sklearn.metrics import f1_score
import numpy as np

# Generate imbalanced dataset
X, y = make_classification(n_samples=1000, n_classes=2, weights=[0.9, 0.1], random_state=42)

# Regular K-Fold
kf = KFold(n_splits=5, shuffle=True, random_state=42)
regular_scores = []

for train_index, test_index in kf.split(X):
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]
    
    model = SVC()
    model.fit(X_train, y_train)
    y_pred = model.predict(X_test)
    regular_scores.append(f1_score(y_test, y_pred))

# Stratified K-Fold
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
stratified_scores = []

for train_index, test_index in skf.split(X, y):
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]
    
    model = SVC()
    model.fit(X_train, y_train)
    y_pred = model.predict(X_test)
    stratified_scores.append(f1_score(y_test, y_pred))

print(f"Regular CV F1 Score: {np.mean(regular_scores):.3f} (+/- {np.std(regular_scores):.3f})")
print(f"Stratified CV F1 Score: {np.mean(stratified_scores):.3f} (+/- {np.std(stratified_scores):.3f})")

Cross-Validation with Grouped Data

When dealing with grouped data, such as multiple samples from the same subject, standard cross-validation can lead to overly optimistic results due to data leakage. Group-based cross-validation helps address this issue.

from sklearn.model_selection import GroupKFold
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
import numpy as np

# Generate grouped data
n_subjects = 50
n_samples_per_subject = 10
X = np.random.rand(n_subjects * n_samples_per_subject, 5)
y = np.random.randint(0, 2, n_subjects * n_samples_per_subject)
groups = np.repeat(np.arange(n_subjects), n_samples_per_subject)

# Regular K-Fold
kf = KFold(n_splits=5, shuffle=True, random_state=42)
regular_scores = []

for train_index, test_index in kf.split(X):
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]
    
    model = LogisticRegression()
    model.fit(X_train, y_train)
    y_pred = model.predict(X_test)
    regular_scores.append(accuracy_score(y_test, y_pred))

# Group K-Fold
gkf = GroupKFold(n_splits=5)
group_scores = []

for train_index, test_index in gkf.split(X, y, groups):
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]
    
    model = LogisticRegression()
    model.fit(X_train, y_train)
    y_pred = model.predict(X_test)
    group_scores.append(accuracy_score(y_test, y_pred))

print(f"Regular CV Accuracy: {np.mean(regular_scores):.3f} (+/- {np.std(regular_scores):.3f})")
print(f"Group CV Accuracy: {np.mean(group_scores):.3f} (+/- {np.std(group_scores):.3f})")

Cross-Validation in Small Datasets

Cross-validation can be particularly challenging with small datasets. In these cases, techniques like leave-one-out cross-validation (LOOCV) or repeated k-fold cross-validation can provide more stable estimates.

from sklearn.model_selection import LeaveOneOut, RepeatedKFold
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
import numpy as np

# Generate small dataset
X = np.random.rand(30, 5)
y = np.random.randint(0, 2, 30)

# Leave-One-Out Cross-Validation
loo = LeaveOneOut()
loo_scores = []

for train_index, test_index in loo.split(X):
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]
    
    model = RandomForestClassifier(random_state=42)
    model.fit(X_train, y_train)
    y_pred = model.predict(X_test)
    loo_scores.append(accuracy_score(y_test, y_pred))

# Repeated K-Fold Cross-Validation
rkf = RepeatedKFold(n_splits=5, n_repeats=10, random_state=42)
rkf_scores = []

for train_index, test_index in rkf.split(X):
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]
    
    model = RandomForestClassifier(random_state=42)
    model.fit(X_train, y_train)
    y_pred = model.predict(X_test)
    rkf_scores.append(accuracy_score(y_test, y_pred))

print(f"LOOCV Accuracy: {np.mean(loo_scores):.3f} (+/- {np.std(loo_scores):.3f})")
print(f"Repeated K-Fold Accuracy: {np.mean(rkf_scores):.3f} (+/- {np.std(rkf_scores):.3f})")

Additional Resources

For more information on cross-validation and its challenges, consider exploring these resources:

  1. "A Survey of Cross-Validation Procedures for Model Selection" by Sylvain Arlot and Alain Celisse (2010). Available on arXiv: https://arxiv.org/abs/0907.4728
  2. "Cross-validation pitfalls when selecting and assessing regression and classification models" by Derek Dalton and Gregory Dirichlet (2014). Available on arXiv: https://arxiv.org/abs/1211.0307

These papers provide in-depth discussions on cross-validation techniques, their limitations, and best practices for model selection and evaluation.

More in Model evaluation and validation

  1. Understanding Confusion Matrix in Machine Learning7 min

    A confusion matrix is a fundamental evaluation metric in machine learning that displays the performance of a classification model by comparing predicted...

  2. Response:10 min

    Precision and Recall are fundamental metrics in machine learning and information retrieval. They help evaluate the performance of classification models...

  3. Understanding ROC Curve AUC10 min

    An Area Under the Curve (AUC) of 0.5 in a Receiver Operating Characteristic (ROC) curve indicates that the classifier performs no better than random...

  4. Response:11 min

    Accuracy is often used as a primary metric for evaluating machine learning models. However, relying solely on accuracy can lead to incorrect conclusions...

  5. Understanding Silhouette Score for Clustering8 min

    The silhouette score measures how similar an object is to its own cluster compared to other clusters. It ranges from -1 to 1, where a high value indicates...

  6. Regression Model Evaluation Metrics in Python8 min

    Mean Squared Error serves as a fundamental metric in regression analysis, measuring the average squared difference between predicted and actual values. It...