xbe.at

Machine learning / Model evaluation and validation

Comprehensive Guide to Cross-Validation in Machine Learning

11 min read, 15 sections

Contents
  1. Cross-Validation in Machine Learning
  2. Holdout Method
  3. K-Fold Cross-Validation
  4. Stratified K-Fold Cross-Validation
  5. Leave-One-Out Cross-Validation (LOO-CV)
  6. Time Series Cross-Validation
  7. Nested Cross-Validation
  8. Cross-Validation for Imbalanced Datasets
  9. Cross-Validation for Feature Selection
  10. Cross-Validation for Model Comparison
  11. Cross-Validation for Time Series Data
  12. Cross-Validation for Hyperparameter Tuning
  13. Cross-Validation for Ensemble Methods
  14. Real-Life Example: Predicting Customer Churn
  15. Additional Resources

Cross-Validation in Machine Learning

Cross-validation is a crucial technique in machine learning that helps assess a model's performance and generalization ability. It involves splitting the data into subsets, training the model on some subsets, and testing it on others. This process helps prevent overfitting and provides a more reliable estimate of how the model will perform on unseen data.

from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_iris
from sklearn.tree import DecisionTreeClassifier

# Load the iris dataset
iris = load_iris()
X, y = iris.data, iris.target

# Create a decision tree classifier
clf = DecisionTreeClassifier(random_state=42)

# Perform 5-fold cross-validation
scores = cross_val_score(clf, X, y, cv=5)

print(f"Cross-validation scores: {scores}")
print(f"Mean accuracy: {scores.mean():.2f} (+/- {scores.std() * 2:.2f})")

Holdout Method

The holdout method is the simplest form of cross-validation. It involves splitting the dataset into two parts: a training set and a test set. The model is trained on the training set and evaluated on the test set. While quick and easy to implement, this method can be sensitive to how the data is split.

from sklearn.model_selection import train_test_split
from sklearn.datasets import load_breast_cancer
from sklearn.svm import SVC
from sklearn.metrics import accuracy_score

# Load the breast cancer dataset
X, y = load_breast_cancer(return_X_y=True)

# Split the data into training and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Create and train the model
clf = SVC(kernel='rbf', random_state=42)
clf.fit(X_train, y_train)

# Make predictions and calculate accuracy
y_pred = clf.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)

print(f"Accuracy: {accuracy:.2f}")

K-Fold Cross-Validation

K-Fold Cross-Validation divides the dataset into K equal-sized folds. The model is trained on K-1 folds and tested on the remaining fold. This process is repeated K times, with each fold serving as the test set once. The final performance metric is the average of all K iterations.

from sklearn.model_selection import KFold
from sklearn.datasets import load_diabetes
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
import numpy as np

# Load the diabetes dataset
X, y = load_diabetes(return_X_y=True)

# Create a KFold object
kf = KFold(n_splits=5, shuffle=True, random_state=42)

# Initialize lists to store results
mse_scores = []

# Perform K-Fold Cross-Validation
for train_index, test_index in kf.split(X):
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]
    
    model = LinearRegression()
    model.fit(X_train, y_train)
    
    y_pred = model.predict(X_test)
    mse = mean_squared_error(y_test, y_pred)
    mse_scores.append(mse)

print(f"Mean MSE: {np.mean(mse_scores):.2f}")
print(f"Standard deviation of MSE: {np.std(mse_scores):.2f}")

Stratified K-Fold Cross-Validation

Stratified K-Fold ensures that the proportion of samples for each class is roughly the same in each fold as in the whole dataset. This method is particularly useful for imbalanced datasets or when dealing with classification problems.

from sklearn.model_selection import StratifiedKFold
from sklearn.datasets import load_wine
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
import numpy as np

# Load the wine dataset
X, y = load_wine(return_X_y=True)

# Create a StratifiedKFold object
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

# Initialize lists to store results
accuracies = []

# Perform Stratified K-Fold Cross-Validation
for train_index, test_index in skf.split(X, y):
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]
    
    model = RandomForestClassifier(n_estimators=100, random_state=42)
    model.fit(X_train, y_train)
    
    y_pred = model.predict(X_test)
    accuracy = accuracy_score(y_test, y_pred)
    accuracies.append(accuracy)

print(f"Mean accuracy: {np.mean(accuracies):.2f}")
print(f"Standard deviation of accuracy: {np.std(accuracies):.2f}")

Leave-One-Out Cross-Validation (LOO-CV)

Leave-One-Out Cross-Validation is an extreme case of K-Fold cross-validation where K equals the number of samples in the dataset. In each iteration, the model is trained on all but one sample and tested on that single left-out sample. This method provides an almost unbiased estimate of the model's performance but can be computationally expensive for large datasets.

from sklearn.model_selection import LeaveOneOut
from sklearn.datasets import load_boston
from sklearn.neighbors import KNeighborsRegressor
from sklearn.metrics import mean_squared_error
import numpy as np

# Load the Boston housing dataset
X, y = load_boston(return_X_y=True)

# Create a LeaveOneOut object
loo = LeaveOneOut()

# Initialize lists to store results
mse_scores = []

# Perform Leave-One-Out Cross-Validation
for train_index, test_index in loo.split(X):
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]
    
    model = KNeighborsRegressor(n_neighbors=5)
    model.fit(X_train, y_train)
    
    y_pred = model.predict(X_test)
    mse = mean_squared_error(y_test, y_pred)
    mse_scores.append(mse)

print(f"Mean MSE: {np.mean(mse_scores):.2f}")
print(f"Standard deviation of MSE: {np.std(mse_scores):.2f}")

Time Series Cross-Validation

Time Series Cross-Validation is designed for temporal data where the order of observations matters. It uses a sliding window approach to simulate how a model would perform in a real-world scenario, where you train on past data and predict future data.

from sklearn.model_selection import TimeSeriesSplit
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
import numpy as np
import pandas as pd

# Create a simple time series dataset
dates = pd.date_range(start='2020-01-01', end='2022-12-31', freq='D')
np.random.seed(42)
values = np.cumsum(np.random.randn(len(dates)))
df = pd.DataFrame({'date': dates, 'value': values})

X = df.index.values.reshape(-1, 1)
y = df['value'].values

# Create a TimeSeriesSplit object
tscv = TimeSeriesSplit(n_splits=5)

# Initialize lists to store results
mse_scores = []

# Perform Time Series Cross-Validation
for train_index, test_index in tscv.split(X):
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]
    
    scaler = StandardScaler()
    X_train_scaled = scaler.fit_transform(X_train)
    X_test_scaled = scaler.transform(X_test)
    
    model = LinearRegression()
    model.fit(X_train_scaled, y_train)
    
    y_pred = model.predict(X_test_scaled)
    mse = mean_squared_error(y_test, y_pred)
    mse_scores.append(mse)

print(f"Mean MSE: {np.mean(mse_scores):.2f}")
print(f"Standard deviation of MSE: {np.std(mse_scores):.2f}")

Nested Cross-Validation

Nested Cross-Validation is used when you need to tune hyperparameters and get an unbiased estimate of the model's performance. It consists of an outer loop for estimating the model's performance and an inner loop for hyperparameter tuning.

from sklearn.model_selection import KFold, GridSearchCV
from sklearn.svm import SVC
from sklearn.datasets import load_digits
from sklearn.metrics import accuracy_score
import numpy as np

# Load the digits dataset
X, y = load_digits(return_X_y=True)

# Set up the nested cross-validation
outer_cv = KFold(n_splits=5, shuffle=True, random_state=42)
inner_cv = KFold(n_splits=3, shuffle=True, random_state=42)

# Define the parameter grid for SVC
param_grid = {'C': [0.1, 1, 10], 'kernel': ['rbf', 'linear']}

# Initialize lists to store results
outer_scores = []

# Perform nested cross-validation
for train_index, test_index in outer_cv.split(X):
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]
    
    # Inner loop for hyperparameter tuning
    grid_search = GridSearchCV(SVC(), param_grid, cv=inner_cv)
    grid_search.fit(X_train, y_train)
    
    # Use the best model from grid search to make predictions
    best_model = grid_search.best_estimator_
    y_pred = best_model.predict(X_test)
    
    # Calculate and store the accuracy score
    accuracy = accuracy_score(y_test, y_pred)
    outer_scores.append(accuracy)

print(f"Mean accuracy: {np.mean(outer_scores):.2f}")
print(f"Standard deviation of accuracy: {np.std(outer_scores):.2f}")

Cross-Validation for Imbalanced Datasets

When dealing with imbalanced datasets, standard cross-validation techniques may not be sufficient. Stratified K-Fold can help, but sometimes we need additional techniques like oversampling or undersampling within the cross-validation loop.

from sklearn.model_selection import StratifiedKFold
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import f1_score
from imblearn.over_sampling import SMOTE
import numpy as np

# Create an imbalanced dataset
X, y = make_classification(n_samples=1000, n_classes=2, weights=[0.9, 0.1], 
                           n_informative=3, n_redundant=1, flip_y=0, random_state=42)

# Set up Stratified K-Fold
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

# Initialize lists to store results
f1_scores_no_sampling = []
f1_scores_with_smote = []

# Perform cross-validation with and without SMOTE
for train_index, test_index in skf.split(X, y):
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]
    
    # Without SMOTE
    model_no_sampling = RandomForestClassifier(random_state=42)
    model_no_sampling.fit(X_train, y_train)
    y_pred_no_sampling = model_no_sampling.predict(X_test)
    f1_no_sampling = f1_score(y_test, y_pred_no_sampling)
    f1_scores_no_sampling.append(f1_no_sampling)
    
    # With SMOTE
    smote = SMOTE(random_state=42)
    X_train_resampled, y_train_resampled = smote.fit_resample(X_train, y_train)
    model_with_smote = RandomForestClassifier(random_state=42)
    model_with_smote.fit(X_train_resampled, y_train_resampled)
    y_pred_with_smote = model_with_smote.predict(X_test)
    f1_with_smote = f1_score(y_test, y_pred_with_smote)
    f1_scores_with_smote.append(f1_with_smote)

print(f"Mean F1-score without SMOTE: {np.mean(f1_scores_no_sampling):.2f}")
print(f"Mean F1-score with SMOTE: {np.mean(f1_scores_with_smote):.2f}")

Cross-Validation for Feature Selection

Cross-validation can be used in feature selection to ensure that the selected features generalize well to unseen data. This process helps prevent overfitting that can occur when feature selection is performed on the entire dataset.

from sklearn.model_selection import KFold
from sklearn.datasets import load_breast_cancer
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.svm import SVC
from sklearn.metrics import accuracy_score
import numpy as np

# Load the breast cancer dataset
X, y = load_breast_cancer(return_X_y=True)

# Set up K-Fold cross-validation
kf = KFold(n_splits=5, shuffle=True, random_state=42)

# Initialize lists to store results
accuracies = []

# Perform cross-validation with feature selection
for train_index, test_index in kf.split(X):
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]
    
    # Perform feature selection on training data
    selector = SelectKBest(f_classif, k=10)
    X_train_selected = selector.fit_transform(X_train, y_train)
    
    # Apply feature selection to test data
    X_test_selected = selector.transform(X_test)
    
    # Train and evaluate the model
    model = SVC(kernel='rbf', random_state=42)
    model.fit(X_train_selected, y_train)
    y_pred = model.predict(X_test_selected)
    accuracy = accuracy_score(y_test, y_pred)
    accuracies.append(accuracy)

print(f"Mean accuracy: {np.mean(accuracies):.2f}")
print(f"Standard deviation of accuracy: {np.std(accuracies):.2f}")

Cross-Validation for Model Comparison

Cross-validation is an excellent tool for comparing different models or algorithms. By using the same cross-validation splits for each model, we can obtain a fair comparison of their performances.

from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_digits
from sklearn.svm import SVC
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
import numpy as np

# Load the digits dataset
X, y = load_digits(return_X_y=True)

# Define the models to compare
models = {
    'SVM': SVC(kernel='rbf', random_state=42),
    'Decision Tree': DecisionTreeClassifier(random_state=42),
    'KNN': KNeighborsClassifier()
}

# Perform cross-validation for each model
for name, model in models.items():
    scores = cross_val_score(model, X, y, cv=5, scoring='accuracy')
    print(f"{name} - Mean accuracy: {np.mean(scores):.2f} (+/- {np.std(scores) * 2:.2f})")

Cross-Validation for Time Series Data

When working with time series data, traditional cross-validation methods can lead to data leakage. Time series cross-validation maintains the temporal order of the data, ensuring that we always predict future values based on past observations.

from sklearn.model_selection import TimeSeriesSplit
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
import numpy as np
import pandas as pd

# Create a simple time series dataset
dates = pd.date_range(start='2020-01-01', end='2021-12-31', freq='D')
np.random.seed(42)
values = np.cumsum(np.random.randn(len(dates)))
df = pd.DataFrame({'date': dates, 'value': values})

X = df.index.values.reshape(-1, 1)
y = df['value'].values

# Set up TimeSeriesSplit
tscv = TimeSeriesSplit(n_splits=5)

mse_scores = []

for train_index, test_index in tscv.split(X):
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]
    
    model = LinearRegression()
    model.fit(X_train, y_train)
    
    y_pred = model.predict(X_test)
    mse = mean_squared_error(y_test, y_pred)
    mse_scores.append(mse)

print(f"Mean MSE: {np.mean(mse_scores):.2f}")
print(f"Standard deviation of MSE: {np.std(mse_scores):.2f}")

Cross-Validation for Hyperparameter Tuning

Cross-validation is crucial for hyperparameter tuning to prevent overfitting. By using techniques like GridSearchCV, we can systematically explore different hyperparameter combinations and select the best-performing model.

from sklearn.model_selection import GridSearchCV
from sklearn.svm import SVC
from sklearn.datasets import load_iris
from sklearn.metrics import make_scorer, accuracy_score

# Load the iris dataset
X, y = load_iris(return_X_y=True)

# Define the parameter grid
param_grid = {
    'C': [0.1, 1, 10],
    'kernel': ['rbf', 'linear'],
    'gamma': ['scale', 'auto', 0.1, 1]
}

# Create the GridSearchCV object
grid_search = GridSearchCV(
    SVC(random_state=42),
    param_grid,
    cv=5,
    scoring=make_scorer(accuracy_score),
    n_jobs=-1
)

# Fit the GridSearchCV
grid_search.fit(X, y)

print(f"Best parameters: {grid_search.best_params_}")
print(f"Best cross-validation score: {grid_search.best_score_:.2f}")

Cross-Validation for Ensemble Methods

Ensemble methods, such as bagging and boosting, often use cross-validation internally. For example, Random Forests use bootstrap sampling, which is a form of cross-validation. When evaluating ensemble methods, it's important to use cross-validation to get a robust estimate of their performance.

from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.datasets import load_breast_cancer
import numpy as np

# Load the breast cancer dataset
X, y = load_breast_cancer(return_X_y=True)

# Create ensemble models
rf_model = RandomForestClassifier(n_estimators=100, random_state=42)
gb_model = GradientBoostingClassifier(n_estimators=100, random_state=42)

# Perform cross-validation
rf_scores = cross_val_score(rf_model, X, y, cv=5)
gb_scores = cross_val_score(gb_model, X, y, cv=5)

print("Random Forest:")
print(f"Mean accuracy: {np.mean(rf_scores):.2f}")
print(f"Standard deviation: {np.std(rf_scores):.2f}")

print("\nGradient Boosting:")
print(f"Mean accuracy: {np.mean(gb_scores):.2f}")
print(f"Standard deviation: {np.std(gb_scores):.2f}")

Real-Life Example: Predicting Customer Churn

In this example, we'll use cross-validation to evaluate a model that predicts customer churn for a telecommunications company. This showcases how cross-validation can be applied to a real-world business problem.

from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.datasets import make_classification
import numpy as np

# Simulate a customer churn dataset
X, y = make_classification(n_samples=1000, n_features=20, n_informative=10, 
                           n_redundant=5, n_classes=2, weights=[0.7, 0.3], 
                           random_state=42)

# Create a pipeline with preprocessing and model
pipeline = make_pipeline(
    StandardScaler(),
    RandomForestClassifier(n_estimators=100, random_state=42)
)

# Perform cross-validation
scores = cross_val_score(pipeline, X, y, cv=5, scoring='roc_auc')

print("Customer Churn Prediction Model")
print(f"Mean ROC AUC: {np.mean(scores):.2f}")
print(f"Standard deviation: {np.std(scores):.2f}")

Additional Resources

For those interested in diving deeper into cross-validation techniques and their applications in machine learning, here are some valuable resources:

  1. "Cross-validation: evaluating estimator performance" - Scikit-learn documentation https://scikit-learn.org/stable/modules/cross_validation.html
  2. "A Survey of Cross-Validation Procedures for Model Selection" by Arlot, S. and Celisse, A. (2010) ArXiv link: https://arxiv.org/abs/0907.4728
  3. "Nested Cross Validation" by Cawley, G. C. and Talbot, N. L. C. (2010) Journal of Machine Learning Research
  4. "An Introduction to Statistical Learning" by James, G., Witten, D., Hastie, T., and Tibshirani, R. Available online: https://www.statlearning.com/

These resources provide in-depth explanations and advanced techniques for cross-validation in various machine learning scenarios.

More in Model evaluation and validation

  1. Conformal Predictions in Machine Learning with Python17 min

    Conformal prediction is a framework for making reliable predictions with a guaranteed error rate. It provides a way to quantify uncertainty in machine...

  2. Calculating Precision from Confusion Matrix in Machine Learning10 min

    A confusion matrix serves as the foundation for calculating precision and other key performance metrics in binary classification. It represents a structured...

  3. Confusion Matrix and Performance Metrics in Python10 min

    A confusion matrix is a fundamental tool in machine learning for evaluating classification models. It provides a tabular summary of the model's predictions...

  4. Calculating Metrics from Confusion Matrix4 min

    A confusion matrix is a fundamental tool in machine learning that tabulates predicted versus actual values, enabling evaluation of classification model...

  5. Confusion Matrix Concepts and Python Examples10 min

    A confusion matrix serves as a fundamental evaluation tool in machine learning, particularly for classification tasks. It organizes predictions into four key...

  6. Calculating Confusion Matrix Metrics7 min

    In binary classification, a confusion matrix is a 2x2 table that visualizes the performance of a model by comparing predicted values against actual values....