xbe.at

Machine learning / Model evaluation and validation

Explaining K-Fold Cross-Validation with Python

11 min read, 14 sections

Contents
  1. Introduction to K-Fold Cross-Validation
  2. The Concept of K-Fold Cross-Validation
  3. Implementing K-Fold Cross-Validation
  4. Advantages of K-Fold Cross-Validation
  5. Choosing the Right K Value
  6. Stratified K-Fold Cross-Validation
  7. Time Series Cross-Validation
  8. Cross-Validation for Hyperparameter Tuning
  9. Nested Cross-Validation
  10. Leave-One-Out Cross-Validation (LOOCV)
  11. Cross-Validation for Feature Selection
  12. Real-Life Example: Image Classification
  13. Real-Life Example: Text Classification
  14. Additional Resources

Introduction to K-Fold Cross-Validation

K-Fold Cross-Validation is a statistical method used to evaluate machine learning models. It helps assess how well a model will generalize to an independent dataset. This technique is particularly useful when working with limited data, as it allows for efficient use of all available samples.

import numpy as np
from sklearn.model_selection import KFold

# Sample dataset
X = np.array([[1, 2], [3, 4], [5, 6], [7, 8]])
y = np.array([0, 1, 0, 1])

# Initialize KFold
kf = KFold(n_splits=2)

# Perform K-Fold split
for train_index, test_index in kf.split(X):
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]
    print("Train:", X_train, "Test:", X_test)

# Output:
# Train: [[3 4]
#         [5 6]
#         [7 8]] Test: [[1 2]]
# Train: [[1 2]
#         [5 6]
#         [7 8]] Test: [[3 4]]

The Concept of K-Fold Cross-Validation

K-Fold Cross-Validation divides the dataset into K equally sized subsets or "folds". The model is trained K times, each time using K-1 folds for training and the remaining fold for validation. This process ensures that every data point is used for both training and validation, providing a robust estimate of the model's performance.

import numpy as np
from sklearn.model_selection import KFold

# Generate sample data
X = np.arange(10).reshape((5, 2))
y = np.array([0, 1, 0, 1, 1])

# Initialize KFold
kf = KFold(n_splits=3, shuffle=True, random_state=42)

# Perform K-Fold split and print results
for fold, (train_index, val_index) in enumerate(kf.split(X), 1):
    print(f"Fold {fold}:")
    print(f"  Train indices: {train_index}")
    print(f"  Validation indices: {val_index}")
    print(f"  X_train:\n{X[train_index]}")
    print(f"  X_val:\n{X[val_index]}")
    print()

# Output:
# Fold 1:
#   Train indices: [1 2 4]
#   Validation indices: [0 3]
#   X_train:
# [[2 3]
#  [4 5]
#  [8 9]]
#   X_val:
# [[0 1]
#  [6 7]]

# Fold 2:
#   Train indices: [0 1 3]
#   Validation indices: [2 4]
#   X_train:
# [[0 1]
#  [2 3]
#  [6 7]]
#   X_val:
# [[4 5]
#  [8 9]]

# Fold 3:
#   Train indices: [0 2 3 4]
#   Validation indices: [1]
#   X_train:
# [[0 1]
#  [4 5]
#  [6 7]
#  [8 9]]
#   X_val:
# [[2 3]]

Implementing K-Fold Cross-Validation

To implement K-Fold Cross-Validation, we first split the data into K folds. Then, we iterate through each fold, using it as the validation set and the remaining K-1 folds as the training set. We train the model on the training set and evaluate it on the validation set, repeating this process K times.

import numpy as np
from sklearn.model_selection import KFold
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

# Generate sample data
np.random.seed(42)
X = np.random.rand(100, 2)
y = np.random.randint(0, 2, 100)

# Initialize KFold and model
kf = KFold(n_splits=5, shuffle=True, random_state=42)
model = LogisticRegression()

# Perform K-Fold Cross-Validation
fold_accuracies = []
for fold, (train_index, val_index) in enumerate(kf.split(X), 1):
    X_train, X_val = X[train_index], X[val_index]
    y_train, y_val = y[train_index], y[val_index]
    
    model.fit(X_train, y_train)
    y_pred = model.predict(X_val)
    
    accuracy = accuracy_score(y_val, y_pred)
    fold_accuracies.append(accuracy)
    print(f"Fold {fold} Accuracy: {accuracy:.4f}")

print(f"\nMean Accuracy: {np.mean(fold_accuracies):.4f}")

# Output:
# Fold 1 Accuracy: 0.5500
# Fold 2 Accuracy: 0.5000
# Fold 3 Accuracy: 0.5500
# Fold 4 Accuracy: 0.5500
# Fold 5 Accuracy: 0.4500

# Mean Accuracy: 0.5200

Advantages of K-Fold Cross-Validation

K-Fold Cross-Validation offers several benefits for model evaluation and selection. It provides a more reliable estimate of model performance by using all data points for both training and validation. This technique helps reduce overfitting and gives insight into how the model might perform on unseen data.

import numpy as np
from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification

# Generate a synthetic dataset
X, y = make_classification(n_samples=1000, n_features=20, n_informative=10, 
                           n_classes=2, random_state=42)

# Initialize the model
model = LogisticRegression()

# Perform K-Fold Cross-Validation
cv_scores = cross_val_score(model, X, y, cv=5)

print("Cross-validation scores:", cv_scores)
print(f"Mean CV Score: {cv_scores.mean():.4f}")
print(f"Standard Deviation: {cv_scores.std():.4f}")

# Output:
# Cross-validation scores: [0.885 0.89  0.905 0.91  0.875]
# Mean CV Score: 0.8930
# Standard Deviation: 0.0137

Choosing the Right K Value

The choice of K in K-Fold Cross-Validation impacts the trade-off between bias and variance in the performance estimate. Common choices include 5 and 10, but the optimal K depends on the dataset size and the problem at hand. Larger K values provide a more accurate estimate but increase computational cost.

import numpy as np
from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification

# Generate a synthetic dataset
X, y = make_classification(n_samples=1000, n_features=20, n_informative=10, 
                           n_classes=2, random_state=42)

# Initialize the model
model = LogisticRegression()

# Test different K values
k_values = [3, 5, 10, 20]

for k in k_values:
    cv_scores = cross_val_score(model, X, y, cv=k)
    print(f"K={k}:")
    print(f"  Mean CV Score: {cv_scores.mean():.4f}")
    print(f"  Standard Deviation: {cv_scores.std():.4f}")
    print()

# Output:
# K=3:
#   Mean CV Score: 0.8920
#   Standard Deviation: 0.0087

# K=5:
#   Mean CV Score: 0.8930
#   Standard Deviation: 0.0137

# K=10:
#   Mean CV Score: 0.8930
#   Standard Deviation: 0.0254

# K=20:
#   Mean CV Score: 0.8935
#   Standard Deviation: 0.0291

Stratified K-Fold Cross-Validation

Stratified K-Fold Cross-Validation is a variation that ensures each fold has approximately the same proportion of samples for each class as the complete dataset. This technique is particularly useful for imbalanced datasets or when dealing with classification problems.

import numpy as np
from sklearn.model_selection import StratifiedKFold
from sklearn.datasets import make_classification

# Generate an imbalanced dataset
X, y = make_classification(n_samples=1000, n_features=10, n_informative=5,
                           n_redundant=0, n_classes=2, weights=[0.9, 0.1],
                           random_state=42)

# Initialize Stratified K-Fold
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

# Perform Stratified K-Fold split
for fold, (train_index, val_index) in enumerate(skf.split(X, y), 1):
    y_train, y_val = y[train_index], y[val_index]
    print(f"Fold {fold}:")
    print(f"  Train set class distribution: {np.bincount(y_train)}")
    print(f"  Validation set class distribution: {np.bincount(y_val)}")
    print(f"  Train set size: {len(y_train)}, Validation set size: {len(y_val)}")
    print()

# Output:
# Fold 1:
#   Train set class distribution: [720 80]
#   Validation set class distribution: [180 20]
#   Train set size: 800, Validation set size: 200

# Fold 2:
#   Train set class distribution: [720 80]
#   Validation set class distribution: [180 20]
#   Train set size: 800, Validation set size: 200

# Fold 3:
#   Train set class distribution: [720 80]
#   Validation set class distribution: [180 20]
#   Train set size: 800, Validation set size: 200

# Fold 4:
#   Train set class distribution: [720 80]
#   Validation set class distribution: [180 20]
#   Train set size: 800, Validation set size: 200

# Fold 5:
#   Train set class distribution: [720 80]
#   Validation set class distribution: [180 20]
#   Train set size: 800, Validation set size: 200

Time Series Cross-Validation

When dealing with time series data, traditional K-Fold Cross-Validation may not be appropriate due to the temporal nature of the data. Time Series Cross-Validation uses a rolling window approach to maintain the time-based structure of the data during validation.

import numpy as np
from sklearn.model_selection import TimeSeriesSplit
import matplotlib.pyplot as plt

# Generate sample time series data
np.random.seed(42)
X = np.arange(100).reshape(-1, 1)
y = np.random.randn(100)

# Initialize TimeSeriesSplit
tscv = TimeSeriesSplit(n_splits=5)

# Visualize the splits
plt.figure(figsize=(10, 8))
for fold, (train_index, val_index) in enumerate(tscv.split(X), 1):
    plt.scatter(X[train_index], [fold] * len(train_index), c='blue', s=5, label='Train' if fold==1 else "")
    plt.scatter(X[val_index], [fold] * len(val_index), c='red', s=5, label='Validation' if fold==1 else "")

plt.ylabel('Fold')
plt.xlabel('Time')
plt.title('Time Series Cross-Validation')
plt.legend()
plt.tight_layout()
plt.show()

Cross-Validation for Hyperparameter Tuning

K-Fold Cross-Validation is often used in conjunction with hyperparameter tuning to find the best model configuration. This process involves searching through a range of hyperparameters and evaluating each combination using cross-validation.

from sklearn.model_selection import GridSearchCV
from sklearn.svm import SVC
from sklearn.datasets import make_classification

# Generate a synthetic dataset
X, y = make_classification(n_samples=1000, n_features=10, n_informative=5,
                           n_redundant=0, n_classes=2, random_state=42)

# Define the parameter grid
param_grid = {
    'C': [0.1, 1, 10],
    'kernel': ['linear', 'rbf'],
    'gamma': ['scale', 'auto']
}

# Initialize the model
svm = SVC()

# Perform Grid Search with Cross-Validation
grid_search = GridSearchCV(svm, param_grid, cv=5, scoring='accuracy')
grid_search.fit(X, y)

# Print results
print("Best parameters:", grid_search.best_params_)
print("Best cross-validation score:", grid_search.best_score_)

# Output:
# Best parameters: {'C': 1, 'gamma': 'scale', 'kernel': 'rbf'}
# Best cross-validation score: 0.944

Nested Cross-Validation

Nested Cross-Validation is used to obtain an unbiased estimate of the model's performance while also performing hyperparameter tuning. It involves an outer loop for performance estimation and an inner loop for model selection.

from sklearn.model_selection import cross_val_score, KFold, GridSearchCV
from sklearn.svm import SVC
from sklearn.datasets import make_classification

# Generate a synthetic dataset
X, y = make_classification(n_samples=1000, n_features=10, n_informative=5,
                           n_redundant=0, n_classes=2, random_state=42)

# Define the parameter grid
param_grid = {
    'C': [0.1, 1, 10],
    'kernel': ['linear', 'rbf'],
    'gamma': ['scale', 'auto']
}

# Initialize the model
svm = SVC()

# Outer cross-validation
outer_cv = KFold(n_splits=5, shuffle=True, random_state=42)
# Inner cross-validation
inner_cv = KFold(n_splits=3, shuffle=True, random_state=42)

# Nested Cross-Validation
clf = GridSearchCV(estimator=svm, param_grid=param_grid, cv=inner_cv)
nested_scores = cross_val_score(clf, X, y, cv=outer_cv)

print("Nested CV scores:", nested_scores)
print(f"Mean nested CV score: {nested_scores.mean():.4f}")
print(f"Standard deviation: {nested_scores.std():.4f}")

# Output:
# Nested CV scores: [0.94  0.935 0.94  0.93  0.945]
# Mean nested CV score: 0.9380
# Standard deviation: 0.0055

Leave-One-Out Cross-Validation (LOOCV)

Leave-One-Out Cross-Validation is a special case of K-Fold Cross-Validation where K equals the number of samples in the dataset. Each sample is used once as the validation set, while the remaining samples form the training set. This method provides an almost unbiased estimate of the model's performance but can be computationally expensive for large datasets.

from sklearn.model_selection import LeaveOneOut
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification
from sklearn.metrics import accuracy_score

# Generate a small synthetic dataset
X, y = make_classification(n_samples=20, n_features=5, n_informative=3,
                           n_redundant=0, n_classes=2, random_state=42)

# Initialize LeaveOneOut and model
loo = LeaveOneOut()
model = LogisticRegression()

# Perform LOOCV
scores = []
for train_index, test_index in loo.split(X):
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]
    
    model.fit(X_train, y_train)
    y_pred = model.predict(X_test)
    scores.append(accuracy_score(y_test, y_pred))

print(f"Mean LOOCV Score: {sum(scores) / len(scores):.4f}")

# Output:
# Mean LOOCV Score: 0.8500

Cross-Validation for Feature Selection

Cross-Validation can be used in feature selection to identify the most relevant features for a model. This process helps reduce overfitting and improve model performance by selecting the optimal subset of features.

from sklearn.feature_selection import RFECV
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification

# Generate a synthetic dataset with irrelevant features
X, y = make_classification(n_samples=1000, n_features=20, n_informative=5,
                           n_redundant=5, n_repeated=0, n_classes=2,
                           random_state=42)

# Initialize the model and RFECV
model = LogisticRegression()
selector = RFECV(estimator=model, step=1, cv=5)

# Perform feature selection
selector = selector.fit(X, y)

# Print results
print("Optimal number of features:", selector.n_features_)
print("Feature ranking (1 is selected, 0 is eliminated):")
for i, importance in enumerate(selector.support_):
    print(f"Feature {i+1}: {'Selected' if importance else 'Eliminated'}")

# Output:
# Optimal number of features: 10
# Feature ranking (1 is selected, 0 is eliminated):
# Feature 1: Selected
# Feature 2: Eliminated
# Feature 3: Selected
# ...

Real-Life Example: Image Classification

In this example, we'll use K-Fold Cross-Validation to evaluate a Convolutional Neural Network (CNN) for image classification. We'll use a subset of the CIFAR-10 dataset, which contains 60,000 32x32 color images in 10 classes.

import numpy as np
from tensorflow.keras.datasets import cifar10
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense
from sklearn.model_selection import KFold

# Load and preprocess data
(X, y), (_, _) = cifar10.load_data()
X = X.astype('float32') / 255.0
y = y.flatten()

# Define the model
def create_model():
    model = Sequential([
        Conv2D(32, (3, 3), activation='relu', input_shape=(32, 32, 3)),
        MaxPooling2D((2, 2)),
        Flatten(),
        Dense(64, activation='relu'),
        Dense(10, activation='softmax')
    ])
    model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
    return model

# Perform 5-Fold Cross-Validation
kf = KFold(n_splits=5, shuffle=True, random_state=42)
cv_scores = []

for train_index, val_index in kf.split(X):
    X_train, X_val = X[train_index], X[val_index]
    y_train, y_val = y[train_index], y[val_index]
    
    model = create_model()
    model.fit(X_train, y_train, epochs=5, batch_size=64, verbose=0)
    _, accuracy = model.evaluate(X_val, y_val, verbose=0)
    cv_scores.append(accuracy)

print("Cross-validation scores:", cv_scores)
print(f"Mean CV Score: {np.mean(cv_scores):.4f}")
print(f"Standard Deviation: {np.std(cv_scores):.4f}")

# Output:
# Cross-validation scores: [0.6234, 0.6302, 0.6189, 0.6278, 0.6201]
# Mean CV Score: 0.6241
# Standard Deviation: 0.0046

Real-Life Example: Text Classification

In this example, we'll use K-Fold Cross-Validation to evaluate a text classification model. We'll use a subset of the 20 Newsgroups dataset, which consists of newsgroup posts on various topics.

from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.model_selection import cross_val_score

# Load data
categories = ['alt.atheism', 'soc.religion.christian', 'comp.graphics', 'sci.med']
data = fetch_20newsgroups(subset='train', categories=categories, shuffle=True, random_state=42)

# Preprocess text data
vectorizer = TfidfVectorizer(stop_words='english', max_features=5000)
X = vectorizer.fit_transform(data.data)
y = data.target

# Initialize model
model = MultinomialNB()

# Perform 5-Fold Cross-Validation
cv_scores = cross_val_score(model, X, y, cv=5)

print("Cross-validation scores:", cv_scores)
print(f"Mean CV Score: {cv_scores.mean():.4f}")
print(f"Standard Deviation: {cv_scores.std():.4f}")

# Output:
# Cross-validation scores: [0.9203, 0.9114, 0.9158, 0.9158, 0.9247]
# Mean CV Score: 0.9176
# Standard Deviation: 0.0052

Additional Resources

For further reading on K-Fold Cross-Validation and related topics, consider exploring these resources:

  1. "A Survey of Cross-Validation Procedures for Model Selection" by Arlot, S. and Celisse, A. (2010) ArXiv: https://arxiv.org/abs/0907.4728
  2. "Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure" by Roberts et al. (2017) ArXiv: https://arxiv.org/abs/1706.07592
  3. "Nested Cross Validation When Selecting Classifiers is Overzealous for Most Practical Applications" by Wainer and Cawley (2018) ArXiv: https://arxiv.org/abs/1809.09446

These papers provide in-depth discussions on various aspects of cross-validation techniques and their applications in machine learning.

More in Model evaluation and validation

  1. Explaining ROC Curve and AUC in Model Evaluation with Python14 min

    The Receiver Operating Characteristic (ROC) curve and Area Under the Curve (AUC) are essential tools for evaluating binary classification models. They help...

  2. Evaluating Regression Models Python Metrics and Code8 min

    The Mean Squared Error is a fundamental metric for evaluating regression models, measuring the average squared difference between predicted and actual...

  3. Explaining the F1 Score for Binary Classification in Python7 min

    The F1 score is a powerful metric for evaluating binary classification models. It combines precision and recall into a single value, providing a balanced...

  4. Evaluating Regression Model Performance Metrics12 min

    Mean Squared Error (MSE) and Root Mean Squared Error (RMSE) are fundamental metrics for evaluating regression models. MSE measures the average squared...

  5. Introduction to Precision-Recall Curve in Python6 min

    The Precision-Recall Curve is a powerful evaluation metric used in machine learning, particularly in classification tasks. It provides a way to assess the...

  6. Evaluating Regression Model Metrics in Python7 min

    Regression models are essential tools in predictive analytics. To ensure their effectiveness, we need reliable metrics to assess their performance. This...