xbe.at

Machine learning / Model evaluation and validation

Validation Set Approach in Machine Learning

9 min read, 15 sections

Contents
  1. Understanding the Validation Set Approach
  2. Implementation from Scratch
  3. Mathematical Foundation
  4. Cross-Validation vs. Validation Set
  5. Real-world Example: Credit Card Fraud Detection
  6. Validation Set Sizing Strategy
  7. Stratified Validation Split Implementation
  8. Time Series Validation
  9. Model Selection with Validation Sets
  10. Real-world Example: Customer Churn Prediction
  11. Validation Set with K-Fold Averaging
  12. Validation Set Size Impact Analysis
  13. Handling Class Imbalance in Validation Sets
  14. Time-Based Rolling Validation
  15. Additional Resources

Understanding the Validation Set Approach

The validation set approach is a fundamental technique in machine learning that involves splitting available data into training and validation sets. This method provides an unbiased evaluation of model performance on unseen data while maintaining statistical independence between model training and evaluation.

import numpy as np
from sklearn.model_selection import train_test_split

# Generate sample data
X = np.random.randn(1000, 10)  # 1000 samples, 10 features
y = np.random.randint(0, 2, 1000)  # Binary classification

# Split data into training and validation sets (80-20 split)
X_train, X_val, y_train, y_val = train_test_split(
    X, y, test_size=0.2, random_state=42
)

print(f"Training set shape: {X_train.shape}")
print(f"Validation set shape: {X_val.shape}")

Implementation from Scratch

Building a custom validation split function helps understand the underlying mechanics of data partitioning. This implementation demonstrates how to randomly shuffle and split data while maintaining corresponding feature-target relationships.

def custom_train_val_split(X, y, val_size=0.2, random_state=None):
    if random_state:
        np.random.seed(random_state)
    
    # Generate shuffled indices
    indices = np.random.permutation(len(X))
    
    # Calculate split point
    val_samples = int(len(X) * val_size)
    val_idx = indices[:val_samples]
    train_idx = indices[val_samples:]
    
    return X[train_idx], X[val_idx], y[train_idx], y[val_idx]

# Example usage
X_train, X_val, y_train, y_val = custom_train_val_split(X, y, 0.2, 42)

Mathematical Foundation

The validation set approach relies on probability theory and sampling statistics to ensure reliable model evaluation. The mathematical framework establishes the relationship between sample size, model complexity, and estimation error.

# Mathematical formulas for validation set approach
"""
$$E_{val}(h) = \frac{1}{n_{val}} \sum_{i=1}^{n_{val}} L(h(x_i), y_i)$$

$$Var(E_{val}) = \frac{\sigma^2}{n_{val}}$$

$$CI = E_{val} \pm z_{\alpha/2} \sqrt{\frac{\sigma^2}{n_{val}}}$$
"""

def calculate_validation_error(y_true, y_pred):
    return np.mean((y_true - y_pred) ** 2)

def confidence_interval(y_true, y_pred, confidence=0.95):
    mse = calculate_validation_error(y_true, y_pred)
    std = np.std((y_true - y_pred) ** 2)
    z = 1.96  # 95% confidence
    n = len(y_true)
    return mse, mse - z * std/np.sqrt(n), mse + z * std/np.sqrt(n)

Cross-Validation vs. Validation Set

The validation set approach trades off between simplicity and stability compared to cross-validation. This implementation demonstrates both methods to highlight their differences in practice and computational requirements.

from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LogisticRegression

# Validation set approach
model_val = LogisticRegression(random_state=42)
model_val.fit(X_train, y_train)
val_score = model_val.score(X_val, y_val)

# Cross-validation approach
model_cv = LogisticRegression(random_state=42)
cv_scores = cross_val_score(model_cv, X, y, cv=5)

print(f"Validation set accuracy: {val_score:.3f}")
print(f"Cross-validation accuracy: {cv_scores.mean():.3f} ± {cv_scores.std()*2:.3f}")

Real-world Example: Credit Card Fraud Detection

In this practical application, we implement the validation set approach for credit card fraud detection, demonstrating proper handling of imbalanced datasets and feature preprocessing in a real-world context.

import pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import classification_report

# Simulated credit card transaction data
np.random.seed(42)
n_samples = 10000
n_features = 15

# Generate synthetic transaction data
X_transactions = np.random.randn(n_samples, n_features)
y_fraud = np.random.choice([0, 1], size=n_samples, p=[0.998, 0.002])

# Preprocess and split data
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X_transactions)

X_train, X_val, y_train, y_val = train_test_split(
    X_scaled, y_fraud, test_size=0.2, stratify=y_fraud
)

# Train model
model = LogisticRegression(class_weight='balanced')
model.fit(X_train, y_train)

# Evaluate
y_pred = model.predict(X_val)
print(classification_report(y_val, y_pred))

Validation Set Sizing Strategy

The size of validation set significantly impacts model evaluation reliability. This implementation demonstrates how different validation set sizes affect performance estimates using learning curves and error analysis.

import matplotlib.pyplot as plt
from sklearn.metrics import mean_squared_error

def analyze_validation_sizes(X, y, val_sizes=[0.1, 0.2, 0.3, 0.4]):
    results = {}
    for val_size in val_sizes:
        # Split data with different validation sizes
        X_t, X_v, y_t, y_v = train_test_split(
            X, y, test_size=val_size, random_state=42
        )
        
        # Train and evaluate model
        model = LogisticRegression()
        model.fit(X_t, y_t)
        train_score = model.score(X_t, y_t)
        val_score = model.score(X_v, y_v)
        
        results[val_size] = {
            'train_score': train_score,
            'val_score': val_score,
            'sample_size': len(X_v)
        }
    
    return results

# Example usage
results = analyze_validation_sizes(X, y)
for size, metrics in results.items():
    print(f"Val Size: {size:.1f} - Train: {metrics['train_score']:.3f} "
          f"Val: {metrics['val_score']:.3f} N={metrics['sample_size']}")

Stratified Validation Split Implementation

Stratified sampling ensures representative class distributions in both training and validation sets, crucial for imbalanced datasets. This implementation shows how to maintain class proportions during splitting.

def stratified_val_split(X, y, val_size=0.2, random_state=None):
    # Get unique classes and their indices
    classes = np.unique(y)
    class_indices = [np.where(y == c)[0] for c in classes]
    
    train_indices = []
    val_indices = []
    
    if random_state:
        np.random.seed(random_state)
    
    for indices in class_indices:
        # Shuffle indices for each class
        np.random.shuffle(indices)
        # Calculate split point
        n_val = int(len(indices) * val_size)
        
        val_indices.extend(indices[:n_val])
        train_indices.extend(indices[n_val:])
    
    return (X[train_indices], X[val_indices], 
            y[train_indices], y[val_indices])

# Example with imbalanced data
X_imb = np.random.randn(1000, 5)
y_imb = np.random.choice([0, 1], 1000, p=[0.95, 0.05])

X_train, X_val, y_train, y_val = stratified_val_split(X_imb, y_imb)
print(f"Training class distribution: {np.bincount(y_train) / len(y_train)}")
print(f"Validation class distribution: {np.bincount(y_val) / len(y_val)}")

Time Series Validation

Time series data requires special validation approaches to maintain temporal ordering. This implementation demonstrates proper validation set creation for time-dependent data.

import pandas as pd
from datetime import datetime, timedelta

# Generate time series data
dates = pd.date_range(start='2023-01-01', periods=1000, freq='D')
X_ts = np.random.randn(1000, 5)
y_ts = np.random.randn(1000)

def temporal_validation_split(X, y, dates, val_ratio=0.2):
    # Sort by date
    sort_idx = np.argsort(dates)
    X_sorted = X[sort_idx]
    y_sorted = y[sort_idx]
    
    # Split maintaining temporal order
    split_idx = int(len(X) * (1 - val_ratio))
    
    return (X_sorted[:split_idx], X_sorted[split_idx:],
            y_sorted[:split_idx], y_sorted[split_idx:])

X_train, X_val, y_train, y_val = temporal_validation_split(
    X_ts, y_ts, dates
)

print(f"Training period: {dates[0]} to {dates[800]}")
print(f"Validation period: {dates[801]} to {dates[-1]}")

Model Selection with Validation Sets

The validation set serves as a crucial tool for model selection and hyperparameter tuning. This implementation shows how to use validation performance to choose optimal model configurations.

from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC

def model_selection_with_validation(X_train, X_val, y_train, y_val):
    models = {
        'rf': RandomForestClassifier(random_state=42),
        'svm': SVC(random_state=42),
        'lr': LogisticRegression(random_state=42)
    }
    
    results = {}
    for name, model in models.items():
        # Train model
        model.fit(X_train, y_train)
        
        # Evaluate on both sets
        train_score = model.score(X_train, y_train)
        val_score = model.score(X_val, y_val)
        
        results[name] = {
            'train_score': train_score,
            'val_score': val_score,
            'gap': train_score - val_score
        }
    
    return results

# Example usage
selection_results = model_selection_with_validation(
    X_train, X_val, y_train, y_val
)
for model, scores in selection_results.items():
    print(f"\nModel: {model}")
    print(f"Training Score: {scores['train_score']:.3f}")
    print(f"Validation Score: {scores['val_score']:.3f}")
    print(f"Generalization Gap: {scores['gap']:.3f}")

Real-world Example: Customer Churn Prediction

A practical implementation of validation set approach in customer churn prediction, demonstrating feature engineering, model selection, and performance evaluation in a business context.

from sklearn.preprocessing import StandardScaler, LabelEncoder
from sklearn.ensemble import GradientBoostingClassifier

# Simulate customer data
np.random.seed(42)
n_customers = 5000

# Generate synthetic customer features
customer_data = {
    'usage_minutes': np.random.exponential(100, n_customers),
    'contract_length': np.random.choice([12, 24, 36], n_customers),
    'monthly_charges': np.random.normal(70, 25, n_customers),
    'support_calls': np.random.poisson(2, n_customers),
    'satisfaction_score': np.random.randint(1, 6, n_customers)
}

X_customers = pd.DataFrame(customer_data)
y_churn = (X_customers['satisfaction_score'] < 3) | (X_customers['support_calls'] > 4)

# Preprocess
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X_customers)

# Split with stratification
X_train, X_val, y_train, y_val = train_test_split(
    X_scaled, y_churn, test_size=0.2, stratify=y_churn, random_state=42
)

# Train model
churn_model = GradientBoostingClassifier(random_state=42)
churn_model.fit(X_train, y_train)

# Evaluate
y_pred = churn_model.predict(X_val)
y_prob = churn_model.predict_proba(X_val)[:, 1]

print("Churn Prediction Results:")
print(classification_report(y_val, y_pred))

Validation Set with K-Fold Averaging

This advanced technique combines validation set principles with k-fold cross-validation to provide more robust performance estimates while maintaining computational efficiency.

def k_fold_validation_set(X, y, k=5, val_size=0.2, random_state=42):
    np.random.seed(random_state)
    n_samples = len(X)
    
    # Generate k different validation splits
    val_scores = []
    for i in range(k):
        # Create different random split
        X_t, X_v, y_t, y_v = train_test_split(
            X, y, test_size=val_size, random_state=random_state+i
        )
        
        # Train and evaluate
        model = LogisticRegression()
        model.fit(X_t, y_t)
        val_scores.append(model.score(X_v, y_v))
    
    return {
        'mean_score': np.mean(val_scores),
        'std_score': np.std(val_scores),
        'individual_scores': val_scores
    }

# Example usage
results = k_fold_validation_set(X, y, k=5)
print(f"Average Validation Score: {results['mean_score']:.3f} "
      f"± {results['std_score']:.3f}")
print("Individual Fold Scores:", 
      [f"{score:.3f}" for score in results['individual_scores']])

Validation Set Size Impact Analysis

This implementation explores how validation set size affects model performance estimation reliability through empirical analysis and visualization of confidence intervals.

def analyze_validation_size_impact(X, y, sizes=[0.1, 0.2, 0.3, 0.4], repeats=10):
    results = {size: [] for size in sizes}
    
    for size in sizes:
        for _ in range(repeats):
            # Split with current size
            X_t, X_v, y_t, y_v = train_test_split(
                X, y, test_size=size, random_state=np.random.randint(0, 1000)
            )
            
            # Train and evaluate
            model = LogisticRegression()
            model.fit(X_t, y_t)
            val_score = model.score(X_v, y_v)
            results[size].append(val_score)
    
    # Calculate statistics
    stats = {}
    for size, scores in results.items():
        stats[size] = {
            'mean': np.mean(scores),
            'std': np.std(scores),
            'ci_lower': np.percentile(scores, 2.5),
            'ci_upper': np.percentile(scores, 97.5)
        }
    
    return stats

# Example usage
size_impact = analyze_validation_size_impact(X, y)
for size, metrics in size_impact.items():
    print(f"\nValidation Size: {size:.1f}")
    print(f"Mean Score: {metrics['mean']:.3f}")
    print(f"95% CI: [{metrics['ci_lower']:.3f}, {metrics['ci_upper']:.3f}]")

Handling Class Imbalance in Validation Sets

When dealing with imbalanced datasets, special attention must be given to validation set creation to ensure meaningful performance evaluation. This implementation demonstrates advanced techniques for handling class imbalance during validation.

from sklearn.metrics import precision_recall_curve
from imblearn.over_sampling import SMOTE

def balanced_validation_evaluation(X, y, val_size=0.2):
    # First split to avoid data leakage
    X_train, X_val, y_train, y_val = train_test_split(
        X, y, test_size=val_size, stratify=y, random_state=42
    )
    
    # Apply SMOTE only to training data
    smote = SMOTE(random_state=42)
    X_train_balanced, y_train_balanced = smote.fit_resample(X_train, y_train)
    
    # Train model on balanced data
    model = LogisticRegression(class_weight='balanced')
    model.fit(X_train_balanced, y_train_balanced)
    
    # Evaluate on original validation distribution
    y_pred_proba = model.predict_proba(X_val)[:, 1]
    precision, recall, thresholds = precision_recall_curve(y_val, y_pred_proba)
    
    results = {
        'precision': precision,
        'recall': recall,
        'thresholds': thresholds,
        'original_dist': np.bincount(y_val) / len(y_val),
        'training_dist': np.bincount(y_train_balanced) / len(y_train_balanced)
    }
    
    return results, model

# Example with imbalanced dataset
X_imb = np.random.randn(10000, 10)
y_imb = np.random.choice([0, 1], size=10000, p=[0.95, 0.05])

results, model = balanced_validation_evaluation(X_imb, y_imb)
print(f"Original class distribution: {results['original_dist']}")
print(f"Balanced training distribution: {results['training_dist']}")

Time-Based Rolling Validation

For time series problems, implementing a rolling validation scheme provides more robust model evaluation. This implementation shows how to create and evaluate using time-based validation windows.

def rolling_validation(X, y, dates, window_size=30, step_size=7):
    dates = pd.to_datetime(dates)
    results = []
    
    # Sort data by date
    sort_idx = np.argsort(dates)
    X = X[sort_idx]
    y = y[sort_idx]
    dates = dates[sort_idx]
    
    # Create rolling windows
    start_date = dates.min()
    end_date = dates.max()
    current_date = start_date
    
    while current_date + pd.Timedelta(days=window_size) <= end_date:
        # Define window
        train_mask = (dates < current_date)
        val_mask = (dates >= current_date) & \
                  (dates < current_date + pd.Timedelta(days=window_size))
        
        # Split data
        X_train = X[train_mask]
        X_val = X[val_mask]
        y_train = y[train_mask]
        y_val = y[val_mask]
        
        if len(X_train) > 0 and len(X_val) > 0:
            # Train and evaluate
            model = LogisticRegression()
            model.fit(X_train, y_train)
            score = model.score(X_val, y_val)
            
            results.append({
                'window_start': current_date,
                'window_end': current_date + pd.Timedelta(days=window_size),
                'score': score,
                'train_samples': len(X_train),
                'val_samples': len(X_val)
            })
        
        current_date += pd.Timedelta(days=step_size)
    
    return pd.DataFrame(results)

# Example usage with time series data
dates = pd.date_range('2023-01-01', '2023-12-31', freq='D')
X_ts = np.random.randn(len(dates), 5)
y_ts = np.random.randint(0, 2, len(dates))

rolling_results = rolling_validation(X_ts, y_ts, dates)
print("\nRolling Validation Results:")
print(f"Average Score: {rolling_results['score'].mean():.3f}")
print(f"Score Std: {rolling_results['score'].std():.3f}")

Additional Resources

  • "A Systematic Analysis of Performance Measures for Classification Tasks"
    • Search on Google Scholar for this paper by Marina Sokolova and Guy Lapalme
  • "Cross-validation pitfalls when selecting and assessing regression and classification models"
  • "Learning from imbalanced data: open challenges and future directions"
  • "On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation"
    • Search on Google Scholar for this paper by Gavin Cawley and Nicola Talbot
  • "A Survey on Deep Learning for Time Series Forecasting"

More in Model evaluation and validation

  1. Understanding the Silhouette Score in Clustering9 min

    The silhouette score is a metric used to evaluate the quality of clustering results. It measures how similar an object is to its own cluster compared to...

  2. Understanding Silhouette Score in Clustering7 min

    The silhouette score measures how similar an object is to its own cluster compared to other clusters. It ranges from -1 to +1, where higher values indicate...

  3. Understanding Silhouette Score for Clustering8 min

    The silhouette score measures how similar an object is to its own cluster compared to other clusters. It ranges from -1 to 1, where a high value indicates...

  4. Understanding ROC Curve AUC10 min

    An Area Under the Curve (AUC) of 0.5 in a Receiver Operating Characteristic (ROC) curve indicates that the classifier performs no better than random...

  5. Understanding Confusion Matrix in Machine Learning7 min

    A confusion matrix is a fundamental evaluation metric in machine learning that displays the performance of a classification model by comparing predicted...

  6. Understanding Confusion Matrices in Machine Learning8 min

    A confusion matrix is a fundamental evaluation metric in machine learning that tabulates the performance of a classification model by comparing predicted...