xbe.at

Machine learning / Regularization overfitting and optimization

Overfitting in Machine Learning

8 min read, 14 sections

Contents
  1. Understanding Overfitting in Machine Learning
  2. Detecting Overfitting Through Learning Curves
  3. Cross-Validation for Overfitting Detection
  4. Implementing Early Stopping
  5. L1 and L2 Regularization to Combat Overfitting
  6. Dropout Implementation for Neural Networks
  7. Data Augmentation to Prevent Overfitting
  8. Ensemble Methods for Reducing Overfitting
  9. K-Fold Cross-Validation Implementation
  10. Bias-Variance Decomposition Analysis
  11. Validation Curves Implementation
  12. Model Complexity Analysis
  13. Real-world Example - Housing Price Prediction
  14. Additional Resources

Understanding Overfitting in Machine Learning

Overfitting occurs when a machine learning model learns the training data too perfectly, including its noise and outliers, resulting in poor generalization to new, unseen data. This phenomenon is characterized by high training accuracy but significantly lower test accuracy.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression

# Generate synthetic data
np.random.seed(42)
X = 2 * np.random.rand(100, 1)
y = 3 * X + np.random.randn(100, 1) * 0.2

# Create polynomial features
poly = PolynomialFeatures(degree=15)
X_poly = poly.fit_transform(X)

# Train model
model = LinearRegression()
model.fit(X_poly, y)

# Predictions
X_test = np.linspace(0, 2, 100).reshape(-1, 1)
X_test_poly = poly.transform(X_test)
y_pred = model.predict(X_test_poly)

# Plot results
plt.scatter(X, y, label='Training data')
plt.plot(X_test, y_pred, 'r-', label='Overfitted model')
plt.xlabel('X')
plt.ylabel('y')
plt.legend()
plt.show()

Detecting Overfitting Through Learning Curves

Learning curves provide a visual diagnostic tool to identify overfitting by plotting training and validation errors over time or epochs. Diverging curves indicate the model is memorizing rather than learning generalizable patterns.

from sklearn.model_selection import learning_curve

def plot_learning_curves(model, X, y):
    train_sizes, train_scores, val_scores = learning_curve(
        model, X, y, cv=5, n_jobs=-1, 
        train_sizes=np.linspace(0.1, 1.0, 10),
        scoring='neg_mean_squared_error'
    )
    
    train_scores_mean = -np.mean(train_scores, axis=1)
    val_scores_mean = -np.mean(val_scores, axis=1)
    
    plt.plot(train_sizes, train_scores_mean, label='Training error')
    plt.plot(train_sizes, val_scores_mean, label='Validation error')
    plt.xlabel('Training set size')
    plt.ylabel('Mean Squared Error')
    plt.legend()
    plt.show()

# Example usage
model = LinearRegression()
plot_learning_curves(model, X_poly, y)

Cross-Validation for Overfitting Detection

Cross-validation provides a robust method to assess model generalization by partitioning data into multiple training and validation sets. This systematic approach helps identify overfitting by comparing performance across different data splits.

from sklearn.model_selection import cross_val_score
from sklearn.model_selection import KFold

def assess_overfitting(model, X, y, cv=5):
    # Perform k-fold cross-validation
    kfold = KFold(n_splits=cv, shuffle=True, random_state=42)
    
    # Get training and validation scores
    train_scores = []
    val_scores = []
    
    for train_idx, val_idx in kfold.split(X):
        X_train, X_val = X[train_idx], X[val_idx]
        y_train, y_val = y[train_idx], y[val_idx]
        
        model.fit(X_train, y_train)
        train_scores.append(model.score(X_train, y_train))
        val_scores.append(model.score(X_val, y_val))
    
    print(f"Training scores: {np.mean(train_scores):.3f} ± {np.std(train_scores):.3f}")
    print(f"Validation scores: {np.mean(val_scores):.3f} ± {np.std(val_scores):.3f}")
    
    return train_scores, val_scores

Implementing Early Stopping

Early stopping prevents overfitting by monitoring validation performance during training and stopping when it begins to deteriorate. This technique is particularly useful for neural networks and gradient boosting models.

import tensorflow as tf
from tensorflow.keras.callbacks import EarlyStopping

def create_model_with_early_stopping(X_train, y_train, X_val, y_val):
    model = tf.keras.Sequential([
        tf.keras.layers.Dense(64, activation='relu'),
        tf.keras.layers.Dense(32, activation='relu'),
        tf.keras.layers.Dense(1)
    ])
    
    early_stopping = EarlyStopping(
        monitor='val_loss',
        patience=5,
        restore_best_weights=True
    )
    
    model.compile(optimizer='adam', loss='mse')
    
    history = model.fit(
        X_train, y_train,
        validation_data=(X_val, y_val),
        epochs=100,
        callbacks=[early_stopping],
        verbose=0
    )
    
    return model, history

L1 and L2 Regularization to Combat Overfitting

Regularization techniques add penalty terms to the loss function, discouraging complex models that might overfit. L1 regularization promotes sparsity, while L2 prevents extreme parameter values through weight decay.

from sklearn.linear_model import Ridge, Lasso

def compare_regularization(X_train, X_test, y_train, y_test):
    # L1 Regularization (Lasso)
    lasso = Lasso(alpha=0.1)
    lasso.fit(X_train, y_train)
    lasso_score = lasso.score(X_test, y_test)
    
    # L2 Regularization (Ridge)
    ridge = Ridge(alpha=0.1)
    ridge.fit(X_train, y_train)
    ridge_score = ridge.score(X_test, y_test)
    
    print(f"Lasso R2 Score: {lasso_score:.3f}")
    print(f"Ridge R2 Score: {ridge_score:.3f}")
    
    # Plot coefficients
    plt.figure(figsize=(10, 5))
    plt.subplot(1, 2, 1)
    plt.stem(lasso.coef_)
    plt.title('Lasso Coefficients')
    plt.subplot(1, 2, 2)
    plt.stem(ridge.coef_)
    plt.title('Ridge Coefficients')
    plt.show()

Dropout Implementation for Neural Networks

Dropout is a powerful regularization technique that randomly deactivates neurons during training, forcing the network to learn robust features and prevent co-adaptation of neurons, effectively reducing overfitting.

import tensorflow as tf

def create_model_with_dropout():
    model = tf.keras.Sequential([
        tf.keras.layers.Dense(128, activation='relu'),
        tf.keras.layers.Dropout(0.3),
        tf.keras.layers.Dense(64, activation='relu'),
        tf.keras.layers.Dropout(0.2),
        tf.keras.layers.Dense(1)
    ])
    
    model.compile(
        optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),
        loss='mse',
        metrics=['mae']
    )
    
    return model

# Example usage
X_train_normalized = (X_train - X_train.mean()) / X_train.std()
model = create_model_with_dropout()
history = model.fit(
    X_train_normalized, y_train,
    validation_split=0.2,
    epochs=100,
    batch_size=32,
    verbose=0
)

Data Augmentation to Prevent Overfitting

Data augmentation artificially increases the training set size by creating modified versions of existing samples, helping models learn invariant features and reduce overfitting, particularly effective in computer vision tasks.

from tensorflow.keras.preprocessing.image import ImageDataGenerator

def create_augmentation_pipeline():
    datagen = ImageDataGenerator(
        rotation_range=20,
        width_shift_range=0.2,
        height_shift_range=0.2,
        horizontal_flip=True,
        fill_mode='nearest'
    )
    
    # Example usage with a sample image
    img = np.random.rand(1, 28, 28, 1)  # Sample image shape
    
    # Generate augmented samples
    aug_iterator = datagen.flow(img, batch_size=1)
    
    # Display original and augmented images
    plt.figure(figsize=(10, 2))
    for i in range(5):
        plt.subplot(1, 5, i+1)
        plt.imshow(next(aug_iterator)[0].reshape(28, 28), cmap='gray')
        plt.axis('off')
    plt.show()
    
    return datagen

Ensemble Methods for Reducing Overfitting

Ensemble methods combine multiple models to create a more robust predictor, reducing overfitting through model averaging and diversification. This implementation demonstrates bagging and random forest approaches.

from sklearn.ensemble import BaggingRegressor, RandomForestRegressor
from sklearn.tree import DecisionTreeRegressor

def compare_ensemble_methods(X_train, X_test, y_train, y_test):
    # Base model (Decision Tree)
    base_model = DecisionTreeRegressor(max_depth=5)
    base_score = base_model.fit(X_train, y_train).score(X_test, y_test)
    
    # Bagging
    bagging = BaggingRegressor(
        base_estimator=DecisionTreeRegressor(max_depth=5),
        n_estimators=100,
        random_state=42
    )
    bagging_score = bagging.fit(X_train, y_train).score(X_test, y_test)
    
    # Random Forest
    rf = RandomForestRegressor(
        n_estimators=100,
        max_depth=5,
        random_state=42
    )
    rf_score = rf.fit(X_train, y_train).score(X_test, y_test)
    
    print(f"Base Model R2: {base_score:.3f}")
    print(f"Bagging R2: {bagging_score:.3f}")
    print(f"Random Forest R2: {rf_score:.3f}")

K-Fold Cross-Validation Implementation

K-fold cross-validation systematically evaluates model performance by splitting data into k subsets, training on k-1 folds and validating on the remaining fold. This comprehensive approach provides robust overfitting detection.

import numpy as np
from sklearn.model_selection import KFold
from sklearn.metrics import mean_squared_error, r2_score

def advanced_kfold_validation(model, X, y, k=5):
    kf = KFold(n_splits=k, shuffle=True, random_state=42)
    train_scores = []
    val_scores = []
    mse_scores = []
    
    for fold, (train_idx, val_idx) in enumerate(kf.split(X)):
        X_train_fold = X[train_idx]
        y_train_fold = y[train_idx]
        X_val_fold = X[val_idx]
        y_val_fold = y[val_idx]
        
        model.fit(X_train_fold, y_train_fold)
        
        train_pred = model.predict(X_train_fold)
        val_pred = model.predict(X_val_fold)
        
        train_scores.append(r2_score(y_train_fold, train_pred))
        val_scores.append(r2_score(y_val_fold, val_pred))
        mse_scores.append(mean_squared_error(y_val_fold, val_pred))
        
        print(f"Fold {fold+1}:")
        print(f"Training R2: {train_scores[-1]:.3f}")
        print(f"Validation R2: {val_scores[-1]:.3f}")
        print(f"MSE: {mse_scores[-1]:.3f}\n")
    
    return np.mean(train_scores), np.mean(val_scores), np.mean(mse_scores)

Bias-Variance Decomposition Analysis

Bias-variance decomposition helps understand the trade-off between underfitting and overfitting by analyzing prediction errors in terms of bias, variance, and irreducible error components.

def bias_variance_decomposition(model, X_train, X_test, y_train, y_test, n_bootstraps=100):
    predictions = np.zeros((n_bootstraps, len(X_test)))
    
    for i in range(n_bootstraps):
        # Bootstrap sample
        indices = np.random.randint(0, len(X_train), len(X_train))
        X_boot = X_train[indices]
        y_boot = y_train[indices]
        
        # Fit model and predict
        model.fit(X_boot, y_boot)
        predictions[i, :] = model.predict(X_test).ravel()
    
    # Calculate statistics
    expected_pred = np.mean(predictions, axis=0)
    bias = np.mean((y_test - expected_pred) ** 2)
    variance = np.mean(np.var(predictions, axis=0))
    
    print(f"Bias² : {bias:.3f}")
    print(f"Variance: {variance:.3f}")
    print(f"Expected Total Error: {bias + variance:.3f}")
    
    return bias, variance

Validation Curves Implementation

Validation curves analyze model performance across different hyperparameter values, helping identify the optimal complexity that balances underfitting and overfitting.

from sklearn.model_selection import validation_curve

def plot_validation_curves(model, X, y, param_name, param_range):
    train_scores, val_scores = validation_curve(
        model, X, y,
        param_name=param_name,
        param_range=param_range,
        cv=5,
        scoring='neg_mean_squared_error'
    )
    
    train_mean = -np.mean(train_scores, axis=1)
    train_std = np.std(train_scores, axis=1)
    val_mean = -np.mean(val_scores, axis=1)
    val_std = np.std(val_scores, axis=1)
    
    plt.figure(figsize=(10, 6))
    plt.plot(param_range, train_mean, label='Training score')
    plt.fill_between(param_range, train_mean - train_std,
                     train_mean + train_std, alpha=0.1)
    plt.plot(param_range, val_mean, label='Cross-validation score')
    plt.fill_between(param_range, val_mean - val_std,
                     val_mean + val_std, alpha=0.1)
    plt.xlabel(param_name)
    plt.ylabel('Mean Squared Error')
    plt.legend()
    plt.show()

Model Complexity Analysis

Model complexity analysis examines the relationship between model sophistication and performance, helping identify the optimal level of complexity that prevents both underfitting and overfitting through systematic evaluation.

def analyze_model_complexity(X_train, X_test, y_train, y_test, max_degree=15):
    train_scores = []
    test_scores = []
    model_complexities = range(1, max_degree + 1)
    
    for degree in model_complexities:
        # Create polynomial features
        poly = PolynomialFeatures(degree=degree)
        X_train_poly = poly.fit_transform(X_train)
        X_test_poly = poly.transform(X_test)
        
        # Train model
        model = LinearRegression()
        model.fit(X_train_poly, y_train)
        
        # Calculate scores
        train_scores.append(model.score(X_train_poly, y_train))
        test_scores.append(model.score(X_test_poly, y_test))
    
    plt.figure(figsize=(10, 6))
    plt.plot(model_complexities, train_scores, label='Training Score')
    plt.plot(model_complexities, test_scores, label='Test Score')
    plt.xlabel('Model Complexity (Polynomial Degree)')
    plt.ylabel('R² Score')
    plt.legend()
    plt.grid(True)
    plt.show()
    
    return train_scores, test_scores

Real-world Example - Housing Price Prediction

This implementation demonstrates overfitting detection and prevention in a practical housing price prediction scenario, incorporating data preprocessing, feature engineering, and model evaluation.

from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
import pandas as pd

def housing_price_analysis():
    # Generate synthetic housing data
    np.random.seed(42)
    n_samples = 1000
    
    # Features: size, bedrooms, age, location_score
    X = np.random.rand(n_samples, 4)
    X[:, 1] = np.round(X[:, 1] * 5 + 1)  # bedrooms 1-6
    X[:, 2] = X[:, 2] * 50  # age 0-50 years
    
    # Price with some noise
    y = (300000 + X[:, 0] * 200000 + X[:, 1] * 50000 - 
         X[:, 2] * 2000 + X[:, 3] * 100000 + 
         np.random.randn(n_samples) * 20000)
    
    # Split and scale data
    X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size=0.2, random_state=42)
    
    scaler = StandardScaler()
    X_train_scaled = scaler.fit_transform(X_train)
    X_test_scaled = scaler.transform(X_test)
    
    # Train multiple models
    models = {
        'Linear': LinearRegression(),
        'Ridge': Ridge(alpha=1.0),
        'Lasso': Lasso(alpha=1.0)
    }
    
    results = {}
    for name, model in models.items():
        model.fit(X_train_scaled, y_train)
        train_score = model.score(X_train_scaled, y_train)
        test_score = model.score(X_test_scaled, y_test)
        results[name] = {'train_score': train_score, 'test_score': test_score}
        
        print(f"{name} Results:")
        print(f"Training R²: {train_score:.3f}")
        print(f"Test R²: {test_score:.3f}")
        print(f"Overfitting gap: {train_score - test_score:.3f}\n")
    
    return results

Additional Resources

More in Regularization overfitting and optimization

  1. Parameters vs Hyperparameters in Machine Learning9 min

    Parameters and hyperparameters are fundamental concepts in machine learning, but they are often confused. This presentation will clarify the differences...

  2. Model Capacity in Machine Learning with Python14 min

    Model capacity refers to a model's ability to capture complex patterns in data. It's influenced by factors like the number of parameters, model architecture...

  3. Pitfalls of Zero Initialization in Machine Learning11 min

    The zero initialization trap is a common mistake in machine learning where model parameters (weights) are initialized with zeros. This approach can severely...

  4. Mastering Hyperparameter Tuning in Machine Learning with Python8 min

    Hyperparameter tuning is a crucial step in optimizing machine learning models. It involves adjusting the configuration settings that control the learning...

  5. Pitfalls of Zero Initialization in Machine Learning Models7 min

    Neural networks initialized with zero weights exhibit symmetric learning patterns across all neurons, leading to identical gradients during backpropagation....

  6. Mastering Grid Search and Random Search for Hyperparameter Optimization8 min

    Grid search is a systematic hyperparameter optimization technique that works by exhaustively searching through a predefined set of hyperparameter values....