xbe.at

Machine learning / Regularization overfitting and optimization

Combating Overfitting with Regularization in Python

10 min read, 15 sections

Contents
  1. Understanding Overfitting
  2. Introduction to Regularization
  3. L1 Regularization (Lasso)
  4. L2 Regularization (Ridge)
  5. Elastic Net
  6. Early Stopping
  7. Dropout
  8. Data Augmentation
  9. Cross-Validation
  10. Ensemble Methods
  11. Hyperparameter Tuning
  12. Real-Life Example: House Price Prediction
  13. Real-Life Example: Image Classification with Regularization
  14. Comparing Regularization Techniques
  15. Additional Resources

Understanding Overfitting

Overfitting occurs when a machine learning model learns the training data too well, including its noise and fluctuations. This results in poor generalization to new, unseen data. Let's visualize this concept:

import numpy as np
import matplotlib.pyplot as plt
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression

# Generate sample data
np.random.seed(0)
X = np.sort(np.random.rand(20, 1), axis=0)
y = np.cos(1.5 * np.pi * X).ravel() + np.random.randn(20) * 0.1

# Fit models with varying degrees
X_test = np.linspace(0, 1, 100)[:, np.newaxis]
plt.figure(figsize=(10, 6))

for degree in [1, 4, 15]:
    poly_features = PolynomialFeatures(degree=degree, include_bias=False)
    X_poly = poly_features.fit_transform(X)
    model = LinearRegression()
    model.fit(X_poly, y)
    y_pred = model.predict(poly_features.transform(X_test))
    
    plt.plot(X_test, y_pred, label=f'Degree {degree}')

plt.scatter(X, y, color='r', label='Training points')
plt.xlabel('X')
plt.ylabel('y')
plt.legend()
plt.title('Polynomial Regression: Underfitting vs Overfitting')
plt.show()

Introduction to Regularization

Regularization is a technique used to prevent overfitting by adding a penalty term to the loss function. This encourages the model to have smaller weights, leading to a simpler model that generalizes better. Let's implement a simple linear regression with L2 regularization:

import numpy as np
from sklearn.linear_model import Ridge
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split

# Generate sample data
np.random.seed(0)
X = np.random.randn(100, 5)
y = 2 * X[:, 0] + 3 * X[:, 1] - X[:, 2] + 0.5 * X[:, 3] + np.random.randn(100) * 0.1

# Split the data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Standardize features
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Train Ridge regression model
ridge = Ridge(alpha=1.0)
ridge.fit(X_train_scaled, y_train)

# Print coefficients
print("Ridge regression coefficients:")
for i, coef in enumerate(ridge.coef_):
    print(f"Feature {i+1}: {coef:.4f}")

# Evaluate model
train_score = ridge.score(X_train_scaled, y_train)
test_score = ridge.score(X_test_scaled, y_test)
print(f"Train R² score: {train_score:.4f}")
print(f"Test R² score: {test_score:.4f}")

L1 Regularization (Lasso)

L1 regularization, also known as Lasso (Least Absolute Shrinkage and Selection Operator), adds the absolute value of the coefficients as a penalty term. This can lead to sparse models by driving some coefficients to exactly zero. Let's implement Lasso regression:

from sklearn.linear_model import Lasso

# Train Lasso regression model
lasso = Lasso(alpha=0.1)
lasso.fit(X_train_scaled, y_train)

# Print coefficients
print("Lasso regression coefficients:")
for i, coef in enumerate(lasso.coef_):
    print(f"Feature {i+1}: {coef:.4f}")

# Evaluate model
train_score = lasso.score(X_train_scaled, y_train)
test_score = lasso.score(X_test_scaled, y_test)
print(f"Train R² score: {train_score:.4f}")
print(f"Test R² score: {test_score:.4f}")

L2 Regularization (Ridge)

L2 regularization, also known as Ridge regression, adds the squared magnitude of the coefficients as a penalty term. This encourages smaller weights across all features. We've already seen Ridge regression in action, but let's compare it with Lasso:

import matplotlib.pyplot as plt

# Train models with different alphas
alphas = [0.01, 0.1, 1, 10, 100]
lasso_coefs = []
ridge_coefs = []

for alpha in alphas:
    lasso = Lasso(alpha=alpha)
    ridge = Ridge(alpha=alpha)
    lasso.fit(X_train_scaled, y_train)
    ridge.fit(X_train_scaled, y_train)
    lasso_coefs.append(lasso.coef_)
    ridge_coefs.append(ridge.coef_)

# Plot coefficients
plt.figure(figsize=(12, 6))
plt.subplot(121)
plt.plot(alphas, lasso_coefs)
plt.xscale('log')
plt.xlabel('Alpha')
plt.ylabel('Coefficients')
plt.title('Lasso Coefficients')

plt.subplot(122)
plt.plot(alphas, ridge_coefs)
plt.xscale('log')
plt.xlabel('Alpha')
plt.ylabel('Coefficients')
plt.title('Ridge Coefficients')

plt.tight_layout()
plt.show()

Elastic Net

Elastic Net combines L1 and L2 regularization, providing a balance between feature selection (L1) and handling correlated features (L2). Let's implement Elastic Net:

from sklearn.linear_model import ElasticNet

# Train Elastic Net model
elastic_net = ElasticNet(alpha=0.1, l1_ratio=0.5)
elastic_net.fit(X_train_scaled, y_train)

# Print coefficients
print("Elastic Net coefficients:")
for i, coef in enumerate(elastic_net.coef_):
    print(f"Feature {i+1}: {coef:.4f}")

# Evaluate model
train_score = elastic_net.score(X_train_scaled, y_train)
test_score = elastic_net.score(X_test_scaled, y_test)
print(f"Train R² score: {train_score:.4f}")
print(f"Test R² score: {test_score:.4f}")

Early Stopping

Early stopping is a form of regularization that stops the training process when the model's performance on a validation set starts to degrade. Let's implement early stopping with a simple neural network:

import tensorflow as tf
from sklearn.model_selection import train_test_split

# Prepare data
X_train, X_val, y_train, y_val = train_test_split(X_train_scaled, y_train, test_size=0.2, random_state=42)

# Define model
model = tf.keras.Sequential([
    tf.keras.layers.Dense(64, activation='relu', input_shape=(5,)),
    tf.keras.layers.Dense(32, activation='relu'),
    tf.keras.layers.Dense(1)
])

model.compile(optimizer='adam', loss='mse')

# Define early stopping callback
early_stopping = tf.keras.callbacks.EarlyStopping(
    monitor='val_loss', patience=10, restore_best_weights=True)

# Train model
history = model.fit(X_train, y_train, epochs=1000, batch_size=32,
                    validation_data=(X_val, y_val), callbacks=[early_stopping],
                    verbose=0)

# Plot training history
plt.plot(history.history['loss'], label='Training Loss')
plt.plot(history.history['val_loss'], label='Validation Loss')
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.legend()
plt.title('Training History with Early Stopping')
plt.show()

Dropout

Dropout is a regularization technique specific to neural networks. It randomly sets a fraction of input units to 0 at each update during training, which helps prevent overfitting. Let's modify our previous neural network to include dropout:

# Define model with dropout
model_dropout = tf.keras.Sequential([
    tf.keras.layers.Dense(64, activation='relu', input_shape=(5,)),
    tf.keras.layers.Dropout(0.3),
    tf.keras.layers.Dense(32, activation='relu'),
    tf.keras.layers.Dropout(0.3),
    tf.keras.layers.Dense(1)
])

model_dropout.compile(optimizer='adam', loss='mse')

# Train model
history_dropout = model_dropout.fit(X_train, y_train, epochs=1000, batch_size=32,
                                    validation_data=(X_val, y_val), callbacks=[early_stopping],
                                    verbose=0)

# Plot training history
plt.plot(history_dropout.history['loss'], label='Training Loss (with Dropout)')
plt.plot(history_dropout.history['val_loss'], label='Validation Loss (with Dropout)')
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.legend()
plt.title('Training History with Dropout')
plt.show()

Data Augmentation

Data augmentation is a technique to increase the diversity of your training set by applying random (but realistic) transformations. While commonly used in image processing, it can be applied to other types of data as well. Let's implement a simple data augmentation for our regression problem:

def augment_data(X, y, noise_factor=0.05):
    X_aug = X + np.random.normal(0, noise_factor, X.shape)
    y_aug = y + np.random.normal(0, noise_factor, y.shape)
    return np.vstack((X, X_aug)), np.hstack((y, y_aug))

# Augment training data
X_train_aug, y_train_aug = augment_data(X_train, y_train)

# Train model on augmented data
model_aug = tf.keras.Sequential([
    tf.keras.layers.Dense(64, activation='relu', input_shape=(5,)),
    tf.keras.layers.Dense(32, activation='relu'),
    tf.keras.layers.Dense(1)
])

model_aug.compile(optimizer='adam', loss='mse')

history_aug = model_aug.fit(X_train_aug, y_train_aug, epochs=1000, batch_size=32,
                            validation_data=(X_val, y_val), callbacks=[early_stopping],
                            verbose=0)

# Plot training history
plt.plot(history_aug.history['loss'], label='Training Loss (Augmented)')
plt.plot(history_aug.history['val_loss'], label='Validation Loss (Augmented)')
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.legend()
plt.title('Training History with Data Augmentation')
plt.show()

Cross-Validation

Cross-validation is a resampling procedure used to evaluate machine learning models on a limited data sample. It helps in assessing how the model will generalize to an independent dataset. Let's implement k-fold cross-validation:

from sklearn.model_selection import cross_val_score
from sklearn.linear_model import Ridge

# Perform 5-fold cross-validation
cv_scores = cross_val_score(Ridge(alpha=1.0), X_train_scaled, y_train, cv=5, scoring='neg_mean_squared_error')

# Convert MSE to RMSE
rmse_scores = np.sqrt(-cv_scores)

print("Cross-validation RMSE scores:", rmse_scores)
print("Mean RMSE:", rmse_scores.mean())
print("Standard deviation of RMSE:", rmse_scores.std())

# Visualize cross-validation results
plt.boxplot(rmse_scores)
plt.title('5-Fold Cross-Validation RMSE Scores')
plt.ylabel('RMSE')
plt.show()

Ensemble Methods

Ensemble methods combine multiple models to create a more robust and accurate model. Random Forests and Gradient Boosting are popular ensemble methods that inherently help prevent overfitting. Let's implement a Random Forest regressor:

from sklearn.ensemble import RandomForestRegressor

# Train Random Forest model
rf_model = RandomForestRegressor(n_estimators=100, random_state=42)
rf_model.fit(X_train_scaled, y_train)

# Evaluate model
train_score = rf_model.score(X_train_scaled, y_train)
test_score = rf_model.score(X_test_scaled, y_test)
print(f"Random Forest - Train R² score: {train_score:.4f}")
print(f"Random Forest - Test R² score: {test_score:.4f}")

# Feature importance
importances = rf_model.feature_importances_
indices = np.argsort(importances)[::-1]

plt.figure(figsize=(10, 6))
plt.title("Feature Importances")
plt.bar(range(X.shape[1]), importances[indices])
plt.xticks(range(X.shape[1]), [f"Feature {i+1}" for i in indices])
plt.xlabel("Features")
plt.ylabel("Importance")
plt.tight_layout()
plt.show()

Hyperparameter Tuning

Hyperparameter tuning is crucial for finding the right balance between model complexity and generalization. Let's use GridSearchCV to find the best hyperparameters for our Ridge regression model:

from sklearn.model_selection import GridSearchCV

# Define parameter grid
param_grid = {
    'alpha': [0.01, 0.1, 1, 10, 100],
    'fit_intercept': [True, False],
    'solver': ['auto', 'svd', 'cholesky', 'lsqr', 'sparse_cg']
}

# Perform grid search
grid_search = GridSearchCV(Ridge(), param_grid, cv=5, scoring='neg_mean_squared_error')
grid_search.fit(X_train_scaled, y_train)

# Print best parameters and score
print("Best parameters:", grid_search.best_params_)
print("Best RMSE:", np.sqrt(-grid_search.best_score_))

# Evaluate best model
best_model = grid_search.best_estimator_
train_score = best_model.score(X_train_scaled, y_train)
test_score = best_model.score(X_test_scaled, y_test)
print(f"Best model - Train R² score: {train_score:.4f}")
print(f"Best model - Test R² score: {test_score:.4f}")

Real-Life Example: House Price Prediction

Let's apply regularization techniques to predict house prices based on various features using the Boston Housing dataset:

from sklearn.datasets import load_boston
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LinearRegression, Ridge, Lasso

# Load Boston Housing dataset
boston = load_boston()
X, y = boston.data, boston.target

# Split the data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Standardize features
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Train models
models = {
    'Linear Regression': LinearRegression(),
    'Ridge Regression': Ridge(alpha=1.0),
    'Lasso Regression': Lasso(alpha=0.1)
}

# Evaluate models
for name, model in models.items():
    model.fit(X_train_scaled, y_train)
    train_score = model.score(X_train_scaled, y_train)
    test_score = model.score(X_test_scaled, y_test)
    print(f"{name}:")
    print(f"  Train R² score: {train_score:.4f}")
    print(f"  Test R² score: {test_score:.4f}")

# Plot feature importance for Lasso
lasso_model = models['Lasso Regression']
feature_importance = np.abs(lasso_model.coef_)
feature_names = boston.feature_names

plt.figure(figsize=(10, 6))
plt.bar(feature_names, feature_importance)
plt.title("Feature Importance (Lasso Regression)")
plt.xlabel("Features")
plt.ylabel("Absolute Coefficient Value")
plt.xticks(rotation=90)
plt.tight_layout()
plt.show()

Real-Life Example: Image Classification with Regularization

In this example, we'll use a convolutional neural network (CNN) with regularization techniques to classify images from the CIFAR-10 dataset:

import tensorflow as tf
from tensorflow.keras import layers, models
from tensorflow.keras.datasets import cifar10

# Load and preprocess data
(train_images, train_labels), (test_images, test_labels) = cifar10.load_data()
train_images, test_images = train_images / 255.0, test_images / 255.0

# Define model with regularization
model = models.Sequential([
    layers.Conv2D(32, (3, 3), activation='relu', input_shape=(32, 32, 3),
                  kernel_regularizer=tf.keras.regularizers.l2(0.001)),
    layers.MaxPooling2D((2, 2)),
    layers.Conv2D(64, (3, 3), activation='relu',
                  kernel_regularizer=tf.keras.regularizers.l2(0.001)),
    layers.MaxPooling2D((2, 2)),
    layers.Conv2D(64, (3, 3), activation='relu',
                  kernel_regularizer=tf.keras.regularizers.l2(0.001)),
    layers.Flatten(),
    layers.Dense(64, activation='relu',
                 kernel_regularizer=tf.keras.regularizers.l2(0.001)),
    layers.Dropout(0.5),
    layers.Dense(10)
])

# Compile and train model
model.compile(optimizer='adam',
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
              metrics=['accuracy'])

history = model.fit(train_images, train_labels, epochs=10, 
                    validation_data=(test_images, test_labels))

# Plot training history
plt.plot(history.history['accuracy'], label='Training Accuracy')
plt.plot(history.history['val_accuracy'], label='Validation Accuracy')
plt.xlabel('Epoch')
plt.ylabel('Accuracy')
plt.legend()
plt.title('Model Accuracy with Regularization')
plt.show()

Comparing Regularization Techniques

Let's compare the performance of different regularization techniques on a synthetic dataset:

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LinearRegression, Ridge, Lasso, ElasticNet
from sklearn.metrics import mean_squared_error

# Generate synthetic data
np.random.seed(42)
X = np.random.randn(1000, 20)
y = 5 * X[:, 0] + 3 * X[:, 1] - 2 * X[:, 2] + np.random.randn(1000) * 0.5

# Split and scale data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Define models
models = {
    'Linear Regression': LinearRegression(),
    'Ridge Regression': Ridge(alpha=1.0),
    'Lasso Regression': Lasso(alpha=0.1),
    'ElasticNet': ElasticNet(alpha=0.1, l1_ratio=0.5)
}

# Train and evaluate models
results = {}
for name, model in models.items():
    model.fit(X_train_scaled, y_train)
    y_pred = model.predict(X_test_scaled)
    mse = mean_squared_error(y_test, y_pred)
    results[name] = mse

# Plot results
plt.figure(figsize=(10, 6))
plt.bar(results.keys(), results.values())
plt.title('Mean Squared Error for Different Regularization Techniques')
plt.xlabel('Model')
plt.ylabel('Mean Squared Error')
plt.xticks(rotation=45)
plt.tight_layout()
plt.show()

Additional Resources

For more information on regularization techniques and fighting overfitting, consider exploring these resources:

  1. "Regularization for Machine Learning: L1, L2, and Elastic Net" by Jason Brownlee ArXiv URL: https://arxiv.org/abs/1908.03930
  2. "A Survey of Deep Learning Techniques for Neural Machine Translation" by Shuohang Wang and Jing Jiang ArXiv URL: https://arxiv.org/abs/1703.03906
  3. "Dropout: A Simple Way to Prevent Neural Networks from Overfitting" by Nitish Srivastava et al. ArXiv URL: https://arxiv.org/abs/1207.0580

These papers provide in-depth discussions on various regularization techniques and their applications in different machine learning domains.

More in Regularization overfitting and optimization

  1. Diagnosing Overfitting in Machine Learning Models11 min

    In machine learning, high training accuracy coupled with poor test accuracy is a classic symptom of overfitting. This occurs when a model learns the training...

  2. Bias-Variance Tradeoff in Machine Learning6 min

    The bias-variance tradeoff is a fundamental concept in machine learning that explains the balance between two types of errors in predictive models. This...

  3. Early Stopping Preventing Overfitting in Machine Learning8 min

    Early stopping serves as a regularization technique that monitors model performance during training by evaluating a validation set after each epoch. When the...

  4. Bias-Variance Tradeoff Clearly Explained18 min

    The bias-variance decomposition mathematically breaks down the prediction error of a machine learning model into its fundamental components. Understanding...

  5. Early Stopping vs Callbacks in Machine Learning Using Python11 min

    Introduction to Early Stopping in Machine Learning

  6. Bias-Variance Trade-off in Machine Learning with Python9 min

    The bias-variance trade-off is a fundamental concept in machine learning that helps us understand the balance between model complexity and generalization....