xbe.at

Machine learning / Regularization overfitting and optimization

Avoiding Overfitting in Machine Learning with Python

10 min read, 15 sections

Contents
  1. Understanding Overfitting in Machine Learning
  2. Underfitting vs. Overfitting
  3. Bias-Variance Tradeoff
  4. Cross-Validation
  5. Regularization: Ridge Regression
  6. Regularization: Lasso Regression
  7. Early Stopping
  8. Dropout
  9. Real-life Example: Image Classification
  10. Real-life Example: Sentiment Analysis
  11. Detecting Overfitting
  12. Preventing Overfitting: Data Augmentation
  13. Feature Selection and Dimensionality Reduction
  14. Ensemble Methods
  15. Additional Resources

Understanding Overfitting in Machine Learning

Overfitting occurs when a machine learning model learns the training data too well, capturing noise and specific data points rather than general patterns. This leads to poor generalization on unseen data. Let's explore this concept with Python examples.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.polynomial_features import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

# Generate sample data
np.random.seed(0)
X = np.sort(5 * np.random.rand(80, 1), axis=0)
y = np.sin(X).ravel() + np.random.normal(0, 0.1, X.shape[0])

# Split data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Plot the data
plt.scatter(X_train, y_train, color='blue', label='Training data')
plt.scatter(X_test, y_test, color='red', label='Testing data')
plt.legend()
plt.title('Sample Data')
plt.show()

Underfitting vs. Overfitting

Underfitting occurs when a model is too simple to capture the underlying patterns in the data. Overfitting happens when a model is too complex and fits the noise in the training data. The goal is to find the right balance.

# Create models with different complexities
degrees = [1, 3, 15]
plt.figure(figsize=(14, 4))

for i, degree in enumerate(degrees):
    ax = plt.subplot(1, 3, i + 1)
    poly_features = PolynomialFeatures(degree=degree, include_bias=False)
    X_poly = poly_features.fit_transform(X_train)
    
    model = LinearRegression()
    model.fit(X_poly, y_train)
    
    X_plot = np.linspace(0, 5, 100).reshape(-1, 1)
    X_plot_poly = poly_features.transform(X_plot)
    y_plot = model.predict(X_plot_poly)
    
    plt.scatter(X_train, y_train, color='blue', s=10, label='Training data')
    plt.plot(X_plot, y_plot, color='red', label=f'Degree {degree}')
    plt.title(f'Polynomial Degree {degree}')
    plt.legend()

plt.tight_layout()
plt.show()

Bias-Variance Tradeoff

The bias-variance tradeoff is a fundamental concept in machine learning. Bias refers to the error introduced by approximating a real-world problem with a simplified model. Variance is the model's sensitivity to small fluctuations in the training data.

def bias_variance_demo(X, y, test_size=0.2, degrees=range(1, 20), num_trials=100):
    mse_scores = {degree: [] for degree in degrees}
    
    for _ in range(num_trials):
        X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=test_size)
        
        for degree in degrees:
            poly_features = PolynomialFeatures(degree=degree, include_bias=False)
            X_poly = poly_features.fit_transform(X_train)
            X_test_poly = poly_features.transform(X_test)
            
            model = LinearRegression()
            model.fit(X_poly, y_train)
            
            y_pred = model.predict(X_test_poly)
            mse = np.mean((y_test - y_pred)**2)
            mse_scores[degree].append(mse)
    
    avg_mse = [np.mean(mse_scores[degree]) for degree in degrees]
    return degrees, avg_mse

degrees, avg_mse = bias_variance_demo(X, y)

plt.plot(degrees, avg_mse)
plt.xlabel('Polynomial Degree')
plt.ylabel('Average Mean Squared Error')
plt.title('Bias-Variance Tradeoff')
plt.show()

Cross-Validation

Cross-validation is a technique used to assess how well a model generalizes to unseen data. It helps in detecting overfitting by evaluating the model's performance on multiple subsets of the data.

from sklearn.model_selection import cross_val_score

def cross_validation_demo(X, y, degrees=range(1, 20), cv=5):
    cv_scores = []
    
    for degree in degrees:
        poly_features = PolynomialFeatures(degree=degree, include_bias=False)
        X_poly = poly_features.fit_transform(X)
        
        model = LinearRegression()
        scores = cross_val_score(model, X_poly, y, cv=cv, scoring='neg_mean_squared_error')
        cv_scores.append(-scores.mean())
    
    plt.plot(degrees, cv_scores)
    plt.xlabel('Polynomial Degree')
    plt.ylabel('Mean Squared Error')
    plt.title(f'{cv}-Fold Cross-Validation')
    plt.show()

cross_validation_demo(X, y)

Regularization: Ridge Regression

Regularization is a technique used to prevent overfitting by adding a penalty term to the loss function. Ridge regression (L2 regularization) adds the squared magnitude of coefficients as a penalty term.

from sklearn.linear_model import Ridge

def ridge_demo(X, y, degrees=[1, 3, 15], alphas=[0, 0.1, 1, 10]):
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
    
    plt.figure(figsize=(15, 5))
    
    for i, degree in enumerate(degrees):
        poly_features = PolynomialFeatures(degree=degree, include_bias=False)
        X_poly = poly_features.fit_transform(X_train)
        X_test_poly = poly_features.transform(X_test)
        
        ax = plt.subplot(1, 3, i + 1)
        
        for alpha in alphas:
            model = Ridge(alpha=alpha)
            model.fit(X_poly, y_train)
            
            X_plot = np.linspace(0, 5, 100).reshape(-1, 1)
            X_plot_poly = poly_features.transform(X_plot)
            y_plot = model.predict(X_plot_poly)
            
            plt.plot(X_plot, y_plot, label=f'α={alpha}')
        
        plt.scatter(X_train, y_train, color='blue', s=10)
        plt.title(f'Ridge Regression (Degree {degree})')
        plt.legend()
    
    plt.tight_layout()
    plt.show()

ridge_demo(X, y)

Regularization: Lasso Regression

Lasso regression (L1 regularization) is another form of regularization that adds the absolute value of coefficients as a penalty term. This can lead to sparse models by forcing some coefficients to zero.

from sklearn.linear_model import Lasso

def lasso_demo(X, y, degrees=[1, 3, 15], alphas=[0, 0.1, 1, 10]):
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
    
    plt.figure(figsize=(15, 5))
    
    for i, degree in enumerate(degrees):
        poly_features = PolynomialFeatures(degree=degree, include_bias=False)
        X_poly = poly_features.fit_transform(X_train)
        X_test_poly = poly_features.transform(X_test)
        
        ax = plt.subplot(1, 3, i + 1)
        
        for alpha in alphas:
            model = Lasso(alpha=alpha)
            model.fit(X_poly, y_train)
            
            X_plot = np.linspace(0, 5, 100).reshape(-1, 1)
            X_plot_poly = poly_features.transform(X_plot)
            y_plot = model.predict(X_plot_poly)
            
            plt.plot(X_plot, y_plot, label=f'α={alpha}')
        
        plt.scatter(X_train, y_train, color='blue', s=10)
        plt.title(f'Lasso Regression (Degree {degree})')
        plt.legend()
    
    plt.tight_layout()
    plt.show()

lasso_demo(X, y)

Early Stopping

Early stopping is a technique used to prevent overfitting by monitoring the model's performance on a validation set during training and stopping when the performance starts to degrade.

from sklearn.neural_network import MLPRegressor
from sklearn.preprocessing import StandardScaler

def early_stopping_demo(X, y):
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
    
    scaler = StandardScaler()
    X_train_scaled = scaler.fit_transform(X_train)
    X_test_scaled = scaler.transform(X_test)
    
    model = MLPRegressor(hidden_layer_sizes=(100,), max_iter=1000, early_stopping=True, validation_fraction=0.2, random_state=42)
    model.fit(X_train_scaled, y_train)
    
    plt.plot(model.loss_curve_, label='Training loss')
    plt.plot(model.validation_scores_, label='Validation score')
    plt.xlabel('Iterations')
    plt.ylabel('Loss / Score')
    plt.title('Early Stopping')
    plt.legend()
    plt.show()

early_stopping_demo(X, y)

Dropout

Dropout is a regularization technique commonly used in neural networks. It randomly "drops out" a proportion of neurons during training, which helps prevent overfitting by reducing complex co-adaptations of neurons.

import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout

def dropout_demo(X, y):
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
    
    model = Sequential([
        Dense(64, activation='relu', input_shape=(1,)),
        Dropout(0.5),
        Dense(64, activation='relu'),
        Dropout(0.5),
        Dense(1)
    ])
    
    model.compile(optimizer='adam', loss='mse')
    history = model.fit(X_train, y_train, epochs=200, validation_split=0.2, verbose=0)
    
    plt.plot(history.history['loss'], label='Training loss')
    plt.plot(history.history['val_loss'], label='Validation loss')
    plt.xlabel('Epoch')
    plt.ylabel('Loss')
    plt.title('Training with Dropout')
    plt.legend()
    plt.show()

dropout_demo(X, y)

Real-life Example: Image Classification

In image classification tasks, overfitting can occur when a model learns specific features of training images rather than generalizing to new images. Let's demonstrate this using a simple CNN for MNIST digit classification.

import tensorflow as tf
from tensorflow.keras.datasets import mnist
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense

# Load and preprocess MNIST data
(X_train, y_train), (X_test, y_test) = mnist.load_data()
X_train = X_train.reshape(-1, 28, 28, 1) / 255.0
X_test = X_test.reshape(-1, 28, 28, 1) / 255.0

# Create a simple CNN model
model = Sequential([
    Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1)),
    MaxPooling2D((2, 2)),
    Conv2D(64, (3, 3), activation='relu'),
    MaxPooling2D((2, 2)),
    Flatten(),
    Dense(64, activation='relu'),
    Dense(10, activation='softmax')
])

model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

# Train the model
history = model.fit(X_train, y_train, epochs=10, validation_split=0.2, verbose=0)

# Plot training and validation accuracy
plt.plot(history.history['accuracy'], label='Training accuracy')
plt.plot(history.history['val_accuracy'], label='Validation accuracy')
plt.xlabel('Epoch')
plt.ylabel('Accuracy')
plt.title('MNIST Classification: Training vs Validation Accuracy')
plt.legend()
plt.show()

Real-life Example: Sentiment Analysis

In natural language processing tasks like sentiment analysis, overfitting can occur when a model memorizes specific phrases or patterns in the training data instead of learning general sentiment indicators. Let's demonstrate this using a simple recurrent neural network for IMDB movie review sentiment analysis.

import tensorflow as tf
from tensorflow.keras.datasets import imdb
from tensorflow.keras.preprocessing.sequence import pad_sequences
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Embedding, LSTM, Dense

# Load and preprocess IMDB data
vocab_size = 10000
max_length = 200
(X_train, y_train), (X_test, y_test) = imdb.load_data(num_words=vocab_size)
X_train = pad_sequences(X_train, maxlen=max_length)
X_test = pad_sequences(X_test, maxlen=max_length)

# Create a simple RNN model
model = Sequential([
    Embedding(vocab_size, 32, input_length=max_length),
    LSTM(64),
    Dense(1, activation='sigmoid')
])

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

# Train the model
history = model.fit(X_train, y_train, epochs=5, validation_split=0.2, verbose=0)

# Plot training and validation accuracy
plt.plot(history.history['accuracy'], label='Training accuracy')
plt.plot(history.history['val_accuracy'], label='Validation accuracy')
plt.xlabel('Epoch')
plt.ylabel('Accuracy')
plt.title('IMDB Sentiment Analysis: Training vs Validation Accuracy')
plt.legend()
plt.show()

Detecting Overfitting

Detecting overfitting is crucial for building robust machine learning models. Here are some common signs of overfitting:

  1. Large gap between training and validation performance
  2. Model performs well on training data but poorly on new data
  3. Complex model with many parameters relative to the amount of training data
  4. Increasing model complexity leads to worse generalization

Let's visualize these signs using our previous polynomial regression example:

def overfitting_detection_demo(X, y, degrees=range(1, 20)):
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
    
    train_scores = []
    test_scores = []
    
    for degree in degrees:
        poly_features = PolynomialFeatures(degree=degree, include_bias=False)
        X_train_poly = poly_features.fit_transform(X_train)
        X_test_poly = poly_features.transform(X_test)
        
        model = LinearRegression()
        model.fit(X_train_poly, y_train)
        
        train_scores.append(model.score(X_train_poly, y_train))
        test_scores.append(model.score(X_test_poly, y_test))
    
    plt.plot(degrees, train_scores, label='Training R² score')
    plt.plot(degrees, test_scores, label='Testing R² score')
    plt.xlabel('Polynomial Degree')
    plt.ylabel('R² Score')
    plt.title('Overfitting Detection: Training vs Testing Performance')
    plt.legend()
    plt.show()

overfitting_detection_demo(X, y)

Preventing Overfitting: Data Augmentation

Data augmentation is a technique used to increase the diversity of your training set by applying various transformations to your existing data. This is particularly useful in image processing tasks. Let's demonstrate a simple image augmentation:

from tensorflow.keras.preprocessing.image import ImageDataGenerator
import matplotlib.pyplot as plt

# Load a sample image (assuming X_train[0] is from MNIST dataset)
image = X_train[0].reshape(28, 28)

# Create an ImageDataGenerator with some augmentation parameters
datagen = ImageDataGenerator(
    rotation_range=20,
    width_shift_range=0.2,
    height_shift_range=0.2,
    shear_range=0.2,
    zoom_range=0.2,
    horizontal_flip=True,
    fill_mode='nearest'
)

# Generate augmented images
aug_iter = datagen.flow(image.reshape((1, 28, 28, 1)), batch_size=1)

# Display original and augmented images
fig, axs = plt.subplots(1, 5, figsize=(15, 3))
axs[0].imshow(image, cmap='gray')
axs[0].set_title('Original')
for i in range(4):
    aug_image = next(aug_iter)[0].reshape(28, 28)
    axs[i+1].imshow(aug_image, cmap='gray')
    axs[i+1].set_title(f'Augmented {i+1}')
plt.tight_layout()
plt.show()

Feature Selection and Dimensionality Reduction

Feature selection and dimensionality reduction techniques can help prevent overfitting by reducing the number of input features. This is particularly useful when dealing with high-dimensional data. Let's demonstrate Principal Component Analysis (PCA) for dimensionality reduction:

from sklearn.decomposition import PCA
from sklearn.datasets import load_digits

# Load the digits dataset
digits = load_digits()
X, y = digits.data, digits.target

# Apply PCA
pca = PCA(n_components=2)
X_pca = pca.fit_transform(X)

# Plot the results
plt.figure(figsize=(10, 8))
scatter = plt.scatter(X_pca[:, 0], X_pca[:, 1], c=y, cmap='viridis')
plt.colorbar(scatter)
plt.xlabel('First Principal Component')
plt.ylabel('Second Principal Component')
plt.title('PCA of Digits Dataset')
plt.show()

# Print the explained variance ratio
print(f"Explained variance ratio: {pca.explained_variance_ratio_}")

Ensemble Methods

Ensemble methods combine predictions from multiple models to create a more robust and generalized prediction. This can help reduce overfitting by averaging out individual model biases. Let's demonstrate a simple ensemble using Random Forest:

from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_squared_error
import numpy as np

# Create and train a Random Forest model
rf_model = RandomForestRegressor(n_estimators=100, random_state=42)
rf_model.fit(X_train, y_train)

# Make predictions
y_pred_rf = rf_model.predict(X_test)

# Calculate MSE
mse_rf = mean_squared_error(y_test, y_pred_rf)

# Plot the results
plt.scatter(X_test, y_test, color='blue', label='Actual')
plt.scatter(X_test, y_pred_rf, color='red', label='Predicted')
plt.xlabel('X')
plt.ylabel('y')
plt.title(f'Random Forest Regression (MSE: {mse_rf:.4f})')
plt.legend()
plt.show()

# Plot feature importances
importances = rf_model.feature_importances_
indices = np.argsort(importances)[::-1]

plt.figure()
plt.title("Feature Importances")
plt.bar(range(X.shape[1]), importances[indices])
plt.xticks(range(X.shape[1]), indices)
plt.xlabel("Feature Index")
plt.ylabel("Importance")
plt.tight_layout()
plt.show()

Additional Resources

For further exploration of overfitting and machine learning concepts, consider these resources:

  1. "Understanding the Bias-Variance Tradeoff" by Scott Fortmann-Roe (http://scott.fortmann-roe.com/docs/BiasVariance.html)
  2. "A Comprehensive Guide to Machine Learning" by Soroush Nasiriany, Garrett Thomas, William Wang, Alex Yang (https://arxiv.org/abs/1906.10742)
  3. "Deep Learning" by Ian Goodfellow, Yoshua Bengio, and Aaron Courville (https://www.deeplearningbook.org/)
  4. "Machine Learning Yearning" by Andrew Ng (https://www.deeplearning.ai/machine-learning-yearning/)

These resources provide in-depth explanations and practical advice for dealing with overfitting and other machine learning challenges.

More in Regularization overfitting and optimization

  1. Balancing Model Complexity! Main Tendencies vs. Overfitting in Python7 min

    In machine learning, finding the right balance between model complexity and generalization is crucial. This balance is often described in terms of bias and...

  2. Balancing Model Fit and Generalization Bias-Variance Tradeoff8 min

    The bias-variance tradeoff is a fundamental concept in machine learning that describes the balance between a model's ability to fit training data and its...

  3. Bayesian Optimization for ML Hyperparameter Tuning8 min

    Bayesian optimization provides a sophisticated approach to hyperparameter tuning by modeling the objective function probabilistically. It constructs a...

  4. Benefits of L1 Regularization7 min

    L1 regularization, also known as Lasso regularization, adds the absolute value of model parameters to the loss function, promoting sparsity by driving some...

  5. Bias-Variance Trade-off in Machine Learning with Python9 min

    The bias-variance trade-off is a fundamental concept in machine learning that helps us understand the balance between model complexity and generalization....

  6. Bias-Variance Tradeoff Clearly Explained18 min

    The bias-variance decomposition mathematically breaks down the prediction error of a machine learning model into its fundamental components. Understanding...