xbe.at

Machine learning / Regularization overfitting and optimization

Balancing Model Complexity! Main Tendencies vs. Overfitting in Python

7 min read, 14 sections

Contents
  1. Understanding Model Bias and Variance
  2. Underfitting: High Bias, Low Variance
  3. Overfitting: Low Bias, High Variance
  4. The Bias-Variance Tradeoff
  5. Cross-Validation: A Tool for Model Selection
  6. Regularization: Controlling Model Complexity
  7. Learning Curves: Diagnosing Bias and Variance
  8. Feature Engineering and Selection
  9. Ensemble Methods: Combining Models
  10. Early Stopping: Preventing Overtraining
  11. Real-Life Example: Predicting House Prices
  12. Real-Life Example: Image Classification
  13. Conclusion and Best Practices
  14. Additional Resources

Understanding Model Bias and Variance

In machine learning, finding the right balance between model complexity and generalization is crucial. This balance is often described in terms of bias and variance. Let's explore these concepts using Python examples.

import numpy as np
import matplotlib.pyplot as plt

# Generate sample data
np.random.seed(0)
X = np.linspace(0, 10, 100)
y = 2 * X + 1 + np.random.randn(100)

plt.scatter(X, y)
plt.title('Sample Data')
plt.xlabel('X')
plt.ylabel('y')
plt.show()

Underfitting: High Bias, Low Variance

Underfitting occurs when a model is too simple to capture the underlying pattern in the data. It has high bias (systematic error) but low variance (sensitivity to changes in the training data).

from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures

# Fit a linear model (underfitting for this data)
model = LinearRegression()
model.fit(X.reshape(-1, 1), y)

plt.scatter(X, y)
plt.plot(X, model.predict(X.reshape(-1, 1)), color='red')
plt.title('Underfitting: Linear Model')
plt.xlabel('X')
plt.ylabel('y')
plt.show()

Overfitting: Low Bias, High Variance

Overfitting happens when a model is too complex and starts to capture noise in the training data. It has low bias but high variance, performing well on training data but poorly on unseen data.

# Fit a high-degree polynomial model (overfitting)
poly_features = PolynomialFeatures(degree=15)
X_poly = poly_features.fit_transform(X.reshape(-1, 1))
model = LinearRegression()
model.fit(X_poly, y)

plt.scatter(X, y)
plt.plot(X, model.predict(X_poly), color='red')
plt.title('Overfitting: High-Degree Polynomial')
plt.xlabel('X')
plt.ylabel('y')
plt.show()

The Bias-Variance Tradeoff

The bias-variance tradeoff is the balance between underfitting and overfitting. As model complexity increases, bias tends to decrease while variance increases. The goal is to find the sweet spot that minimizes both.

import seaborn as sns

# Simulate bias-variance tradeoff
model_complexity = np.linspace(1, 10, 100)
bias = 1 / model_complexity
variance = np.exp(model_complexity) / 100

plt.figure(figsize=(10, 6))
sns.lineplot(x=model_complexity, y=bias, label='Bias')
sns.lineplot(x=model_complexity, y=variance, label='Variance')
sns.lineplot(x=model_complexity, y=bias+variance, label='Total Error')
plt.xlabel('Model Complexity')
plt.ylabel('Error')
plt.title('Bias-Variance Tradeoff')
plt.legend()
plt.show()

Cross-Validation: A Tool for Model Selection

Cross-validation helps us estimate how well a model will generalize to unseen data. It's crucial for finding the right balance between underfitting and overfitting.

from sklearn.model_selection import cross_val_score

def evaluate_model(degree):
    poly_features = PolynomialFeatures(degree=degree)
    X_poly = poly_features.fit_transform(X.reshape(-1, 1))
    model = LinearRegression()
    scores = cross_val_score(model, X_poly, y, cv=5)
    return np.mean(scores)

degrees = range(1, 20)
scores = [evaluate_model(degree) for degree in degrees]

plt.plot(degrees, scores)
plt.xlabel('Polynomial Degree')
plt.ylabel('Cross-Validation Score')
plt.title('Model Performance vs Complexity')
plt.show()

Regularization: Controlling Model Complexity

Regularization techniques like L1 (Lasso) and L2 (Ridge) regularization help prevent overfitting by adding a penalty term to the loss function based on model coefficients.

from sklearn.linear_model import Ridge, Lasso

# Ridge Regression
ridge = Ridge(alpha=1.0)
ridge.fit(X.reshape(-1, 1), y)

# Lasso Regression
lasso = Lasso(alpha=1.0)
lasso.fit(X.reshape(-1, 1), y)

plt.scatter(X, y)
plt.plot(X, ridge.predict(X.reshape(-1, 1)), label='Ridge', color='red')
plt.plot(X, lasso.predict(X.reshape(-1, 1)), label='Lasso', color='green')
plt.legend()
plt.title('Ridge vs Lasso Regression')
plt.show()

Learning Curves: Diagnosing Bias and Variance

Learning curves show how model performance changes with increasing amounts of training data. They can help diagnose whether a model is suffering from high bias or high variance.

from sklearn.model_selection import learning_curve

def plot_learning_curve(estimator, X, y):
    train_sizes, train_scores, test_scores = learning_curve(
        estimator, X, y, cv=5, n_jobs=-1, train_sizes=np.linspace(0.1, 1.0, 10))
    
    train_scores_mean = np.mean(train_scores, axis=1)
    test_scores_mean = np.mean(test_scores, axis=1)

    plt.plot(train_sizes, train_scores_mean, label='Training score')
    plt.plot(train_sizes, test_scores_mean, label='Cross-validation score')
    plt.xlabel('Training examples')
    plt.ylabel('Score')
    plt.legend()
    plt.title('Learning Curve')
    plt.show()

plot_learning_curve(LinearRegression(), X.reshape(-1, 1), y)

Feature Engineering and Selection

Proper feature engineering and selection can help reduce model complexity and prevent overfitting. Let's look at an example using the Boston Housing dataset.

from sklearn.datasets import load_boston
from sklearn.feature_selection import SelectKBest, f_regression

boston = load_boston()
X, y = boston.data, boston.target

# Select top 5 features
selector = SelectKBest(f_regression, k=5)
X_selected = selector.fit_transform(X, y)

print("Original features:", boston.feature_names)
print("Selected features:", [boston.feature_names[i] for i in selector.get_support(indices=True)])

Ensemble Methods: Combining Models

Ensemble methods like Random Forests and Gradient Boosting can help reduce overfitting by combining multiple models.

from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

rf_model = RandomForestRegressor(n_estimators=100, random_state=42)
rf_model.fit(X_train.reshape(-1, 1), y_train)

print("Random Forest R² score:", rf_model.score(X_test.reshape(-1, 1), y_test))

Early Stopping: Preventing Overtraining

Early stopping is a technique used in iterative methods like neural networks to prevent overfitting by stopping training when performance on a validation set starts to degrade.

from sklearn.neural_network import MLPRegressor
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

mlp = MLPRegressor(hidden_layer_sizes=(100, 100), max_iter=1000, early_stopping=True,
                   validation_fraction=0.2, random_state=42)
mlp.fit(X_train.reshape(-1, 1), y_train)

print("Number of iterations:", mlp.n_iter_)
print("Best validation score:", mlp.best_validation_score_)

Real-Life Example: Predicting House Prices

Let's apply what we've learned to a real-world problem: predicting house prices based on various features.

from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LinearRegression, Ridge
from sklearn.metrics import mean_squared_error

# Load and prepare the data
housing = fetch_california_housing()
X, y = housing.data, housing.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Train and evaluate models
lr = LinearRegression().fit(X_train_scaled, y_train)
ridge = Ridge(alpha=1.0).fit(X_train_scaled, y_train)

print("Linear Regression MSE:", mean_squared_error(y_test, lr.predict(X_test_scaled)))
print("Ridge Regression MSE:", mean_squared_error(y_test, ridge.predict(X_test_scaled)))

Real-Life Example: Image Classification

In image classification, overfitting can be a significant problem due to the high dimensionality of the data. Let's look at how data augmentation can help prevent overfitting in a simple CNN model.

import tensorflow as tf
from tensorflow.keras.datasets import cifar10
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense
from tensorflow.keras.preprocessing.image import ImageDataGenerator

# Load and prepare the data
(X_train, y_train), (X_test, y_test) = cifar10.load_data()
X_train, X_test = X_train / 255.0, X_test / 255.0

# Create a simple CNN model
model = Sequential([
    Conv2D(32, (3, 3), activation='relu', input_shape=(32, 32, 3)),
    MaxPooling2D((2, 2)),
    Conv2D(64, (3, 3), activation='relu'),
    MaxPooling2D((2, 2)),
    Flatten(),
    Dense(64, activation='relu'),
    Dense(10, activation='softmax')
])

model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

# Data augmentation
datagen = ImageDataGenerator(
    rotation_range=15,
    width_shift_range=0.1,
    height_shift_range=0.1,
    horizontal_flip=True
)

# Train the model with data augmentation
history = model.fit(datagen.flow(X_train, y_train, batch_size=32),
                    epochs=10, validation_data=(X_test, y_test))

# Plot training history
plt.plot(history.history['accuracy'], label='Training Accuracy')
plt.plot(history.history['val_accuracy'], label='Validation Accuracy')
plt.xlabel('Epoch')
plt.ylabel('Accuracy')
plt.legend()
plt.title('Model Accuracy with Data Augmentation')
plt.show()

Conclusion and Best Practices

To find the right balance between underfitting and overfitting:

  1. Start with simple models and gradually increase complexity.
  2. Use cross-validation to estimate model performance.
  3. Apply regularization techniques to control model complexity.
  4. Monitor training and validation performance using learning curves.
  5. Use feature selection and engineering to focus on relevant information.
  6. Consider ensemble methods to combine multiple models.
  7. Apply early stopping in iterative learning algorithms.
  8. Use data augmentation for high-dimensional data like images.

Remember, the goal is to create models that generalize well to unseen data, not just perform well on the training set.

Additional Resources

For more information on model selection and avoiding overfitting, consider the following resources:

  1. "A Few Useful Things to Know about Machine Learning" by Pedro Domingos ArXiv link: https://arxiv.org/abs/1206.2944
  2. "Understanding the Bias-Variance Tradeoff" by Scott Fortmann-Roe Note: This is not an ArXiv source, but it's a widely referenced article on the topic.
  3. "Regularization and variable selection via the elastic net" by Hui Zou and Trevor Hastie ArXiv link: https://arxiv.org/abs/math/0406049

These resources provide deeper insights into the concepts we've covered and can help you further develop your understanding of model selection and generalization in machine learning.

More in Regularization overfitting and optimization

  1. Bayesian Optimization for ML Hyperparameter Tuning8 min

    Bayesian optimization provides a sophisticated approach to hyperparameter tuning by modeling the objective function probabilistically. It constructs a...

  2. Avoiding Overfitting in Machine Learning with Python10 min

    Overfitting occurs when a machine learning model learns the training data too well, capturing noise and specific data points rather than general patterns....

  3. Benefits of L1 Regularization7 min

    L1 regularization, also known as Lasso regularization, adds the absolute value of model parameters to the loss function, promoting sparsity by driving some...

  4. Addressing High Bias in Machine Learning Models10 min

    High bias in machine learning manifests as underfitting, where the model makes oversimplified assumptions about the data relationships. This results in poor...

  5. Bias-Variance Trade-off in Machine Learning with Python9 min

    The bias-variance trade-off is a fundamental concept in machine learning that helps us understand the balance between model complexity and generalization....

  6. Bias-Variance Tradeoff Clearly Explained18 min

    The bias-variance decomposition mathematically breaks down the prediction error of a machine learning model into its fundamental components. Understanding...