xbe.at

Machine learning / Regularization overfitting and optimization

Avoiding Overfitting Selecting the Best Machine Learning Model

9 min read, 15 sections

Contents
  1. Understanding Model Selection and Overfitting
  2. Linear Regression Model
  3. Polynomial Regression Models
  4. Visualizing Model Performance
  5. Understanding Overfitting
  6. Interpreting R² and MSE
  7. Bias-Variance Tradeoff
  8. Real-life Example: Weather Prediction
  9. Real-life Example: Image Classification
  10. Cross-validation: A Tool to Detect Overfitting
  11. Regularization: Combating Overfitting
  12. Model Complexity vs. Performance
  13. Choosing the Best Model
  14. Conclusion and Best Practices
  15. Additional Resources

Understanding Model Selection and Overfitting

In machine learning, selecting the right model is crucial for accurate predictions. This slideshow will explore the concept of overfitting and how to choose the best model for your data.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures
from sklearn.metrics import r2_score, mean_squared_error

# Generate sample data
np.random.seed(0)
X = np.linspace(0, 10, 100).reshape(-1, 1)
y = 2 * X + 1 + np.random.normal(0, 1, (100, 1))

plt.scatter(X, y)
plt.title("Sample Data")
plt.xlabel("X")
plt.ylabel("y")
plt.show()

Linear Regression Model

Let's start with a simple linear regression model to fit our data.

# Fit linear regression model
linear_model = LinearRegression()
linear_model.fit(X, y)

# Calculate R² and MSE
y_pred_linear = linear_model.predict(X)
r2_linear = r2_score(y, y_pred_linear)
mse_linear = mean_squared_error(y, y_pred_linear)

print(f"Linear Model - R²: {r2_linear:.4f}, MSE: {mse_linear:.4f}")

plt.scatter(X, y)
plt.plot(X, y_pred_linear, color='red', label='Linear Model')
plt.title("Linear Regression")
plt.xlabel("X")
plt.ylabel("y")
plt.legend()
plt.show()

Polynomial Regression Models

Now, let's fit polynomial regression models of different degrees to see how they perform.

def fit_polynomial(degree):
    poly_features = PolynomialFeatures(degree=degree)
    X_poly = poly_features.fit_transform(X)
    poly_model = LinearRegression()
    poly_model.fit(X_poly, y)
    y_pred = poly_model.predict(X_poly)
    r2 = r2_score(y, y_pred)
    mse = mean_squared_error(y, y_pred)
    return poly_model, r2, mse, y_pred

# Fit polynomial models of degree 3, 5, and 9
models = []
for degree in [3, 5, 9]:
    model, r2, mse, y_pred = fit_polynomial(degree)
    models.append((degree, model, r2, mse, y_pred))
    print(f"Polynomial Degree {degree} - R²: {r2:.4f}, MSE: {mse:.4f}")

Visualizing Model Performance

Let's visualize how these models fit our data.

plt.figure(figsize=(12, 8))
plt.scatter(X, y, label='Data')
plt.plot(X, y_pred_linear, color='red', label='Linear Model')

colors = ['green', 'blue', 'purple']
for i, (degree, _, _, _, y_pred) in enumerate(models):
    plt.plot(X, y_pred, color=colors[i], label=f'Polynomial Degree {degree}')

plt.title("Model Comparison")
plt.xlabel("X")
plt.ylabel("y")
plt.legend()
plt.show()

Understanding Overfitting

Overfitting occurs when a model learns the noise in the training data too well, capturing random fluctuations rather than the underlying pattern. This results in poor generalization to new, unseen data.

# Generate new data for testing
X_test = np.linspace(-1, 11, 100).reshape(-1, 1)
y_test = 2 * X_test + 1 + np.random.normal(0, 1, (100, 1))

# Predict using all models
y_pred_linear_test = linear_model.predict(X_test)
y_pred_poly_test = [fit_polynomial(degree)[0].predict(PolynomialFeatures(degree=degree).fit_transform(X_test)) 
                    for degree, _, _, _, _ in models]

# Plot results
plt.figure(figsize=(12, 8))
plt.scatter(X, y, label='Training Data', alpha=0.5)
plt.scatter(X_test, y_test, label='Test Data', alpha=0.5)
plt.plot(X_test, y_pred_linear_test, color='red', label='Linear Model')

for i, (degree, _, _, _, _) in enumerate(models):
    plt.plot(X_test, y_pred_poly_test[i], color=colors[i], label=f'Polynomial Degree {degree}')

plt.title("Model Performance on Test Data")
plt.xlabel("X")
plt.ylabel("y")
plt.legend()
plt.show()

Interpreting R² and MSE

R² (coefficient of determination) measures the proportion of variance in the dependent variable explained by the independent variables. MSE (Mean Squared Error) measures the average squared difference between predicted and actual values.

# Calculate R² and MSE for test data
r2_linear_test = r2_score(y_test, y_pred_linear_test)
mse_linear_test = mean_squared_error(y_test, y_pred_linear_test)

print(f"Linear Model Test - R²: {r2_linear_test:.4f}, MSE: {mse_linear_test:.4f}")

for i, (degree, _, _, _, _) in enumerate(models):
    r2_poly_test = r2_score(y_test, y_pred_poly_test[i])
    mse_poly_test = mean_squared_error(y_test, y_pred_poly_test[i])
    print(f"Polynomial Degree {degree} Test - R²: {r2_poly_test:.4f}, MSE: {mse_poly_test:.4f}")

Bias-Variance Tradeoff

The bias-variance tradeoff is a fundamental concept in machine learning. Bias refers to the error introduced by approximating a real-world problem with a simplified model. Variance refers to the model's sensitivity to small fluctuations in the training data.

def bias_variance_decomposition(y_true, y_pred):
    bias = np.mean((y_true - np.mean(y_pred, axis=0))**2)
    variance = np.mean(np.var(y_pred, axis=0))
    return bias, variance

# Generate multiple datasets
n_datasets = 100
y_datasets = [2 * X + 1 + np.random.normal(0, 1, (100, 1)) for _ in range(n_datasets)]

# Fit models to each dataset
linear_preds = []
poly_preds = [[] for _ in range(3)]

for y_data in y_datasets:
    linear_model = LinearRegression().fit(X, y_data)
    linear_preds.append(linear_model.predict(X))
    
    for i, degree in enumerate([3, 5, 9]):
        poly_model, _, _, y_pred = fit_polynomial(degree)
        poly_preds[i].append(y_pred)

# Calculate bias and variance
bias_linear, var_linear = bias_variance_decomposition(y, np.array(linear_preds))
bias_poly = []
var_poly = []

for poly_pred in poly_preds:
    bias, var = bias_variance_decomposition(y, np.array(poly_pred))
    bias_poly.append(bias)
    var_poly.append(var)

print(f"Linear Model - Bias: {bias_linear:.4f}, Variance: {var_linear:.4f}")
for i, degree in enumerate([3, 5, 9]):
    print(f"Polynomial Degree {degree} - Bias: {bias_poly[i]:.4f}, Variance: {var_poly[i]:.4f}")

Real-life Example: Weather Prediction

Consider a weather prediction model. We'll simulate temperature data and fit different models to predict future temperatures.

import pandas as pd
from sklearn.model_selection import train_test_split

# Generate synthetic weather data
np.random.seed(0)
dates = pd.date_range(start='2022-01-01', end='2023-12-31', freq='D')
temperatures = 20 + 10 * np.sin(2 * np.pi * np.arange(len(dates)) / 365) + np.random.normal(0, 3, len(dates))

df = pd.DataFrame({'date': dates, 'temperature': temperatures})
df['day_of_year'] = df['date'].dt.dayofyear

X = df['day_of_year'].values.reshape(-1, 1)
y = df['temperature'].values

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Fit models
linear_model = LinearRegression().fit(X_train, y_train)
poly_models = [fit_polynomial(degree)[0] for degree in [3, 5, 9]]

# Plot results
plt.figure(figsize=(12, 8))
plt.scatter(X_test, y_test, label='Actual', alpha=0.5)
plt.plot(X_test, linear_model.predict(X_test), color='red', label='Linear Model')

for i, (degree, color) in enumerate(zip([3, 5, 9], colors)):
    plt.plot(X_test, poly_models[i].predict(PolynomialFeatures(degree=degree).fit_transform(X_test)), 
             color=color, label=f'Polynomial Degree {degree}')

plt.title("Temperature Prediction Models")
plt.xlabel("Day of Year")
plt.ylabel("Temperature (°C)")
plt.legend()
plt.show()

Real-life Example: Image Classification

In image classification, overfitting can occur when a model learns to recognize specific training images rather than general features. Let's simulate this with a simple example.

from sklearn.datasets import load_digits
from sklearn.model_selection import learning_curve
from sklearn.svm import SVC

# Load digits dataset
digits = load_digits()
X, y = digits.data, digits.target

# Define models
linear_svm = SVC(kernel='linear', random_state=42)
rbf_svm = SVC(kernel='rbf', random_state=42)

# Calculate learning curves
train_sizes, train_scores_linear, test_scores_linear = learning_curve(
    linear_svm, X, y, cv=5, n_jobs=-1, train_sizes=np.linspace(0.1, 1.0, 10))

train_sizes, train_scores_rbf, test_scores_rbf = learning_curve(
    rbf_svm, X, y, cv=5, n_jobs=-1, train_sizes=np.linspace(0.1, 1.0, 10))

# Plot learning curves
plt.figure(figsize=(12, 6))
plt.plot(train_sizes, np.mean(train_scores_linear, axis=1), 'o-', color='r', label='Linear SVM (Train)')
plt.plot(train_sizes, np.mean(test_scores_linear, axis=1), 'o-', color='g', label='Linear SVM (Test)')
plt.plot(train_sizes, np.mean(train_scores_rbf, axis=1), 'o-', color='b', label='RBF SVM (Train)')
plt.plot(train_sizes, np.mean(test_scores_rbf, axis=1), 'o-', color='y', label='RBF SVM (Test)')

plt.xlabel("Training Examples")
plt.ylabel("Score")
plt.title("Learning Curves for SVM Models")
plt.legend(loc="best")
plt.show()

Cross-validation: A Tool to Detect Overfitting

Cross-validation helps us assess how well our model generalizes to unseen data by training and testing on different subsets of the data.

from sklearn.model_selection import cross_val_score

# Perform cross-validation
cv_scores_linear = cross_val_score(linear_model, X, y, cv=5)
cv_scores_poly = [cross_val_score(fit_polynomial(degree)[0], X, y, cv=5) for degree in [3, 5, 9]]

print("Cross-validation scores:")
print(f"Linear Model: {cv_scores_linear.mean():.4f} (+/- {cv_scores_linear.std() * 2:.4f})")
for i, degree in enumerate([3, 5, 9]):
    print(f"Polynomial Degree {degree}: {cv_scores_poly[i].mean():.4f} (+/- {cv_scores_poly[i].std() * 2:.4f})")

# Visualize cross-validation results
plt.figure(figsize=(10, 6))
plt.boxplot([cv_scores_linear] + cv_scores_poly, labels=['Linear'] + [f'Poly {d}' for d in [3, 5, 9]])
plt.title("Cross-validation Scores for Different Models")
plt.ylabel("R² Score")
plt.show()

Regularization: Combating Overfitting

Regularization techniques help prevent overfitting by adding a penalty term to the loss function, discouraging complex models.

from sklearn.linear_model import Ridge, Lasso

# Fit regularized models
ridge_model = Ridge(alpha=1.0).fit(X, y)
lasso_model = Lasso(alpha=1.0).fit(X, y)

# Predict using regularized models
y_pred_ridge = ridge_model.predict(X)
y_pred_lasso = lasso_model.predict(X)

# Calculate R² scores
r2_ridge = r2_score(y, y_pred_ridge)
r2_lasso = r2_score(y, y_pred_lasso)

print(f"Ridge Regression R²: {r2_ridge:.4f}")
print(f"Lasso Regression R²: {r2_lasso:.4f}")

# Plot results
plt.figure(figsize=(12, 8))
plt.scatter(X, y, label='Data')
plt.plot(X, y_pred_linear, color='red', label='Linear Model')
plt.plot(X, y_pred_ridge, color='green', label='Ridge Regression')
plt.plot(X, y_pred_lasso, color='blue', label='Lasso Regression')
plt.title("Regularized Models Comparison")
plt.xlabel("X")
plt.ylabel("y")
plt.legend()
plt.show()

Model Complexity vs. Performance

As model complexity increases, training error typically decreases, but test error may start to increase due to overfitting.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score

# Generate synthetic data
np.random.seed(0)
X = np.sort(np.random.rand(100, 1), axis=0)
y = np.sin(2 * np.pi * X) + np.random.normal(0, 0.1, (100, 1))

# Split data into training and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Train models with varying complexity
degrees = range(1, 15)
train_scores = []
test_scores = []

for degree in degrees:
    poly_features = PolynomialFeatures(degree=degree)
    X_poly_train = poly_features.fit_transform(X_train)
    X_poly_test = poly_features.transform(X_test)
    
    model = LinearRegression().fit(X_poly_train, y_train)
    train_scores.append(r2_score(y_train, model.predict(X_poly_train)))
    test_scores.append(r2_score(y_test, model.predict(X_poly_test)))

# Plot results
plt.figure(figsize=(10, 6))
plt.plot(degrees, train_scores, 'bo-', label='Training Score')
plt.plot(degrees, test_scores, 'ro-', label='Test Score')
plt.xlabel('Polynomial Degree')
plt.ylabel('R² Score')
plt.title('Model Complexity vs. Performance')
plt.legend()
plt.show()

Choosing the Best Model

When selecting the best model, we need to balance complexity and performance. The ideal model captures the underlying pattern without overfitting to noise.

# Find the best performing model on test data
best_degree = degrees[np.argmax(test_scores)]
best_test_score = max(test_scores)

print(f"Best performing model: Polynomial degree {best_degree}")
print(f"Best test R² score: {best_test_score:.4f}")

# Visualize the best model
best_poly_features = PolynomialFeatures(degree=best_degree)
X_poly_train = best_poly_features.fit_transform(X_train)
X_poly_test = best_poly_features.transform(X_test)

best_model = LinearRegression().fit(X_poly_train, y_train)

plt.figure(figsize=(10, 6))
plt.scatter(X_train, y_train, color='blue', label='Training data')
plt.scatter(X_test, y_test, color='red', label='Test data')

X_plot = np.linspace(0, 1, 100).reshape(-1, 1)
X_plot_poly = best_poly_features.transform(X_plot)
y_plot = best_model.predict(X_plot_poly)

plt.plot(X_plot, y_plot, color='green', label=f'Best model (degree {best_degree})')
plt.xlabel('X')
plt.ylabel('y')
plt.title('Best Model Fit')
plt.legend()
plt.show()

Conclusion and Best Practices

To avoid overfitting and choose the best model:

  1. Use cross-validation to assess model performance.
  2. Monitor both training and validation errors.
  3. Apply regularization techniques when appropriate.
  4. Consider the principle of Occam's razor: prefer simpler models when performance is similar.
  5. Collect more data if possible to improve model generalization.

Remember, the goal is to find a model that generalizes well to unseen data, not just performs well on the training set.

Additional Resources

For further reading on model selection and overfitting:

  1. "A Unified Approach to Model Selection and Sparse Recovery Using Regularized Least Squares" by Michał Derezinski et al. (2019) ArXiv: https://arxiv.org/abs/1905.10377
  2. "On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation" by Gavin C. Cawley and Nicola L. C. Talbot (2010) ArXiv: https://arxiv.org/abs/1006.3188
  3. "Understanding Machine Learning: From Theory to Algorithms" by Shai Shalev-Shwartz and Shai Ben-David (2014) Cambridge University Press

These resources provide in-depth discussions on model selection techniques, overfitting, and related theoretical aspects.

More in Regularization overfitting and optimization

  1. Balancing Model Fit and Generalization Bias-Variance Tradeoff8 min

    The bias-variance tradeoff is a fundamental concept in machine learning that describes the balance between a model's ability to fit training data and its...

  2. Addressing High Bias in Machine Learning Models10 min

    High bias in machine learning manifests as underfitting, where the model makes oversimplified assumptions about the data relationships. This results in poor...

  3. Bayesian Optimization for ML Hyperparameter Tuning8 min

    Bayesian optimization provides a sophisticated approach to hyperparameter tuning by modeling the objective function probabilistically. It constructs a...

  4. Benefits of L1 Regularization7 min

    L1 regularization, also known as Lasso regularization, adds the absolute value of model parameters to the loss function, promoting sparsity by driving some...

  5. Bias-Variance Trade-off in Machine Learning with Python9 min

    The bias-variance trade-off is a fundamental concept in machine learning that helps us understand the balance between model complexity and generalization....

  6. Bias-Variance Tradeoff Clearly Explained18 min

    The bias-variance decomposition mathematically breaks down the prediction error of a machine learning model into its fundamental components. Understanding...