Demystifying Overfitting, Underfitting, Bias, and Variance in Python

Introduction to Overfitting and Underfitting

Overfitting and underfitting are common challenges in machine learning that affect model performance. Overfitting occurs when a model learns the training data too well, including noise, while underfitting happens when a model is too simple to capture the underlying patterns in the data.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

# Generate sample data
np.random.seed(0)
X = np.sort(np.random.rand(100, 1), axis=0)
y = np.sin(2 * np.pi * X).ravel() + np.random.normal(0, 0.1, X.shape[0])

# Create models with different polynomial degrees
degrees = [1, 3, 15]
plt.figure(figsize=(14, 4))

for i, degree in enumerate(degrees):
    ax = plt.subplot(1, 3, i + 1)
    poly_features = PolynomialFeatures(degree=degree, include_bias=False)
    X_poly = poly_features.fit_transform(X)
    
    model = LinearRegression()
    model.fit(X_poly, y)
    
    X_plot = np.linspace(0, 1, 100)[:, np.newaxis]
    X_plot_poly = poly_features.transform(X_plot)
    y_plot = model.predict(X_plot_poly)
    
    plt.scatter(X, y, color='r', s=20, alpha=0.5)
    plt.plot(X_plot, y_plot, color='b')
    plt.title(f'Polynomial Degree {degree}')
    plt.ylim(-1.5, 1.5)

plt.tight_layout()
plt.show()

Bias-Variance Tradeoff

The bias-variance tradeoff is a fundamental concept in machine learning that helps us understand the balance between model complexity and generalization. Bias refers to the error introduced by approximating a real-world problem with a simplified model, while variance is the model's sensitivity to small fluctuations in the training data.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

# Generate sample data
np.random.seed(0)
X = np.sort(np.random.rand(200, 1), axis=0)
y = np.sin(2 * np.pi * X).ravel() + np.random.normal(0, 0.1, X.shape[0])

# Split data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Initialize lists to store errors
train_errors, test_errors = [], []
degrees = range(1, 20)

for degree in degrees:
    poly_features = PolynomialFeatures(degree=degree, include_bias=False)
    X_poly_train = poly_features.fit_transform(X_train)
    X_poly_test = poly_features.transform(X_test)
    
    model = LinearRegression()
    model.fit(X_poly_train, y_train)
    
    train_pred = model.predict(X_poly_train)
    test_pred = model.predict(X_poly_test)
    
    train_errors.append(mean_squared_error(y_train, train_pred))
    test_errors.append(mean_squared_error(y_test, test_pred))

# Plot the results
plt.figure(figsize=(10, 6))
plt.plot(degrees, train_errors, label='Training Error')
plt.plot(degrees, test_errors, label='Testing Error')
plt.xlabel('Polynomial Degree')
plt.ylabel('Mean Squared Error')
plt.title('Bias-Variance Tradeoff')
plt.legend()
plt.show()

Detecting Overfitting

Overfitting can be detected by comparing the model's performance on training and validation datasets. A significant gap between training and validation errors indicates overfitting. We'll demonstrate this using a simple example with polynomial regression.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

# Generate sample data
np.random.seed(0)
X = np.sort(np.random.rand(100, 1), axis=0)
y = np.sin(2 * np.pi * X).ravel() + np.random.normal(0, 0.1, X.shape[0])

# Split data into training and validation sets
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.3, random_state=42)

# Create and train an overfitted model
poly_features = PolynomialFeatures(degree=15, include_bias=False)
X_poly_train = poly_features.fit_transform(X_train)
X_poly_val = poly_features.transform(X_val)

model = LinearRegression()
model.fit(X_poly_train, y_train)

# Calculate training and validation errors
train_pred = model.predict(X_poly_train)
val_pred = model.predict(X_poly_val)

train_mse = mean_squared_error(y_train, train_pred)
val_mse = mean_squared_error(y_val, val_pred)

print(f"Training MSE: {train_mse:.4f}")
print(f"Validation MSE: {val_mse:.4f}")

# Plot the results
plt.figure(figsize=(10, 6))
plt.scatter(X_train, y_train, color='b', label='Training data')
plt.scatter(X_val, y_val, color='r', label='Validation data')
plt.plot(X, model.predict(poly_features.transform(X)), color='g', label='Model prediction')
plt.title('Overfitted Model')
plt.legend()
plt.show()

Addressing Underfitting

Underfitting occurs when a model is too simple to capture the underlying patterns in the data. To address underfitting, we can increase model complexity, add more relevant features, or reduce regularization. Let's demonstrate this using a linear regression model on non-linear data.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression

# Generate non-linear data
np.random.seed(0)
X = np.sort(np.random.rand(100, 1), axis=0)
y = np.sin(2 * np.pi * X).ravel() + np.random.normal(0, 0.1, X.shape[0])

# Create and fit underfitted model (linear)
underfitted_model = LinearRegression()
underfitted_model.fit(X, y)

# Create and fit proper model (polynomial)
poly_features = PolynomialFeatures(degree=3, include_bias=False)
X_poly = poly_features.fit_transform(X)
proper_model = LinearRegression()
proper_model.fit(X_poly, y)

# Plot the results
plt.figure(figsize=(12, 5))

plt.subplot(1, 2, 1)
plt.scatter(X, y, color='b', label='Data')
plt.plot(X, underfitted_model.predict(X), color='r', label='Underfitted model')
plt.title('Underfitted Model (Linear)')
plt.legend()

plt.subplot(1, 2, 2)
plt.scatter(X, y, color='b', label='Data')
plt.plot(X, proper_model.predict(X_poly), color='g', label='Proper model')
plt.title('Proper Model (Polynomial)')
plt.legend()

plt.tight_layout()
plt.show()

Regularization Techniques

Regularization is a technique used to prevent overfitting by adding a penalty term to the loss function. Common regularization methods include L1 (Lasso), L2 (Ridge), and Elastic Net. Let's demonstrate the effect of L2 regularization on a polynomial regression model.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import Ridge
from sklearn.pipeline import make_pipeline

# Generate sample data
np.random.seed(0)
X = np.sort(np.random.rand(20, 1), axis=0)
y = np.sin(2 * np.pi * X).ravel() + np.random.normal(0, 0.1, X.shape[0])

# Create models with different regularization strengths
alphas = [0, 0.1, 1, 10]
degrees = [15] * len(alphas)

plt.figure(figsize=(14, 4))
for i, (alpha, degree) in enumerate(zip(alphas, degrees)):
    plt.subplot(1, len(alphas), i + 1)
    model = make_pipeline(PolynomialFeatures(degree), Ridge(alpha=alpha))
    model.fit(X, y)
    
    X_plot = np.linspace(0, 1, 100)[:, np.newaxis]
    y_plot = model.predict(X_plot)
    
    plt.scatter(X, y, color='r', s=20, alpha=0.5)
    plt.plot(X_plot, y_plot, color='b')
    plt.ylim(-1.5, 1.5)
    plt.title(f'Alpha = {alpha}')

plt.tight_layout()
plt.show()

Cross-Validation for Model Selection

Cross-validation is a technique used to assess model performance and select the best hyperparameters. It helps prevent overfitting by evaluating the model on multiple subsets of the data. Let's use k-fold cross-validation to select the best polynomial degree for our regression model.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline

# Generate sample data
np.random.seed(0)
X = np.sort(np.random.rand(100, 1), axis=0)
y = np.sin(2 * np.pi * X).ravel() + np.random.normal(0, 0.1, X.shape[0])

# Perform cross-validation for different polynomial degrees
degrees = range(1, 20)
scores = []

for degree in degrees:
    model = make_pipeline(PolynomialFeatures(degree), LinearRegression())
    cv_scores = cross_val_score(model, X, y, cv=5, scoring='neg_mean_squared_error')
    scores.append(-cv_scores.mean())

# Plot the results
plt.figure(figsize=(10, 6))
plt.plot(degrees, scores, marker='o')
plt.xlabel('Polynomial Degree')
plt.ylabel('Mean Squared Error')
plt.title('Cross-Validation Scores for Polynomial Regression')
plt.show()

best_degree = degrees[np.argmin(scores)]
print(f"Best polynomial degree: {best_degree}")

Feature Selection to Reduce Overfitting

Feature selection is the process of choosing the most relevant features for your model. It can help reduce overfitting by eliminating irrelevant or redundant features. Let's demonstrate this using the Lasso regression, which performs built-in feature selection.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import Lasso
from sklearn.datasets import make_regression

# Generate sample data with some irrelevant features
X, y = make_regression(n_samples=100, n_features=20, n_informative=5, noise=0.1, random_state=42)

# Standardize features
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# Fit Lasso model
lasso = Lasso(alpha=0.1)
lasso.fit(X_scaled, y)

# Plot feature importances
plt.figure(figsize=(12, 6))
plt.bar(range(X.shape[1]), np.abs(lasso.coef_))
plt.xlabel('Feature Index')
plt.ylabel('Absolute Coefficient Value')
plt.title('Feature Importance using Lasso Regression')
plt.show()

# Print number of selected features
selected_features = np.sum(lasso.coef_ != 0)
print(f"Number of selected features: {selected_features}")

Learning Curves

Learning curves help visualize how model performance changes with increasing training data size. They can indicate whether a model is overfitting, underfitting, or performing well. Let's plot learning curves for a polynomial regression model.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import learning_curve
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline

# Generate sample data
np.random.seed(0)
X = np.sort(np.random.rand(100, 1), axis=0)
y = np.sin(2 * np.pi * X).ravel() + np.random.normal(0, 0.1, X.shape[0])

# Create and fit the model
model = make_pipeline(PolynomialFeatures(degree=3), LinearRegression())

# Generate learning curve data
train_sizes, train_scores, val_scores = learning_curve(
    model, X, y, train_sizes=np.linspace(0.1, 1.0, 10), cv=5, scoring='neg_mean_squared_error'
)

# Calculate mean and standard deviation of scores
train_scores_mean = -np.mean(train_scores, axis=1)
train_scores_std = np.std(train_scores, axis=1)
val_scores_mean = -np.mean(val_scores, axis=1)
val_scores_std = np.std(val_scores, axis=1)

# Plot learning curves
plt.figure(figsize=(10, 6))
plt.fill_between(train_sizes, train_scores_mean - train_scores_std, 
                 train_scores_mean + train_scores_std, alpha=0.1, color='r')
plt.fill_between(train_sizes, val_scores_mean - val_scores_std,
                 val_scores_mean + val_scores_std, alpha=0.1, color='g')
plt.plot(train_sizes, train_scores_mean, 'o-', color='r', label='Training score')
plt.plot(train_sizes, val_scores_mean, 'o-', color='g', label='Cross-validation score')
plt.xlabel('Training Examples')
plt.ylabel('Mean Squared Error')
plt.title('Learning Curves')
plt.legend(loc='best')
plt.show()

Ensemble Methods to Reduce Overfitting

Ensemble methods combine multiple models to create a more robust and accurate predictor. They can help reduce overfitting by averaging out individual model errors. Let's demonstrate this using Random Forest, an ensemble of decision trees.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_squared_error

# Generate sample data
np.random.seed(0)
X = np.sort(np.random.rand(100, 1), axis=0)
y = np.sin(2 * np.pi * X).ravel() + np.random.normal(0, 0.1, X.shape[0])

# Split data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Train Random Forest model
rf_model = RandomForestRegressor(n_estimators=100, random_state=42)
rf_model.fit(X_train, y_train)

# Make predictions
y_pred = rf_model.predict(X_test)

# Calculate MSE
mse = mean_squared_error(y_test, y_pred)
print(f"Mean Squared Error: {mse:.4f}")

# Plot results
plt.figure(figsize=(10, 6))
plt.scatter(X_train, y_train, color='b', label='Training data')
plt.scatter(X_test, y_test, color='r', label='Testing data')
plt.plot(X, rf_model.predict(X), color='g', label='Random Forest prediction')
plt.xlabel('X')
plt.ylabel('y')
plt.title('Random Forest Regression')
plt.legend()
plt.show()

Early Stopping to Prevent Overfitting

Early stopping is a technique used to prevent overfitting by monitoring the model's performance on a validation set during training and stopping when the performance starts to degrade. This approach is particularly useful for iterative algorithms like neural networks.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.neural_network import MLPRegressor

# Generate sample data
np.random.seed(0)
X = np.sort(np.random.rand(200, 1), axis=0)
y = np.sin(2 * np.pi * X).ravel() + np.random.normal(0, 0.1, X.shape[0])

# Split data and scale features
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Train model with early stopping
model = MLPRegressor(hidden_layer_sizes=(100, 100), max_iter=1000, 
                     early_stopping=True, validation_fraction=0.2, random_state=42)
model.fit(X_train_scaled, y_train)

# Plot learning curve
plt.figure(figsize=(10, 6))
plt.plot(model.loss_curve_, label='Training loss')
plt.plot(model.validation_scores_, label='Validation score')
plt.axvline(x=model.n_iter_, color='r', linestyle='--', label='Early stopping point')
plt.xlabel('Iterations')
plt.ylabel('Loss / Score')
plt.title('Learning Curve with Early Stopping')
plt.legend()
plt.show()

print(f"Number of iterations: {model.n_iter_}")

Real-life Example: Image Classification

In image classification tasks, overfitting can occur when a model learns to recognize specific training images rather than generalizing to new, unseen images. Let's simulate this scenario using a simple convolutional neural network (CNN) for digit recognition.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from tensorflow.keras.datasets import mnist
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense
from tensorflow.keras.utils import to_categorical

# Load and preprocess MNIST data
(X_train, y_train), (X_test, y_test) = mnist.load_data()
X_train = X_train.reshape((X_train.shape[0], 28, 28, 1)).astype('float32') / 255
X_test = X_test.reshape((X_test.shape[0], 28, 28, 1)).astype('float32') / 255
y_train = to_categorical(y_train)
y_test = to_categorical(y_test)

# Create a small subset of training data to induce overfitting
X_train_small, _, y_train_small, _ = train_test_split(X_train, y_train, train_size=1000, random_state=42)

# Build the model
model = Sequential([
    Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1)),
    MaxPooling2D((2, 2)),
    Flatten(),
    Dense(100, activation='relu'),
    Dense(10, activation='softmax')
])
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

# Train the model
history = model.fit(X_train_small, y_train_small, epochs=20, validation_split=0.2, verbose=0)

# Plot training history
plt.figure(figsize=(10, 6))
plt.plot(history.history['accuracy'], label='Training Accuracy')
plt.plot(history.history['val_accuracy'], label='Validation Accuracy')
plt.xlabel('Epoch')
plt.ylabel('Accuracy')
plt.title('Model Accuracy')
plt.legend()
plt.show()

# Evaluate on test set
test_loss, test_acc = model.evaluate(X_test, y_test, verbose=0)
print(f"Test accuracy: {test_acc:.4f}")

Real-life Example: Text Classification

In text classification tasks, overfitting can occur when a model memorizes specific phrases or patterns in the training data instead of learning generalizable features. Let's demonstrate this using a simple text classification model for sentiment analysis.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.metrics import accuracy_score

# Sample data (simplified for demonstration)
texts = [
    "I love this movie", "Great film", "Awesome experience",
    "Terrible movie", "Waste of time", "Disappointing film",
    "The acting was good", "Beautiful cinematography", "Boring plot",
    "Exciting action scenes", "Poor character development", "Predictable ending"
]
labels = [1, 1, 1, 0, 0, 0, 1, 1, 0, 1, 0, 0]  # 1 for positive, 0 for negative

# Split data
X_train, X_test, y_train, y_test = train_test_split(texts, labels, test_size=0.3, random_state=42)

# Vectorize text data
vectorizer = CountVectorizer()
X_train_vec = vectorizer.fit_transform(X_train)
X_test_vec = vectorizer.transform(X_test)

# Train model
model = MultinomialNB()
model.fit(X_train_vec, y_train)

# Evaluate model
train_acc = model.score(X_train_vec, y_train)
test_acc = model.score(X_test_vec, y_test)

print(f"Training accuracy: {train_acc:.4f}")
print(f"Testing accuracy: {test_acc:.4f}")

# Plot feature importances
feature_importance = model.feature_log_prob_[1] - model.feature_log_prob_[0]
top_features = np.argsort(feature_importance)[-10:]
plt.figure(figsize=(10, 6))
plt.barh(np.array(vectorizer.get_feature_names())[top_features], feature_importance[top_features])
plt.title("Top 10 Most Important Features")
plt.xlabel("Log Probability Difference")
plt.show()

Conclusion and Best Practices

To avoid overfitting and underfitting, consider the following best practices:

1. Use cross-validation to assess model performance
2. Apply regularization techniques (L1, L2, Elastic Net)
3. Perform feature selection to reduce model complexity
4. Use ensemble methods to improve generalization
5. Implement early stopping for iterative algorithms
6. Collect more diverse and representative training data
7. Monitor learning curves to identify overfitting or underfitting
8. Use appropriate model complexity for the task at hand
9. Apply data augmentation techniques when applicable
10. Regularly test your model on unseen data to ensure generalization

By following these practices and understanding the concepts of overfitting, underfitting, bias, and variance, you can develop more robust and accurate machine learning models.

Additional Resources

For further reading on overfitting, underfitting, bias, and variance in machine learning, consider the following peer-reviewed articles from ArXiv:

1. "Understanding deep learning requires rethinking generalization" by Zhang et al. (2017) ArXiv: https://arxiv.org/abs/1611.03530
2. "Reconciling modern machine learning practice and the bias-variance trade-off" by Belkin et al. (2019) ArXiv: https://arxiv.org/abs/1812.11118
3. "Overfitting in Neural Nets: Backpropagation, Conjugate Gradient, and Early Stopping" by Caruana et al. (2000) ArXiv: https://arxiv.org/abs/cs/0006013

These papers provide in-depth analysis and insights into the topics discussed in this presentation.