xbe.at

Machine learning / Regularization overfitting and optimization

Label Smoothing Technique for Regularizing ML Models in Python

8 min read, 14 sections

Contents
  1. Introduction to Label Smoothing
  2. Why Use Label Smoothing?
  3. Mathematical Formulation
  4. Implementing Label Smoothing in TensorFlow
  5. Label Smoothing in PyTorch
  6. Real-Life Example: Image Classification
  7. Visualizing the Effect of Label Smoothing
  8. Label Smoothing in Multi-Label Classification
  9. Hyperparameter Tuning for Label Smoothing
  10. Real-Life Example: Natural Language Processing
  11. Comparing Label Smoothing with Other Regularization Techniques
  12. Label Smoothing and Model Calibration
  13. Limitations and Considerations
  14. Additional Resources

Introduction to Label Smoothing

Label smoothing is a regularization technique used in machine learning to improve model generalization and prevent overfitting. It works by softening the hard labels in classification tasks, introducing a small amount of uncertainty to the training process.

import numpy as np

def smooth_labels(labels, factor=0.1):
    n_classes = len(np.unique(labels))
    smooth_labels = (1 - factor) * labels + factor / n_classes
    return smooth_labels

# Example usage
original_labels = np.array([1, 0, 0, 1, 1])
smoothed_labels = smooth_labels(original_labels)
print("Original labels:", original_labels)
print("Smoothed labels:", smoothed_labels)

Why Use Label Smoothing?

Label smoothing addresses the issue of overconfidence in neural networks. When models are trained with hard labels (0 or 1), they may become too confident in their predictions, leading to poor generalization. By introducing uncertainty, label smoothing encourages the model to be more robust and adaptable.

import matplotlib.pyplot as plt

def plot_confidence_distribution(predictions):
    plt.hist(predictions, bins=20, range=(0, 1))
    plt.title("Model Confidence Distribution")
    plt.xlabel("Confidence")
    plt.ylabel("Frequency")
    plt.show()

# Example usage
overconfident_preds = np.random.beta(10, 1, 1000)
balanced_preds = np.random.beta(2, 2, 1000)

plt.figure(figsize=(12, 4))
plt.subplot(121)
plot_confidence_distribution(overconfident_preds)
plt.subplot(122)
plot_confidence_distribution(balanced_preds)

Mathematical Formulation

Label smoothing modifies the target distribution by combining the original hard labels with a uniform distribution over all classes. The smoothed label y' for class i is calculated as:

y'_i = (1 - α) * y_i + α / K

Where:

  • y_i is the original label (0 or 1)
  • α is the smoothing factor (typically a small value like 0.1)
  • K is the number of classes
def label_smoothing(y, alpha, K):
    return (1 - alpha) * y + alpha / K

# Example
y = np.array([1, 0, 0])  # One-hot encoded label
alpha = 0.1
K = 3
smoothed_y = label_smoothing(y, alpha, K)
print("Original label:", y)
print("Smoothed label:", smoothed_y)

Implementing Label Smoothing in TensorFlow

TensorFlow provides built-in support for label smoothing through its loss functions. Here's an example of how to use label smoothing with categorical crossentropy loss:

import tensorflow as tf

# Create a model
model = tf.keras.Sequential([
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dense(10, activation='softmax')
])

# Compile the model with label smoothing
model.compile(
    optimizer='adam',
    loss=tf.keras.losses.CategoricalCrossentropy(label_smoothing=0.1),
    metrics=['accuracy']
)

# Train the model (assuming x_train and y_train are defined)
# model.fit(x_train, y_train, epochs=10, validation_split=0.2)

Label Smoothing in PyTorch

PyTorch also supports label smoothing through its loss functions. Here's an example of implementing label smoothing with cross-entropy loss in PyTorch:

import torch
import torch.nn as nn

class LabelSmoothingLoss(nn.Module):
    def __init__(self, classes, smoothing=0.1):
        super(LabelSmoothingLoss, self).__init__()
        self.confidence = 1.0 - smoothing
        self.smoothing = smoothing
        self.classes = classes

    def forward(self, pred, target):
        pred = pred.log_softmax(dim=-1)
        with torch.no_grad():
            true_dist = torch.zeros_like(pred)
            true_dist.fill_(self.smoothing / (self.classes - 1))
            true_dist.scatter_(1, target.data.unsqueeze(1), self.confidence)
        return torch.mean(torch.sum(-true_dist * pred, dim=-1))

# Usage
criterion = LabelSmoothingLoss(classes=10, smoothing=0.1)
# loss = criterion(model_output, target)

Real-Life Example: Image Classification

Label smoothing is commonly used in image classification tasks. Let's consider a model classifying different types of fruits:

import tensorflow as tf
from tensorflow.keras.applications import MobileNetV2
from tensorflow.keras.layers import Dense, GlobalAveragePooling2D
from tensorflow.keras.models import Model

# Create a base model from MobileNetV2
base_model = MobileNetV2(weights='imagenet', include_top=False)

# Add custom layers
x = base_model.output
x = GlobalAveragePooling2D()(x)
x = Dense(1024, activation='relu')(x)
output = Dense(5, activation='softmax')(x)  # 5 fruit classes

model = Model(inputs=base_model.input, outputs=output)

# Compile with label smoothing
model.compile(
    optimizer='adam',
    loss=tf.keras.losses.CategoricalCrossentropy(label_smoothing=0.1),
    metrics=['accuracy']
)

# Train the model (assuming fruit_images and fruit_labels are defined)
# model.fit(fruit_images, fruit_labels, epochs=10, validation_split=0.2)

Visualizing the Effect of Label Smoothing

To better understand how label smoothing affects the model's predictions, let's visualize the softmax output distribution with and without label smoothing:

import numpy as np
import matplotlib.pyplot as plt

def softmax(x):
    return np.exp(x) / np.sum(np.exp(x), axis=-1, keepdims=True)

def plot_softmax_output(logits, smoothing=0.0):
    probs = softmax(logits)
    smoothed_probs = (1 - smoothing) * probs + smoothing / len(probs)
    
    plt.bar(range(len(probs)), smoothed_probs)
    plt.title(f"Softmax Output (Smoothing: {smoothing})")
    plt.xlabel("Class")
    plt.ylabel("Probability")
    plt.ylim(0, 1)
    plt.show()

# Example usage
logits = np.array([2.0, 1.0, 0.5, 0.2, 0.1])
plt.figure(figsize=(12, 4))
plt.subplot(121)
plot_softmax_output(logits, smoothing=0.0)
plt.subplot(122)
plot_softmax_output(logits, smoothing=0.1)

Label Smoothing in Multi-Label Classification

Label smoothing can also be applied to multi-label classification problems. In this case, we need to modify the smoothing approach slightly:

import numpy as np

def smooth_multilabel(labels, alpha=0.1):
    smoothed = labels.()
    smoothed *= (1 - alpha)
    smoothed += alpha / labels.shape[1]
    return smoothed

# Example usage
original_labels = np.array([
    [1, 0, 1, 0],
    [0, 1, 1, 1],
    [1, 1, 0, 0]
])

smoothed_labels = smooth_multilabel(original_labels)
print("Original labels:\n", original_labels)
print("\nSmoothed labels:\n", smoothed_labels)

Hyperparameter Tuning for Label Smoothing

The smoothing factor (α) is a hyperparameter that can be tuned. Here's an example of how to perform a grid search to find the optimal smoothing factor:

from sklearn.model_selection import GridSearchCV
from tensorflow.keras.wrappers.scikit_learn import KerasClassifier

def create_model(label_smoothing=0.1):
    model = tf.keras.Sequential([
        tf.keras.layers.Dense(64, activation='relu', input_shape=(10,)),
        tf.keras.layers.Dense(32, activation='relu'),
        tf.keras.layers.Dense(5, activation='softmax')
    ])
    model.compile(
        optimizer='adam',
        loss=tf.keras.losses.CategoricalCrossentropy(label_smoothing=label_smoothing),
        metrics=['accuracy']
    )
    return model

# Create a KerasClassifier
model = KerasClassifier(build_fn=create_model, epochs=10, batch_size=32, verbose=0)

# Define the grid search parameters
param_grid = {
    'label_smoothing': [0.0, 0.05, 0.1, 0.15, 0.2]
}

# Perform grid search
grid = GridSearchCV(estimator=model, param_grid=param_grid, cv=3)
# grid_result = grid.fit(X, y)

# Print results
# print(f"Best: {grid_result.best_score_} using {grid_result.best_params_}")

Real-Life Example: Natural Language Processing

Label smoothing is also beneficial in NLP tasks, such as text classification. Let's implement a simple sentiment analysis model with label smoothing:

import tensorflow as tf
from tensorflow.keras.layers import Embedding, LSTM, Dense
from tensorflow.keras.models import Sequential

# Assuming we have a vocabulary size of 10000 and maximum sequence length of 100
vocab_size = 10000
max_length = 100
embedding_dim = 16

model = Sequential([
    Embedding(vocab_size, embedding_dim, input_length=max_length),
    LSTM(64),
    Dense(32, activation='relu'),
    Dense(2, activation='softmax')  # Binary classification (positive/negative)
])

model.compile(
    optimizer='adam',
    loss=tf.keras.losses.CategoricalCrossentropy(label_smoothing=0.1),
    metrics=['accuracy']
)

# Train the model (assuming text_sequences and sentiment_labels are defined)
# model.fit(text_sequences, sentiment_labels, epochs=10, validation_split=0.2)

Comparing Label Smoothing with Other Regularization Techniques

Let's compare label smoothing with other common regularization techniques like L2 regularization and dropout:

import tensorflow as tf

def create_model(regularization='none'):
    model = tf.keras.Sequential([
        tf.keras.layers.Dense(64, activation='relu', input_shape=(10,))
    ])
    
    if regularization == 'l2':
        model.add(tf.keras.layers.Dense(32, activation='relu', 
                                        kernel_regularizer=tf.keras.regularizers.l2(0.01)))
    elif regularization == 'dropout':
        model.add(tf.keras.layers.Dropout(0.5))
        model.add(tf.keras.layers.Dense(32, activation='relu'))
    else:
        model.add(tf.keras.layers.Dense(32, activation='relu'))
    
    model.add(tf.keras.layers.Dense(5, activation='softmax'))
    
    if regularization == 'label_smoothing':
        loss = tf.keras.losses.CategoricalCrossentropy(label_smoothing=0.1)
    else:
        loss = 'categorical_crossentropy'
    
    model.compile(optimizer='adam', loss=loss, metrics=['accuracy'])
    return model

# Create models with different regularization techniques
models = {
    'No Regularization': create_model('none'),
    'L2 Regularization': create_model('l2'),
    'Dropout': create_model('dropout'),
    'Label Smoothing': create_model('label_smoothing')
}

# Train and evaluate models (assuming X_train, y_train, X_test, y_test are defined)
# for name, model in models.items():
#     model.fit(X_train, y_train, epochs=10, validation_split=0.2, verbose=0)
#     test_loss, test_acc = model.evaluate(X_test, y_test, verbose=0)
#     print(f"{name}: Test accuracy: {test_acc:.4f}")

Label Smoothing and Model Calibration

Label smoothing can improve model calibration, which is the alignment between a model's confidence and its accuracy. Let's visualize the calibration curve:

import numpy as np
import matplotlib.pyplot as plt
from sklearn.calibration import calibration_curve

def plot_calibration_curve(y_true, y_pred, n_bins=10):
    prob_true, prob_pred = calibration_curve(y_true, y_pred, n_bins=n_bins)
    
    plt.plot(prob_pred, prob_true, marker='o')
    plt.plot([0, 1], [0, 1], linestyle='--')
    plt.xlabel('Mean Predicted Probability')
    plt.ylabel('Fraction of Positives')
    plt.title('Calibration Curve')
    plt.show()

# Generate sample data
np.random.seed(42)
y_true = np.random.randint(0, 2, 1000)
y_pred_uncalibrated = np.random.beta(5, 2, 1000)
y_pred_calibrated = 0.9 * y_pred_uncalibrated + 0.05  # Simulating label smoothing effect

plt.figure(figsize=(12, 4))
plt.subplot(121)
plot_calibration_curve(y_true, y_pred_uncalibrated)
plt.subplot(122)
plot_calibration_curve(y_true, y_pred_calibrated)

Limitations and Considerations

While label smoothing is a powerful technique, it's important to consider its limitations:

  1. Not always beneficial: In some cases, especially with small datasets or simple problems, label smoothing might not provide significant improvements.
  2. Hyperparameter sensitivity: The smoothing factor needs to be carefully tuned, as too much smoothing can harm performance.
  3. Interpretation challenges: Smoothed labels can make model interpretation more difficult, especially in tasks where hard decisions are required.
import numpy as np
import matplotlib.pyplot as plt

def plot_accuracy_vs_smoothing(smoothing_factors, accuracies):
    plt.plot(smoothing_factors, accuracies, marker='o')
    plt.xlabel('Smoothing Factor')
    plt.ylabel('Accuracy')
    plt.title('Accuracy vs. Smoothing Factor')
    plt.show()

# Simulated data
smoothing_factors = np.linspace(0, 0.5, 11)
accuracies = [0.85, 0.87, 0.89, 0.90, 0.91, 0.90, 0.88, 0.86, 0.84, 0.82, 0.80]

plot_accuracy_vs_smoothing(smoothing_factors, accuracies)

Additional Resources

For more information on label smoothing and related techniques, consider exploring these resources:

  1. "Rethinking the Inception Architecture for Computer Vision" by Szegedy et al. (2016) - The original paper introducing label smoothing (arXiv:1512.00567)
  2. "When Does Label Smoothing Help?" by Müller et al. (2019) - An in-depth analysis of label smoothing's effects (arXiv:1906.02629)
  3. "Regularizing Neural Networks by Penalizing Confident Output Distributions" by Pereyra et al. (2017) - Explores the connection between label smoothing and confidence penalties (arXiv:1701.06548)

These papers can be found on ArXiv.org by searching for their respective arXiv identifiers.

More in Regularization overfitting and optimization

  1. Machine Learning Hyperparameter Tuning8 min

    Linear regression hyperparameters significantly influence model performance through regularization strength (alpha) and penalty type (l1/l2). Understanding...

  2. L1 Regularization for Sparse Models8 min

    L1 regularization, also known as Lasso (Least Absolute Shrinkage and Selection Operator), adds a penalty term proportional to the absolute values of model...

  3. Machine Learning with Hyperparameter Tuning in Python10 min

    Machine Learning (ML) is a subset of artificial intelligence that enables systems to learn and improve from experience without being explicitly programmed....

  4. Inductive Biases in Machine Learning! Decoding AI Predictions10 min

    Inductive biases are fundamental assumptions that machine learning models use to generalize from training data to unseen examples. These biases shape how...

  5. Mastering Bias, Variance, Overfitting, and Underfitting in Python14 min

    Bias and variance are fundamental concepts in machine learning that help us understand model performance. Bias refers to the error introduced by...

  6. Identifying Underfitting in Machine Learning9 min

    High bias or underfitting occurs when a model performs poorly on both training and test datasets, indicating the model is too simple to capture the...