xbe.at

Deep learning / Neural network fundamentals

Understanding the Role of Epochs in Training Neural Networks

9 min read, 14 sections

Contents
  1. Understanding Epochs in Neural Networks
  2. Visualizing Training Progress
  3. Implementing Early Stopping
  4. Custom Training Loop with Validation
  5. Learning Rate Scheduling
  6. Model Checkpointing
  7. Batch Normalization Implementation
  8. Dynamic Mini-batch Generation
  9. Advanced Learning Rate Finder
  10. Gradient Clipping Implementation
  11. Custom Loss Function with Regularization
  12. Epoch-wise Performance Metrics
  13. Real-world Implementation: MNIST Training
  14. Additional Resources

Understanding Epochs in Neural Networks

A fundamental concept in deep learning, epochs represent complete iterations through the training dataset. Each epoch allows the model to see every example once, updating weights through backpropagation to minimize the loss function. Understanding epoch behavior is crucial for achieving optimal model performance.

import numpy as np
import matplotlib.pyplot as plt

class SimpleNeuralNetwork:
    def __init__(self, input_size):
        # Initialize weights and biases
        self.weights = np.random.randn(input_size, 1)
        self.bias = np.zeros((1, 1))
        self.losses = []
    
    def train(self, X, y, epochs=100, learning_rate=0.01):
        for epoch in range(epochs):
            # Forward pass
            predictions = np.dot(X, self.weights) + self.bias
            
            # Calculate loss (MSE)
            loss = np.mean((predictions - y) ** 2)
            self.losses.append(loss)
            
            # Backward pass
            d_weights = np.dot(X.T, (predictions - y)) / len(X)
            d_bias = np.mean(predictions - y)
            
            # Update parameters
            self.weights -= learning_rate * d_weights
            self.bias -= learning_rate * d_bias
            
        return self.losses

# Example usage
X = np.random.randn(100, 1)
y = 2 * X + 1 + np.random.randn(100, 1) * 0.1
model = SimpleNeuralNetwork(1)
losses = model.train(X, y)

Visualizing Training Progress

Monitoring model training through loss visualization helps identify convergence patterns and potential issues like underfitting or overfitting. This visualization provides crucial insights into learning dynamics and helps determine optimal training duration.

def plot_training_progress(losses):
    plt.figure(figsize=(10, 6))
    plt.plot(losses)
    plt.title('Training Loss Over Epochs')
    plt.xlabel('Epoch')
    plt.ylabel('Loss')
    plt.grid(True)
    
    # Add annotations for key points
    min_loss_epoch = np.argmin(losses)
    plt.annotate(f'Minimum Loss\n{losses[min_loss_epoch]:.4f}',
                xy=(min_loss_epoch, losses[min_loss_epoch]),
                xytext=(min_loss_epoch+10, losses[min_loss_epoch]+0.1),
                arrowprops=dict(facecolor='black', shrink=0.05))
    
    return plt

# Visualize the training progress
plot = plot_training_progress(losses)
plt.show()

Implementing Early Stopping

Early stopping prevents overfitting by monitoring validation performance and halting training when the model stops improving. This technique is essential for optimal model generalization and computational efficiency.

class EarlyStopping:
    def __init__(self, patience=5, min_delta=0.001):
        self.patience = patience
        self.min_delta = min_delta
        self.counter = 0
        self.best_loss = None
        self.early_stop = False
    
    def __call__(self, val_loss):
        if self.best_loss is None:
            self.best_loss = val_loss
        elif val_loss > self.best_loss - self.min_delta:
            self.counter += 1
            if self.counter >= self.patience:
                self.early_stop = True
        else:
            self.best_loss = val_loss
            self.counter = 0
            
        return self.early_stop

Custom Training Loop with Validation

Implementing a custom training loop with validation provides fine-grained control over the training process and enables monitoring of model performance on unseen data to prevent overfitting.

def train_with_validation(model, X_train, y_train, X_val, y_val, epochs=100):
    early_stopping = EarlyStopping(patience=5)
    train_losses, val_losses = [], []
    
    for epoch in range(epochs):
        # Training step
        train_loss = model.train_step(X_train, y_train)
        train_losses.append(train_loss)
        
        # Validation step
        val_loss = model.evaluate(X_val, y_val)
        val_losses.append(val_loss)
        
        # Check early stopping
        if early_stopping(val_loss):
            print(f"Early stopping triggered at epoch {epoch}")
            break
            
    return train_losses, val_losses

Learning Rate Scheduling

Learning rate scheduling dynamically adjusts the learning rate during training to improve convergence. This technique helps overcome plateaus and allows the model to find better local minima by reducing the learning rate according to a predefined schedule.

class LearningRateScheduler:
    def __init__(self, initial_lr=0.01):
        self.initial_lr = initial_lr
        
    def step_decay(self, epoch):
        drop_rate = 0.5
        epochs_drop = 10.0
        lr = self.initial_lr * math.pow(drop_rate, math.floor((1+epoch)/epochs_drop))
        return lr
        
    def exponential_decay(self, epoch):
        decay_rate = 0.95
        lr = self.initial_lr * math.pow(decay_rate, epoch)
        return lr
    
    def cosine_decay(self, epoch, total_epochs):
        lr = self.initial_lr * (1 + math.cos(math.pi * epoch / total_epochs)) / 2
        return lr

# Example usage
scheduler = LearningRateScheduler(initial_lr=0.1)
for epoch in range(100):
    current_lr = scheduler.cosine_decay(epoch, 100)
    print(f"Epoch {epoch}, Learning Rate: {current_lr:.6f}")

Model Checkpointing

Model checkpointing saves the model's state during training, ensuring that the best performing version can be recovered. This technique is crucial for long training sessions and helps maintain optimal model performance.

class ModelCheckpoint:
    def __init__(self, filepath, monitor='val_loss', mode='min'):
        self.filepath = filepath
        self.monitor = monitor
        self.mode = mode
        self.best = float('inf') if mode == 'min' else float('-inf')
        
    def save_checkpoint(self, model, epoch, metrics):
        current = metrics[self.monitor]
        if (self.mode == 'min' and current < self.best) or \
           (self.mode == 'max' and current > self.best):
            self.best = current
            checkpoint = {
                'epoch': epoch,
                'model_state': model.state_dict(),
                'metrics': metrics,
                'best_score': self.best
            }
            torch.save(checkpoint, f"{self.filepath}/model_epoch_{epoch}.pt")
            return True
        return False

# Example usage
checkpoint = ModelCheckpoint('checkpoints/', monitor='val_loss')
for epoch in range(num_epochs):
    metrics = {'val_loss': current_val_loss}
    is_best = checkpoint.save_checkpoint(model, epoch, metrics)

Batch Normalization Implementation

Batch normalization stabilizes training by normalizing layer inputs, reducing internal covariate shift. This implementation shows how to apply batch normalization in a neural network layer.

class BatchNormalization:
    def __init__(self, num_features, eps=1e-5, momentum=0.1):
        self.eps = eps
        self.momentum = momentum
        self.gamma = np.ones(num_features)
        self.beta = np.zeros(num_features)
        self.running_mean = np.zeros(num_features)
        self.running_var = np.ones(num_features)
        
    def forward(self, x, training=True):
        if training:
            mean = np.mean(x, axis=0)
            var = np.var(x, axis=0)
            
            # Update running statistics
            self.running_mean = (1 - self.momentum) * self.running_mean + \
                              self.momentum * mean
            self.running_var = (1 - self.momentum) * self.running_var + \
                             self.momentum * var
        else:
            mean = self.running_mean
            var = self.running_var
            
        # Normalize
        x_normalized = (x - mean) / np.sqrt(var + self.eps)
        
        # Scale and shift
        return self.gamma * x_normalized + self.beta

Dynamic Mini-batch Generation

Efficient mini-batch generation is crucial for training deep learning models. This implementation shows how to create mini-batches dynamically while maintaining random sampling properties.

class BatchGenerator:
    def __init__(self, X, y, batch_size=32, shuffle=True):
        self.X = X
        self.y = y
        self.batch_size = batch_size
        self.shuffle = shuffle
        self.n_samples = X.shape[0]
        self.idx = np.arange(self.n_samples)
        
    def __iter__(self):
        if self.shuffle:
            np.random.shuffle(self.idx)
        self.current = 0
        return self
        
    def __next__(self):
        if self.current >= self.n_samples:
            raise StopIteration
            
        batch_idx = self.idx[self.current:self.current + self.batch_size]
        self.current += self.batch_size
        
        return self.X[batch_idx], self.y[batch_idx]

# Example usage
batch_gen = BatchGenerator(X_train, y_train, batch_size=32)
for X_batch, y_batch in batch_gen:
    # Training step with batch
    pass

Advanced Learning Rate Finder

The learning rate finder helps determine the optimal learning rate range for training neural networks. It implements the technique described in the "Cyclical Learning Rates for Training Neural Networks" paper.

class LearningRateFinder:
    def __init__(self, model, optimizer, criterion):
        self.model = model
        self.optimizer = optimizer
        self.criterion = criterion
        self.history = {'lr': [], 'loss': []}
        
    def range_test(self, train_loader, start_lr=1e-7, end_lr=10, num_iter=100):
        # Save initial model state
        initial_state = copy.deepcopy(self.model.state_dict())
        
        # Calculate multiplication factor
        mult = (end_lr / start_lr) ** (1/num_iter)
        lr = start_lr
        
        for batch_idx, (inputs, targets) in enumerate(train_loader):
            if batch_idx >= num_iter:
                break
                
            self.optimizer.param_groups[0]['lr'] = lr
            
            # Forward pass
            outputs = self.model(inputs)
            loss = self.criterion(outputs, targets)
            
            # Backward pass
            self.optimizer.zero_grad()
            loss.backward()
            self.optimizer.step()
            
            # Store learning rate and loss
            self.history['lr'].append(lr)
            self.history['loss'].append(loss.item())
            
            lr *= mult
            
        # Restore initial model state
        self.model.load_state_dict(initial_state)
        
        return self.history

Gradient Clipping Implementation

Gradient clipping prevents the exploding gradient problem by scaling gradients when their norm exceeds a threshold. This technique is particularly important for training deep recurrent neural networks.

def clip_gradients(model, max_norm=1.0):
    total_norm = 0
    parameters = [p for p in model.parameters() if p.grad is not None]
    
    # Calculate total norm of gradients
    for p in parameters:
        param_norm = p.grad.data.norm(2)
        total_norm += param_norm.item() ** 2
    total_norm = total_norm ** 0.5
    
    # Apply clipping
    clip_coef = max_norm / (total_norm + 1e-6)
    if clip_coef < 1:
        for p in parameters:
            p.grad.data.mul_(clip_coef)
            
    return total_norm

# Example usage in training loop
optimizer.zero_grad()
loss.backward()
grad_norm = clip_gradients(model, max_norm=1.0)
optimizer.step()

Custom Loss Function with Regularization

Implementing custom loss functions with regularization terms helps control model complexity and improve generalization. This example shows how to combine multiple loss terms with different weightings.

class CustomLoss(nn.Module):
    def __init__(self, lambda_l1=0.01, lambda_l2=0.01):
        super().__init__()
        self.lambda_l1 = lambda_l1
        self.lambda_l2 = lambda_l2
        
    def forward(self, predictions, targets, model):
        # Main loss (MSE)
        mse_loss = F.mse_loss(predictions, targets)
        
        # L1 regularization
        l1_reg = torch.tensor(0.)
        for param in model.parameters():
            l1_reg += torch.norm(param, 1)
            
        # L2 regularization
        l2_reg = torch.tensor(0.)
        for param in model.parameters():
            l2_reg += torch.norm(param, 2)
            
        # Combine losses
        total_loss = mse_loss + self.lambda_l1 * l1_reg + self.lambda_l2 * l2_reg
        
        return total_loss, {
            'mse_loss': mse_loss.item(),
            'l1_reg': l1_reg.item(),
            'l2_reg': l2_reg.item()
        }

Epoch-wise Performance Metrics

Comprehensive performance monitoring across epochs enables better understanding of model training dynamics. This implementation tracks multiple metrics and provides visualization capabilities for training analysis.

class MetricsTracker:
    def __init__(self):
        self.metrics = {
            'train_loss': [], 'val_loss': [],
            'train_acc': [], 'val_acc': [],
            'learning_rates': []
        }
        
    def update(self, epoch_metrics):
        for key, value in epoch_metrics.items():
            if key in self.metrics:
                self.metrics[key].append(value)
                
    def plot_metrics(self):
        fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15, 5))
        
        # Plot losses
        ax1.plot(self.metrics['train_loss'], label='Training Loss')
        ax1.plot(self.metrics['val_loss'], label='Validation Loss')
        ax1.set_title('Loss vs Epochs')
        ax1.set_xlabel('Epoch')
        ax1.set_ylabel('Loss')
        ax1.legend()
        
        # Plot accuracies
        ax2.plot(self.metrics['train_acc'], label='Training Accuracy')
        ax2.plot(self.metrics['val_acc'], label='Validation Accuracy')
        ax2.set_title('Accuracy vs Epochs')
        ax2.set_xlabel('Epoch')
        ax2.set_ylabel('Accuracy')
        ax2.legend()
        
        plt.tight_layout()
        return fig

# Example usage
tracker = MetricsTracker()
for epoch in range(num_epochs):
    epoch_metrics = {
        'train_loss': current_train_loss,
        'val_loss': current_val_loss,
        'train_acc': current_train_acc,
        'val_acc': current_val_acc,
        'learning_rates': current_lr
    }
    tracker.update(epoch_metrics)

Real-world Implementation: MNIST Training

Complete implementation of neural network training on the MNIST dataset, incorporating multiple techniques discussed in previous slides for optimal performance.

import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import datasets, transforms

class MNISTTrainer:
    def __init__(self, model, learning_rate=0.001):
        self.device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
        self.model = model.to(self.device)
        self.optimizer = optim.Adam(model.parameters(), lr=learning_rate)
        self.criterion = nn.CrossEntropyLoss()
        self.scheduler = optim.lr_scheduler.ReduceLROnPlateau(self.optimizer)
        self.metrics = MetricsTracker()
        
    def train_epoch(self, train_loader):
        self.model.train()
        total_loss = 0
        correct = 0
        
        for batch_idx, (data, target) in enumerate(train_loader):
            data, target = data.to(self.device), target.to(self.device)
            
            self.optimizer.zero_grad()
            output = self.model(data)
            loss = self.criterion(output, target)
            
            loss.backward()
            clip_gradients(self.model)
            self.optimizer.step()
            
            total_loss += loss.item()
            pred = output.argmax(dim=1, keepdim=True)
            correct += pred.eq(target.view_as(pred)).sum().item()
            
        return total_loss / len(train_loader), correct / len(train_loader.dataset)

    def validate(self, val_loader):
        self.model.eval()
        val_loss = 0
        correct = 0
        
        with torch.no_grad():
            for data, target in val_loader:
                data, target = data.to(self.device), target.to(self.device)
                output = self.model(data)
                val_loss += self.criterion(output, target).item()
                pred = output.argmax(dim=1, keepdim=True)
                correct += pred.eq(target.view_as(pred)).sum().item()
                
        return val_loss / len(val_loader), correct / len(val_loader.dataset)

Additional Resources

More in Neural network fundamentals

  1. Weight and Bias Fundamentals for Deep Learning11 min

    Neural network weights and biases are critical parameters that determine how input signals are transformed through the network. Proper initialization of...

  2. Simple Neural Network Implementation in Python9 min

    Neural networks are computational models inspired by the human brain. They consist of interconnected nodes (neurons) that process and transmit information....

  3. Weight Initialization Techniques for Deep Learning in Python11 min

    Weight initialization is a crucial step in training deep neural networks. It involves setting initial values for the network's parameters before the training...

  4. Perceptron Model and Implementation in Python6 min

    The perceptron is a fundamental building block of neural networks, representing a mathematical model of a biological neuron. It takes multiple input signals...

  5. Weight Initialization Techniques in Deep Learning12 min

    Deep learning models are highly sensitive to initial weight values. A proper initialization scheme helps prevent vanishing/exploding gradients and ensures...

  6. Neural Networks Layered Learning for Pattern Recognition14 min

    Neural networks are computational models inspired by the human brain, designed to recognize patterns and solve complex problems. They consist of...