xbe.at

Deep learning / Backpropagation and optimization

Exploring the Advantages of ADAM Optimizer for Deep Learning

8 min read, 15 sections

Contents
  1. ADAM Optimizer Fundamentals
  2. ADAM Update Mechanics
  3. Mathematical Foundation
  4. Simple Neural Network with ADAM
  5. Neural Network Forward Pass
  6. Neural Network Backward Pass with ADAM
  7. ADAM Implementation for Image Classification
  8. Training Loop with ADAM
  9. ADAM with Momentum Control
  10. ADAM with Weight Decay
  11. Real-world Application: Time Series Prediction
  12. Time Series RNN Training with ADAM
  13. Performance Metrics and Visualization
  14. Advanced ADAM Variants
  15. Additional Resources

ADAM Optimizer Fundamentals

The ADAM optimizer combines momentum and RMSprop approaches to provide adaptive learning rates for each parameter. It maintains both first and second moment estimates of gradients, enabling efficient parameter updates while accounting for both gradient magnitude and variance.

import numpy as np

class AdamOptimizer:
    def __init__(self, learning_rate=0.001, beta1=0.9, beta2=0.999, epsilon=1e-8):
        self.learning_rate = learning_rate
        self.beta1 = beta1
        self.beta2 = beta2
        self.epsilon = epsilon
        self.m = None  # First moment estimate
        self.v = None  # Second moment estimate
        self.t = 0     # Timestep

    def initialize(self, shape):
        self.m = np.zeros(shape)
        self.v = np.zeros(shape)

ADAM Update Mechanics

The core of ADAM lies in its parameter update rule, which uses exponentially moving averages of gradients and squared gradients. This implementation shows the mathematical operations behind ADAM's adaptive learning process.

def update(self, params, grads):
    if self.m is None:
        self.initialize(params.shape)
    
    self.t += 1
    
    # Update biased first moment estimate
    self.m = self.beta1 * self.m + (1 - self.beta1) * grads
    
    # Update biased second moment estimate
    self.v = self.beta2 * self.v + (1 - self.beta2) * np.square(grads)
    
    # Compute bias-corrected first moment estimate
    m_hat = self.m / (1 - self.beta1**self.t)
    
    # Compute bias-corrected second moment estimate
    v_hat = self.v / (1 - self.beta2**self.t)
    
    # Update parameters
    params -= self.learning_rate * m_hat / (np.sqrt(v_hat) + self.epsilon)
    
    return params

Mathematical Foundation

The ADAM optimizer's mathematical foundation is built on moment estimation and bias correction. The following equations represent the core calculations performed during optimization.

"""
The key equations for ADAM:

$$m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t$$
$$v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2$$
$$\hat{m}_t = \frac{m_t}{1 - \beta_1^t}$$
$$\hat{v}_t = \frac{v_t}{1 - \beta_2^t}$$
$$\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{\hat{v}_t} + \epsilon} \hat{m}_t$$

Where:
- m_t: First moment estimate
- v_t: Second moment estimate
- g_t: Current gradient
- β₁, β₂: Decay rates
- η: Learning rate
- ε: Small constant for numerical stability
"""

Simple Neural Network with ADAM

This implementation demonstrates ADAM's application in training a basic neural network for binary classification, showing how the optimizer handles weight updates in practice.

import numpy as np

class SimpleNeuralNetwork:
    def __init__(self, input_size, hidden_size, output_size):
        self.W1 = np.random.randn(input_size, hidden_size) * 0.01
        self.b1 = np.zeros((1, hidden_size))
        self.W2 = np.random.randn(hidden_size, output_size) * 0.01
        self.b2 = np.zeros((1, output_size))
        
        # Initialize ADAM optimizers for each parameter
        self.optimizer_W1 = AdamOptimizer()
        self.optimizer_b1 = AdamOptimizer()
        self.optimizer_W2 = AdamOptimizer()
        self.optimizer_b2 = AdamOptimizer()

Neural Network Forward Pass

The forward pass computation demonstrates how data flows through the network before the ADAM optimizer updates the parameters during backpropagation.

def forward(self, X):
    # Hidden layer
    self.z1 = np.dot(X, self.W1) + self.b1
    self.a1 = np.tanh(self.z1)
    
    # Output layer
    self.z2 = np.dot(self.a1, self.W2) + self.b2
    self.a2 = 1 / (1 + np.exp(-self.z2))  # Sigmoid activation
    
    return self.a2

def loss(self, y_true, y_pred):
    m = y_true.shape[0]
    return -np.sum(y_true * np.log(y_pred) + 
                  (1 - y_true) * np.log(1 - y_pred)) / m

Neural Network Backward Pass with ADAM

The backward pass calculates gradients and applies ADAM optimization to update network parameters. This implementation shows how ADAM handles different parameter updates during backpropagation.

def backward(self, X, y, learning_rate=0.001):
    m = X.shape[0]
    
    # Output layer gradients
    dZ2 = self.a2 - y
    dW2 = np.dot(self.a1.T, dZ2) / m
    db2 = np.sum(dZ2, axis=0, keepdims=True) / m
    
    # Hidden layer gradients
    dZ1 = np.dot(dZ2, self.W2.T) * (1 - np.square(self.a1))
    dW1 = np.dot(X.T, dZ1) / m
    db1 = np.sum(dZ1, axis=0, keepdims=True) / m
    
    # Update parameters using ADAM
    self.W1 = self.optimizer_W1.update(self.W1, dW1)
    self.b1 = self.optimizer_b1.update(self.b1, db1)
    self.W2 = self.optimizer_W2.update(self.W2, dW2)
    self.b2 = self.optimizer_b2.update(self.b2, db2)

ADAM Implementation for Image Classification

A practical implementation of ADAM optimizer for training a convolutional neural network on image data, demonstrating its effectiveness with high-dimensional inputs.

import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

# Load and preprocess data
digits = load_digits()
X = digits.data / 255.0  # Normalize pixel values
y = np.eye(10)[digits.target]  # One-hot encode targets

# Split dataset
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Initialize network and train
model = SimpleNeuralNetwork(input_size=64, hidden_size=128, output_size=10)
adam_optimizer = AdamOptimizer(learning_rate=0.001)

# Training loop
epochs = 100
batch_size = 32

Training Loop with ADAM

The training process demonstrates how ADAM adaptively adjusts learning rates during optimization, showing the practical implementation of batch processing and gradient updates.

def train_network(model, X_train, y_train, epochs, batch_size):
    losses = []
    n_batches = len(X_train) // batch_size
    
    for epoch in range(epochs):
        epoch_loss = 0
        # Shuffle training data
        indices = np.random.permutation(len(X_train))
        X_shuffled = X_train[indices]
        y_shuffled = y_train[indices]
        
        for batch in range(n_batches):
            start_idx = batch * batch_size
            end_idx = start_idx + batch_size
            
            # Get batch
            X_batch = X_shuffled[start_idx:end_idx]
            y_batch = y_shuffled[start_idx:end_idx]
            
            # Forward pass
            y_pred = model.forward(X_batch)
            
            # Backward pass with ADAM updates
            model.backward(X_batch, y_batch)
            
            # Calculate loss
            batch_loss = model.loss(y_batch, y_pred)
            epoch_loss += batch_loss
            
        losses.append(epoch_loss / n_batches)
        
    return losses

ADAM with Momentum Control

Advanced implementation showing how to adjust ADAM's momentum parameters dynamically during training for better convergence in different optimization phases.

class AdaptiveAdam(AdamOptimizer):
    def __init__(self, learning_rate=0.001, beta1=0.9, beta2=0.999,
                 epsilon=1e-8, momentum_schedule='constant'):
        super().__init__(learning_rate, beta1, beta2, epsilon)
        self.momentum_schedule = momentum_schedule
        self.initial_beta1 = beta1
        
    def adjust_momentum(self, epoch):
        if self.momentum_schedule == 'decay':
            # Decay momentum over time
            self.beta1 = self.initial_beta1 / (1 + epoch * 0.1)
        elif self.momentum_schedule == 'cyclic':
            # Cyclic momentum
            self.beta1 = self.initial_beta1 * (1 + np.sin(epoch * np.pi / 10)) / 2
            
    def update(self, params, grads, epoch=0):
        self.adjust_momentum(epoch)
        return super().update(params, grads)

ADAM with Weight Decay

Implementation of ADAMW, a variant of ADAM that properly decouples weight decay from the gradient update, improving generalization in deep neural networks.

class AdamW(AdamOptimizer):
    def __init__(self, learning_rate=0.001, beta1=0.9, beta2=0.999,
                 epsilon=1e-8, weight_decay=0.01):
        super().__init__(learning_rate, beta1, beta2, epsilon)
        self.weight_decay = weight_decay
    
    def update(self, params, grads):
        if self.m is None:
            self.initialize(params.shape)
        
        self.t += 1
        
        # Apply weight decay
        params = params * (1 - self.learning_rate * self.weight_decay)
        
        # Standard ADAM updates
        self.m = self.beta1 * self.m + (1 - self.beta1) * grads
        self.v = self.beta2 * self.v + (1 - self.beta2) * np.square(grads)
        
        m_hat = self.m / (1 - self.beta1**self.t)
        v_hat = self.v / (1 - self.beta2**self.t)
        
        params -= self.learning_rate * m_hat / (np.sqrt(v_hat) + self.epsilon)
        
        return params

Real-world Application: Time Series Prediction

Implementation of ADAM optimizer for a recurrent neural network tackling time series prediction, demonstrating its effectiveness with sequential data.

import numpy as np
from sklearn.preprocessing import MinMaxScaler

class TimeSeriesRNN:
    def __init__(self, input_size, hidden_size, output_size):
        # Initialize weights
        self.Wx = np.random.randn(input_size, hidden_size) * 0.01
        self.Wh = np.random.randn(hidden_size, hidden_size) * 0.01
        self.Wy = np.random.randn(hidden_size, output_size) * 0.01
        
        # Initialize ADAM optimizers
        self.optimizer_Wx = AdamOptimizer(learning_rate=0.001)
        self.optimizer_Wh = AdamOptimizer(learning_rate=0.001)
        self.optimizer_Wy = AdamOptimizer(learning_rate=0.001)
        
        self.hidden_size = hidden_size
        self.h = np.zeros((1, hidden_size))

Time Series RNN Training with ADAM

The training process for time series prediction showcases ADAM's ability to handle vanishing gradients in recurrent neural networks through adaptive moment estimation.

def train_time_series(self, X, y, epochs=100):
    losses = []
    
    for epoch in range(epochs):
        total_loss = 0
        h = np.zeros((1, self.hidden_size))
        
        for t in range(len(X)):
            # Forward pass
            x_t = X[t:t+1]
            h = np.tanh(np.dot(x_t, self.Wx) + np.dot(h, self.Wh))
            y_pred = np.dot(h, self.Wy)
            
            # Compute loss
            loss = np.mean((y_pred - y[t:t+1]) ** 2)
            total_loss += loss
            
            # Backward pass
            dWy = np.dot(h.T, (y_pred - y[t:t+1]))
            dh = np.dot((y_pred - y[t:t+1]), self.Wy.T)
            dWx = np.dot(x_t.T, dh)
            dWh = np.dot(h.T, dh)
            
            # Update weights using ADAM
            self.Wx = self.optimizer_Wx.update(self.Wx, dWx)
            self.Wh = self.optimizer_Wh.update(self.Wh, dWh)
            self.Wy = self.optimizer_Wy.update(self.Wy, dWy)
            
        losses.append(total_loss / len(X))
    return losses

Performance Metrics and Visualization

Implementation of comprehensive performance tracking and visualization tools for monitoring ADAM optimizer behavior during training.

class AdamPerformanceTracker:
    def __init__(self):
        self.learning_rates = []
        self.gradient_norms = []
        self.parameter_updates = []
        
    def track_update(self, learning_rate, gradient, parameter_update):
        self.learning_rates.append(learning_rate)
        self.gradient_norms.append(np.linalg.norm(gradient))
        self.parameter_updates.append(np.linalg.norm(parameter_update))
        
    def plot_metrics(self):
        import matplotlib.pyplot as plt
        
        fig, (ax1, ax2, ax3) = plt.subplots(3, 1, figsize=(10, 12))
        
        ax1.plot(self.learning_rates)
        ax1.set_title('Effective Learning Rates')
        
        ax2.plot(self.gradient_norms)
        ax2.set_title('Gradient Norms')
        
        ax3.plot(self.parameter_updates)
        ax3.set_title('Parameter Update Magnitudes')
        
        plt.tight_layout()

Advanced ADAM Variants

Implementation of advanced ADAM variants including AMSGrad and AdaFactor, showing improvements for specific optimization scenarios.

class AMSGrad(AdamOptimizer):
    def __init__(self, learning_rate=0.001, beta1=0.9, beta2=0.999, epsilon=1e-8):
        super().__init__(learning_rate, beta1, beta2, epsilon)
        self.v_max = None
        
    def update(self, params, grads):
        if self.m is None:
            self.initialize(params.shape)
            self.v_max = np.zeros_like(params)
            
        self.t += 1
        
        # Update biased first moment estimate
        self.m = self.beta1 * self.m + (1 - self.beta1) * grads
        
        # Update biased second moment estimate
        self.v = self.beta2 * self.v + (1 - self.beta2) * np.square(grads)
        
        # Update maximum of v
        self.v_max = np.maximum(self.v_max, self.v)
        
        # Compute bias-corrected first moment estimate
        m_hat = self.m / (1 - self.beta1**self.t)
        
        # Use maximum of v for update
        params -= self.learning_rate * m_hat / (np.sqrt(self.v_max) + self.epsilon)
        
        return params

Additional Resources

More in Backpropagation and optimization

  1. Gradient Clipping and Learning Rate Scheduling in Python.md7 min

    Gradient clipping is a technique used in training deep neural networks to prevent the problem of exploding gradients, where the gradients become too large...

  2. Explaining the Vanishing Gradient Problem in Neural Networks with Python11 min

    The vanishing gradient problem is a significant challenge in training deep neural networks. It occurs when the gradients of the loss function approach zero...

  3. Gradient Descent & Optimization Explorer in Python7 min

    Gradient Descent is a fundamental optimization algorithm used in machine learning to minimize the cost function of a model. It iteratively adjusts parameters...

  4. Efficient Optimizers for Neural Network Training11 min

    AdamW and Adam are popular optimization algorithms for training neural networks. These algorithms adapt learning rates for each parameter, which can lead to...

  5. Gradient Descent Explained with Python10 min

    Gradient Descent is a fundamental optimization algorithm used in machine learning and deep learning. It's designed to find the minimum of a function by...

  6. Chain Rule in Neural Network Backpropagation16 min

    The chain rule is a fundamental concept in calculus that plays a crucial role in the backpropagation algorithm used to train neural networks. It allows us to...