xbe.at

Deep learning / CNN and computer vision

Identity Mappings in Deep Residual Networks with Python

10 min read, 15 sections

Contents
  1. Identity Mappings in Deep Residual Networks
  2. The Vanishing Gradient Problem
  3. Residual Blocks with Identity Mappings
  4. Pre-activation vs Post-activation
  5. Implementing a Basic ResNet
  6. Training a ResNet with Identity Mappings
  7. Visualizing Feature Maps
  8. Gradient Flow Analysis
  9. Comparing ResNet with and without Identity Mappings
  10. Real-life Example: Image Classification
  11. Real-life Example: Transfer Learning
  12. Handling Different Input Sizes
  13. Residual Networks for Object Detection
  14. ResNet for Semantic Segmentation
  15. Additional Resources

Identity Mappings in Deep Residual Networks

Identity mappings are a key component in deep residual networks, enhancing gradient flow and easing optimization. They allow for the creation of very deep networks by addressing the vanishing gradient problem. Let's explore their implementation and benefits using Python and TensorFlow.

import tensorflow as tf

def identity_block(x, filters, kernel_size):
    shortcut = x
    x = tf.keras.layers.Conv2D(filters, kernel_size, padding='same')(x)
    x = tf.keras.layers.BatchNormalization()(x)
    x = tf.keras.layers.Activation('relu')(x)
    x = tf.keras.layers.Conv2D(filters, kernel_size, padding='same')(x)
    x = tf.keras.layers.BatchNormalization()(x)
    x = tf.keras.layers.Add()([shortcut, x])
    x = tf.keras.layers.Activation('relu')(x)
    return x

The Vanishing Gradient Problem

The vanishing gradient problem occurs when gradients become extremely small as they propagate backwards through deep networks. This hinders learning in early layers. Identity mappings in residual networks provide a direct path for gradients, mitigating this issue.

def visualize_gradients(model, input_data):
    with tf.GradientTape() as tape:
        output = model(input_data)
        loss = tf.reduce_mean(output)
    gradients = tape.gradient(loss, model.trainable_variables)
    for i, grad in enumerate(gradients):
        print(f"Layer {i} gradient norm: {tf.norm(grad).numpy()}")

# Example usage
model = create_resnet_model()
input_data = tf.random.normal((1, 224, 224, 3))
visualize_gradients(model, input_data)

Residual Blocks with Identity Mappings

Residual blocks with identity mappings allow the network to learn residual functions with reference to the layer inputs. This formulation makes it easier for the network to learn identity mappings when needed, improving overall performance.

def residual_block(x, filters, kernel_size=3, stride=1):
    shortcut = x
    
    # First convolution layer
    x = tf.keras.layers.Conv2D(filters, kernel_size, strides=stride, padding='same')(x)
    x = tf.keras.layers.BatchNormalization()(x)
    x = tf.keras.layers.Activation('relu')(x)
    
    # Second convolution layer
    x = tf.keras.layers.Conv2D(filters, kernel_size, padding='same')(x)
    x = tf.keras.layers.BatchNormalization()(x)
    
    # Adjust shortcut dimensions if needed
    if stride != 1 or shortcut.shape[-1] != filters:
        shortcut = tf.keras.layers.Conv2D(filters, 1, strides=stride, padding='same')(shortcut)
        shortcut = tf.keras.layers.BatchNormalization()(shortcut)
    
    # Add shortcut to the output
    x = tf.keras.layers.Add()([x, shortcut])
    x = tf.keras.layers.Activation('relu')(x)
    
    return x

Pre-activation vs Post-activation

In the original ResNet paper, activation functions were applied after element-wise addition. However, pre-activation (applying activation before addition) has shown improved performance in some cases. Let's compare both approaches:

def post_activation_block(x, filters):
    shortcut = x
    x = tf.keras.layers.Conv2D(filters, 3, padding='same')(x)
    x = tf.keras.layers.BatchNormalization()(x)
    x = tf.keras.layers.Add()([shortcut, x])
    x = tf.keras.layers.Activation('relu')(x)
    return x

def pre_activation_block(x, filters):
    shortcut = x
    x = tf.keras.layers.BatchNormalization()(x)
    x = tf.keras.layers.Activation('relu')(x)
    x = tf.keras.layers.Conv2D(filters, 3, padding='same')(x)
    x = tf.keras.layers.Add()([shortcut, x])
    return x

Implementing a Basic ResNet

Now let's implement a basic ResNet architecture using identity mappings. This example creates a simple ResNet-18 model for image classification:

def create_resnet18(input_shape, num_classes):
    inputs = tf.keras.Input(shape=input_shape)
    x = tf.keras.layers.Conv2D(64, 7, strides=2, padding='same')(inputs)
    x = tf.keras.layers.BatchNormalization()(x)
    x = tf.keras.layers.Activation('relu')(x)
    x = tf.keras.layers.MaxPooling2D(3, strides=2, padding='same')(x)

    # ResNet blocks
    x = residual_block(x, 64)
    x = residual_block(x, 64)
    x = residual_block(x, 128, stride=2)
    x = residual_block(x, 128)
    x = residual_block(x, 256, stride=2)
    x = residual_block(x, 256)
    x = residual_block(x, 512, stride=2)
    x = residual_block(x, 512)

    x = tf.keras.layers.GlobalAveragePooling2D()(x)
    outputs = tf.keras.layers.Dense(num_classes, activation='softmax')(x)

    model = tf.keras.Model(inputs, outputs)
    return model

# Example usage
model = create_resnet18((224, 224, 3), 1000)
model.summary()

Training a ResNet with Identity Mappings

Let's set up a training loop for our ResNet model, demonstrating how to use identity mappings in practice:

def train_resnet(model, train_data, val_data, epochs=10):
    optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)
    loss_fn = tf.keras.losses.CategoricalCrossentropy()

    train_acc_metric = tf.keras.metrics.CategoricalAccuracy()
    val_acc_metric = tf.keras.metrics.CategoricalAccuracy()

    @tf.function
    def train_step(x, y):
        with tf.GradientTape() as tape:
            logits = model(x, training=True)
            loss_value = loss_fn(y, logits)
        grads = tape.gradient(loss_value, model.trainable_weights)
        optimizer.apply_gradients(zip(grads, model.trainable_weights))
        train_acc_metric.update_state(y, logits)
        return loss_value

    for epoch in range(epochs):
        print(f"\nStart of epoch {epoch+1}")
        for step, (x_batch_train, y_batch_train) in enumerate(train_data):
            loss_value = train_step(x_batch_train, y_batch_train)
            
            if step % 200 == 0:
                print(f"Training loss (for one batch) at step {step}: {float(loss_value):.4f}")
                print(f"Seen so far: {(step+1)*64} samples")

        train_acc = train_acc_metric.result()
        print(f"Training acc over epoch: {float(train_acc):.4f}")
        train_acc_metric.reset_states()

        for x_batch_val, y_batch_val in val_data:
            val_logits = model(x_batch_val, training=False)
            val_acc_metric.update_state(y_batch_val, val_logits)
        val_acc = val_acc_metric.result()
        val_acc_metric.reset_states()
        print(f"Validation acc: {float(val_acc):.4f}")

# Usage example (assuming you have train_data and val_data)
# train_resnet(model, train_data, val_data)

Visualizing Feature Maps

To understand how identity mappings affect feature representation, let's create a function to visualize feature maps at different layers of our ResNet:

import matplotlib.pyplot as plt

def visualize_feature_maps(model, image, layer_name):
    layer_output = model.get_layer(layer_name).output
    activation_model = tf.keras.Model(inputs=model.input, outputs=layer_output)
    
    activations = activation_model.predict(image[np.newaxis, ...])
    
    fig, axes = plt.subplots(4, 8, figsize=(20, 10))
    for i, ax in enumerate(axes.flat):
        if i < activations.shape[-1]:
            ax.imshow(activations[0, :, :, i], cmap='viridis')
        ax.axis('off')
    plt.tight_layout()
    plt.show()

# Usage example
image = load_and_preprocess_image('path/to/image.jpg')
visualize_feature_maps(model, image, 'conv2_block3_out')

Gradient Flow Analysis

To demonstrate the improved gradient flow in ResNets with identity mappings, let's implement a function to visualize gradients at different layers:

import numpy as np

def analyze_gradient_flow(model, input_data):
    with tf.GradientTape() as tape:
        output = model(input_data)
        loss = tf.reduce_mean(output)
    
    gradients = tape.gradient(loss, model.trainable_variables)
    grad_norms = [np.linalg.norm(grad.numpy().flatten()) for grad in gradients]
    
    plt.figure(figsize=(10, 6))
    plt.bar(range(len(grad_norms)), grad_norms)
    plt.title('Gradient Norm per Layer')
    plt.xlabel('Layer Index')
    plt.ylabel('Gradient Norm')
    plt.yscale('log')
    plt.show()

# Usage example
input_data = np.random.rand(1, 224, 224, 3).astype(np.float32)
analyze_gradient_flow(model, input_data)

Comparing ResNet with and without Identity Mappings

Let's create a function to compare the performance of ResNet architectures with and without identity mappings:

def compare_resnet_variants(input_shape, num_classes, epochs=10):
    # ResNet with identity mappings
    resnet_identity = create_resnet18(input_shape, num_classes)
    
    # ResNet without identity mappings (traditional skip connections)
    resnet_no_identity = create_resnet18_no_identity(input_shape, num_classes)
    
    # Compile both models
    for model in [resnet_identity, resnet_no_identity]:
        model.compile(optimizer='adam',
                      loss='categorical_crossentropy',
                      metrics=['accuracy'])
    
    # Train and evaluate both models
    history_identity = resnet_identity.fit(train_data, validation_data=val_data, epochs=epochs)
    history_no_identity = resnet_no_identity.fit(train_data, validation_data=val_data, epochs=epochs)
    
    # Plot comparison
    plt.figure(figsize=(12, 5))
    plt.subplot(1, 2, 1)
    plt.plot(history_identity.history['val_accuracy'], label='With Identity')
    plt.plot(history_no_identity.history['val_accuracy'], label='Without Identity')
    plt.title('Validation Accuracy')
    plt.legend()
    
    plt.subplot(1, 2, 2)
    plt.plot(history_identity.history['val_loss'], label='With Identity')
    plt.plot(history_no_identity.history['val_loss'], label='Without Identity')
    plt.title('Validation Loss')
    plt.legend()
    
    plt.tight_layout()
    plt.show()

# Usage example
compare_resnet_variants((224, 224, 3), 1000)

Real-life Example: Image Classification

Let's apply our ResNet with identity mappings to a real-world image classification task using the CIFAR-10 dataset:

import tensorflow_datasets as tfds

# Load and preprocess CIFAR-10 dataset
def preprocess_cifar10(image, label):
    image = tf.cast(image, tf.float32) / 255.0
    return image, label

(train_ds, val_ds), ds_info = tfds.load('cifar10', split=['train[:90%]', 'train[90%:]'], as_supervised=True, with_info=True)
train_ds = train_ds.map(preprocess_cifar10).batch(64).prefetch(tf.data.AUTOTUNE)
val_ds = val_ds.map(preprocess_cifar10).batch(64).prefetch(tf.data.AUTOTUNE)

# Create and train the model
model = create_resnet18((32, 32, 3), 10)
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
history = model.fit(train_ds, validation_data=val_ds, epochs=20)

# Evaluate the model
test_ds = tfds.load('cifar10', split='test', as_supervised=True)
test_ds = test_ds.map(preprocess_cifar10).batch(64)
test_loss, test_acc = model.evaluate(test_ds)
print(f"Test accuracy: {test_acc:.4f}")

Real-life Example: Transfer Learning

Now let's use a pre-trained ResNet with identity mappings for transfer learning on a custom dataset:

import tensorflow as tf

# Load pre-trained ResNet50 with weights
base_model = tf.keras.applications.ResNet50(weights='imagenet', include_top=False, input_shape=(224, 224, 3))

# Freeze the base model
base_model.trainable = False

# Add custom layers
x = tf.keras.layers.GlobalAveragePooling2D()(base_model.output)
x = tf.keras.layers.Dense(256, activation='relu')(x)
output = tf.keras.layers.Dense(10, activation='softmax')(x)

model = tf.keras.Model(inputs=base_model.input, outputs=output)

# Compile the model
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

# Assuming you have your custom dataset prepared
# train_ds, val_ds = load_custom_dataset()

# Train the model
# history = model.fit(train_ds, validation_data=val_ds, epochs=10)

# Fine-tuning
base_model.trainable = True
model.compile(optimizer=tf.keras.optimizers.Adam(1e-5),
              loss='categorical_crossentropy',
              metrics=['accuracy'])

# Continue training
# history_fine = model.fit(train_ds, validation_data=val_ds, epochs=5)

Handling Different Input Sizes

Identity mappings in ResNets can handle different input sizes. Let's create a function to adapt our ResNet for various input dimensions:

def create_flexible_resnet(input_shape, num_classes):
    inputs = tf.keras.Input(shape=input_shape)
    
    x = tf.keras.layers.Conv2D(64, 7, strides=2, padding='same')(inputs)
    x = tf.keras.layers.BatchNormalization()(x)
    x = tf.keras.layers.Activation('relu')(x)
    x = tf.keras.layers.MaxPooling2D(3, strides=2, padding='same')(x)
    
    for filters in [64, 128, 256, 512]:
        x = residual_block(x, filters, stride=2)
        x = residual_block(x, filters)
    
    x = tf.keras.layers.GlobalAveragePooling2D()(x)
    outputs = tf.keras.layers.Dense(num_classes, activation='softmax')(x)
    
    return tf.keras.Model(inputs, outputs)

# Usage example
model_224 = create_flexible_resnet((224, 224, 3), 1000)
model_299 = create_flexible_resnet((299, 299, 3), 1000)
model_32 = create_flexible_resnet((32, 32, 3), 10)

Residual Networks for Object Detection

ResNets with identity mappings can be adapted for object detection tasks. Here's a simplified example using a ResNet backbone for feature extraction in an object detection model:

def create_object_detection_model(num_classes, backbone='resnet50'):
    base_model = tf.keras.applications.ResNet50(weights='imagenet', include_top=False, input_shape=(None, None, 3))
    
    # Feature Pyramid Network
    C3 = base_model.get_layer('conv3_block4_out').output
    C4 = base_model.get_layer('conv4_block6_out').output
    C5 = base_model.get_layer('conv5_block3_out').output
    
    P5 = tf.keras.layers.Conv2D(256, (1, 1), name='fpn_c5p5')(C5)
    P4 = tf.keras.layers.Add()([
        tf.keras.layers.UpSampling2D(size=(2, 2))(P5),
        tf.keras.layers.Conv2D(256, (1, 1), name='fpn_c4p4')(C4)
    ])
    P3 = tf.keras.layers.Add()([
        tf.keras.layers.UpSampling2D(size=(2, 2))(P4),
        tf.keras.layers.Conv2D(256, (1, 1), name='fpn_c3p3')(C3)
    ])
    
    # Object detection heads
    def detection_head(feature_map, num_anchors=9):
        cls_output = tf.keras.layers.Conv2D(num_anchors * num_classes, (3, 3), padding='same')(feature_map)
        box_output = tf.keras.layers.Conv2D(num_anchors * 4, (3, 3), padding='same')(feature_map)
        return cls_output, box_output
    
    cls_outputs = []
    box_outputs = []
    for feature in [P3, P4, P5]:
        cls_out, box_out = detection_head(feature)
        cls_outputs.append(cls_out)
        box_outputs.append(box_out)
    
    return tf.keras.Model(inputs=base_model.inputs, outputs=cls_outputs + box_outputs)

# Usage example
detection_model = create_object_detection_model(num_classes=80)

ResNet for Semantic Segmentation

ResNets with identity mappings can also be adapted for semantic segmentation tasks. Here's a simplified example of a ResNet-based U-Net architecture:

def create_unet_resnet(input_shape, num_classes):
    base_model = tf.keras.applications.ResNet50(weights='imagenet', include_top=False, input_shape=input_shape)
    
    # Encoder (ResNet)
    s1 = base_model.get_layer('conv1_relu').output
    s2 = base_model.get_layer('conv2_block3_out').output
    s3 = base_model.get_layer('conv3_block4_out').output
    s4 = base_model.get_layer('conv4_block6_out').output
    
    # Bridge
    b1 = tf.keras.layers.Conv2D(1024, (3, 3), padding='same')(s4)
    b1 = tf.keras.layers.BatchNormalization()(b1)
    b1 = tf.keras.layers.Activation('relu')(b1)
    
    # Decoder
    d1 = tf.keras.layers.UpSampling2D((2, 2))(b1)
    d1 = tf.keras.layers.concatenate([d1, s3])
    d1 = tf.keras.layers.Conv2D(512, (3, 3), padding='same')(d1)
    d1 = tf.keras.layers.BatchNormalization()(d1)
    d1 = tf.keras.layers.Activation('relu')(d1)
    
    d2 = tf.keras.layers.UpSampling2D((2, 2))(d1)
    d2 = tf.keras.layers.concatenate([d2, s2])
    d2 = tf.keras.layers.Conv2D(256, (3, 3), padding='same')(d2)
    d2 = tf.keras.layers.BatchNormalization()(d2)
    d2 = tf.keras.layers.Activation('relu')(d2)
    
    d3 = tf.keras.layers.UpSampling2D((2, 2))(d2)
    d3 = tf.keras.layers.concatenate([d3, s1])
    d3 = tf.keras.layers.Conv2D(128, (3, 3), padding='same')(d3)
    d3 = tf.keras.layers.BatchNormalization()(d3)
    d3 = tf.keras.layers.Activation('relu')(d3)
    
    outputs = tf.keras.layers.Conv2D(num_classes, (1, 1), activation='softmax')(d3)
    
    model = tf.keras.Model(inputs=base_model.inputs, outputs=outputs)
    return model

# Usage example
segmentation_model = create_unet_resnet((256, 256, 3), num_classes=21)

Additional Resources

For more information on Identity Mappings in Deep Residual Networks, consider exploring these resources:

  1. Original ResNet paper: "Deep Residual Learning for Image Recognition" by He et al. (2016) ArXiv: https://arxiv.org/abs/1512.03385
  2. Identity Mappings in Deep Residual Networks by He et al. (2016) ArXiv: https://arxiv.org/abs/1603.05027
  3. TensorFlow ResNet implementation: https://github.com/tensorflow/models/tree/master/official/vision/image_classification
  4. PyTorch ResNet implementation: https://github.com/pytorch/vision/blob/main/torchvision/models/resnet.py

These resources provide in-depth explanations and implementations of ResNets with identity mappings, offering valuable insights for further exploration and understanding of the topic.

More in CNN and computer vision

  1. Introduction to Padding, Strides, and Pooling in Deep Learning with Python9 min

    Padding, strides, and pooling are fundamental concepts in deep learning, particularly in convolutional neural networks (CNNs). These techniques help manage...

  2. Generative Adversarial Networks in Python:10 min

    Generative Adversarial Networks are a class of deep learning models consisting of two neural networks that compete against each other. The generator network...

  3. Matrix Multiplication in Deep Learning4 min

    Matrix multiplication is a fundamental operation in deep learning, crucial for processing inputs, applying weights, and propagating information through...

  4. FractalNet An Alternative to Residual Neural Networks Using Python7 min

    FractalNet is a novel architecture proposed as an alternative to residual neural networks. It is designed to address the optimization difficulties...

  5. Matrix Orientation, Hidden-state Alignment, and Knowledge Distillation in Python9 min

    Matrix orientation is a crucial concept in linear algebra and machine learning. It refers to how matrices are arranged and interpreted, affecting...

  6. Fractal Patterns in Neural Network Hyperparameter Grids12 min

    Neural networks are complex systems with numerous hyperparameters. When performing a grid search over these parameters, unexpected patterns can emerge. This...