Image Padding and Kernel Stride in Convolutional Neural Networks

Introduction to Image Padding and Kernel Stride

Image padding and kernel stride are crucial concepts in Convolutional Neural Networks (CNNs). They play a significant role in controlling the spatial dimensions of the output feature maps and the receptive field of the network. This presentation will explore these concepts, their implementation, and their impact on CNN architectures.

import numpy as np
import matplotlib.pyplot as plt

# Create a simple 5x5 image
image = np.array([
    [1, 1, 1, 0, 0],
    [0, 1, 1, 1, 0],
    [0, 0, 1, 1, 1],
    [0, 0, 1, 1, 0],
    [0, 1, 1, 0, 0]
])

# Display the image
plt.imshow(image, cmap='gray')
plt.title("Original 5x5 Image")
plt.show()

Image Padding

Image padding involves adding extra pixels around the edges of an input image. This technique is used to preserve the spatial dimensions of the image after convolution, allowing for deeper networks without rapid reduction in feature map size.

def pad_image(image, pad_width):
    return np.pad(image, pad_width, mode='constant', constant_values=0)

# Pad the image with a border of 1 pixel
padded_image = pad_image(image, 1)

plt.imshow(padded_image, cmap='gray')
plt.title("Padded 7x7 Image")
plt.show()

Types of Padding

There are several types of padding, including zero padding (filling with zeros), reflection padding (mirroring the edge pixels), and replication padding (ing the edge pixels). Zero padding is the most common type used in CNNs.

def pad_image_types(image, pad_width):
    zero_pad = np.pad(image, pad_width, mode='constant', constant_values=0)
    reflect_pad = np.pad(image, pad_width, mode='reflect')
    edge_pad = np.pad(image, pad_width, mode='edge')
    
    fig, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize=(15, 5))
    ax1.imshow(zero_pad, cmap='gray')
    ax1.set_title("Zero Padding")
    ax2.imshow(reflect_pad, cmap='gray')
    ax2.set_title("Reflection Padding")
    ax3.imshow(edge_pad, cmap='gray')
    ax3.set_title("Edge Padding")
    plt.show()

pad_image_types(image, 2)

Kernel Stride

Kernel stride refers to the number of pixels the convolutional filter moves at each step. A stride of 1 means the filter moves one pixel at a time, while a larger stride results in less overlap and smaller output dimensions.

def apply_convolution(image, kernel, stride):
    h, w = image.shape
    kh, kw = kernel.shape
    oh = (h - kh) // stride + 1
    ow = (w - kw) // stride + 1
    output = np.zeros((oh, ow))
    
    for i in range(0, oh):
        for j in range(0, ow):
            output[i, j] = np.sum(image[i*stride:i*stride+kh, j*stride:j*stride+kw] * kernel)
    
    return output

kernel = np.array([[1, 0, -1], [1, 0, -1], [1, 0, -1]])
stride1_output = apply_convolution(image, kernel, 1)
stride2_output = apply_convolution(image, kernel, 2)

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(10, 5))
ax1.imshow(stride1_output, cmap='gray')
ax1.set_title("Stride 1 Output")
ax2.imshow(stride2_output, cmap='gray')
ax2.set_title("Stride 2 Output")
plt.show()

Impact of Stride on Output Size

The stride affects the spatial dimensions of the output feature map. A larger stride results in a smaller output size, which can be useful for reducing computational complexity but may lead to loss of spatial information.

def calculate_output_size(input_size, kernel_size, stride, padding):
    return ((input_size + 2 * padding - kernel_size) // stride) + 1

input_sizes = range(5, 51, 5)
strides = [1, 2, 3]
kernel_size = 3
padding = 1

plt.figure(figsize=(10, 6))
for stride in strides:
    output_sizes = [calculate_output_size(size, kernel_size, stride, padding) for size in input_sizes]
    plt.plot(input_sizes, output_sizes, label=f'Stride {stride}')

plt.xlabel('Input Size')
plt.ylabel('Output Size')
plt.title('Impact of Stride on Output Size')
plt.legend()
plt.grid(True)
plt.show()

Padding and Stride in PyTorch

PyTorch, a popular deep learning framework, provides built-in functions for applying convolutions with different padding and stride settings. Let's see how to use these parameters in a PyTorch convolutional layer.

import torch
import torch.nn as nn

# Create a random 1x1x28x28 input tensor (batch_size x channels x height x width)
input_tensor = torch.randn(1, 1, 28, 28)

# Create convolutional layers with different padding and stride settings
conv_no_pad_stride1 = nn.Conv2d(1, 1, kernel_size=3, padding=0, stride=1)
conv_pad1_stride1 = nn.Conv2d(1, 1, kernel_size=3, padding=1, stride=1)
conv_pad1_stride2 = nn.Conv2d(1, 1, kernel_size=3, padding=1, stride=2)

# Apply convolutions
output_no_pad_stride1 = conv_no_pad_stride1(input_tensor)
output_pad1_stride1 = conv_pad1_stride1(input_tensor)
output_pad1_stride2 = conv_pad1_stride2(input_tensor)

print(f"Input shape: {input_tensor.shape}")
print(f"Output shape (no padding, stride 1): {output_no_pad_stride1.shape}")
print(f"Output shape (padding 1, stride 1): {output_pad1_stride1.shape}")
print(f"Output shape (padding 1, stride 2): {output_pad1_stride2.shape}")

Real-life Example: Edge Detection

Edge detection is a fundamental image processing technique often used in computer vision tasks. We can implement a simple edge detection filter using convolution with appropriate padding and stride.

import cv2
import numpy as np
import matplotlib.pyplot as plt

# Load an image
image = cv2.imread('path_to_your_image.jpg', 0)  # Load as grayscale
image = cv2.resize(image, (200, 200))  # Resize for demonstration

# Define edge detection kernels
horizontal_kernel = np.array([[-1, -2, -1],
                              [0, 0, 0],
                              [1, 2, 1]])

vertical_kernel = np.array([[-1, 0, 1],
                            [-2, 0, 2],
                            [-1, 0, 1]])

# Apply convolution
horizontal_edges = cv2.filter2D(image, -1, horizontal_kernel)
vertical_edges = cv2.filter2D(image, -1, vertical_kernel)
combined_edges = cv2.addWeighted(horizontal_edges, 0.5, vertical_edges, 0.5, 0)

# Display results
fig, (ax1, ax2, ax3, ax4) = plt.subplots(1, 4, figsize=(20, 5))
ax1.imshow(image, cmap='gray')
ax1.set_title('Original Image')
ax2.imshow(horizontal_edges, cmap='gray')
ax2.set_title('Horizontal Edges')
ax3.imshow(vertical_edges, cmap='gray')
ax3.set_title('Vertical Edges')
ax4.imshow(combined_edges, cmap='gray')
ax4.set_title('Combined Edges')
plt.show()

Dilated Convolutions

Dilated convolutions, also known as atrous convolutions, introduce another parameter called dilation rate. This allows the kernel to skip input values, effectively increasing the receptive field without increasing the number of parameters.

def dilated_convolution(image, kernel, dilation):
    h, w = image.shape
    kh, kw = kernel.shape
    dkh, dkw = (kh-1) * dilation + 1, (kw-1) * dilation + 1
    oh, ow = h - dkh + 1, w - dkw + 1
    output = np.zeros((oh, ow))
    
    for i in range(oh):
        for j in range(ow):
            for ki in range(kh):
                for kj in range(kw):
                    ii = i + ki * dilation
                    jj = j + kj * dilation
                    output[i, j] += image[ii, jj] * kernel[ki, kj]
    
    return output

# Create a larger image for better visualization
larger_image = np.random.rand(15, 15)

kernel = np.array([[1, 0, -1],
                   [2, 0, -2],
                   [1, 0, -1]])

conv_normal = dilated_convolution(larger_image, kernel, dilation=1)
conv_dilated = dilated_convolution(larger_image, kernel, dilation=2)

fig, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize=(15, 5))
ax1.imshow(larger_image, cmap='gray')
ax1.set_title('Original Image')
ax2.imshow(conv_normal, cmap='gray')
ax2.set_title('Normal Convolution')
ax3.imshow(conv_dilated, cmap='gray')
ax3.set_title('Dilated Convolution (rate=2)')
plt.show()

Transposed Convolutions

Transposed convolutions, sometimes incorrectly called deconvolutions, are used to increase the spatial dimensions of the output. They are often used in encoder-decoder architectures and generative models.

import torch
import torch.nn as nn

# Create a random 1x1x4x4 input tensor
input_tensor = torch.randn(1, 1, 4, 4)

# Create a transposed convolution layer
trans_conv = nn.ConvTranspose2d(in_channels=1, out_channels=1, kernel_size=3, stride=2, padding=1, output_padding=1)

# Apply transposed convolution
output = trans_conv(input_tensor)

print(f"Input shape: {input_tensor.shape}")
print(f"Output shape: {output.shape}")

# Visualize input and output
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(10, 5))
ax1.imshow(input_tensor.squeeze().detach().numpy(), cmap='gray')
ax1.set_title('Input')
ax2.imshow(output.squeeze().detach().numpy(), cmap='gray')
ax2.set_title('Output (Transposed Convolution)')
plt.show()

Receptive Field

The receptive field refers to the region in the input space that a particular CNN feature is looking at. Padding and stride affect the receptive field size, which is crucial for understanding what the network "sees" at each layer.

def calculate_receptive_field(num_layers, kernel_size, stride):
    receptive_field = kernel_size
    for _ in range(1, num_layers):
        receptive_field = receptive_field + (kernel_size - 1) * stride
    return receptive_field

num_layers = range(1, 6)
kernel_sizes = [3, 5, 7]
stride = 1

plt.figure(figsize=(10, 6))
for kernel_size in kernel_sizes:
    receptive_fields = [calculate_receptive_field(layers, kernel_size, stride) for layers in num_layers]
    plt.plot(num_layers, receptive_fields, marker='o', label=f'Kernel Size {kernel_size}')

plt.xlabel('Number of Layers')
plt.ylabel('Receptive Field Size')
plt.title('Growth of Receptive Field Size')
plt.legend()
plt.grid(True)
plt.show()

Padding and Stride in Real CNN Architectures

Let's examine how padding and stride are used in popular CNN architectures like VGG16 and ResNet. We'll create a simplified version of these networks to demonstrate the concept.

import torch
import torch.nn as nn

class SimplifiedVGG(nn.Module):
    def __init__(self):
        super(SimplifiedVGG, self).__init__()
        self.features = nn.Sequential(
            nn.Conv2d(3, 64, kernel_size=3, padding=1),
            nn.ReLU(),
            nn.Conv2d(64, 64, kernel_size=3, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2),
            nn.Conv2d(64, 128, kernel_size=3, padding=1),
            nn.ReLU(),
            nn.Conv2d(128, 128, kernel_size=3, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2)
        )

class SimplifiedResNet(nn.Module):
    def __init__(self):
        super(SimplifiedResNet, self).__init__()
        self.conv1 = nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3)
        self.bn1 = nn.BatchNorm2d(64)
        self.relu = nn.ReLU()
        self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)
        self.residual_block = nn.Sequential(
            nn.Conv2d(64, 64, kernel_size=3, padding=1),
            nn.BatchNorm2d(64),
            nn.ReLU(),
            nn.Conv2d(64, 64, kernel_size=3, padding=1),
            nn.BatchNorm2d(64)
        )

    def forward(self, x):
        x = self.conv1(x)
        x = self.bn1(x)
        x = self.relu(x)
        x = self.maxpool(x)
        identity = x
        x = self.residual_block(x)
        x += identity
        return self.relu(x)

# Create sample input
input_tensor = torch.randn(1, 3, 224, 224)

# Instantiate models
vgg = SimplifiedVGG()
resnet = SimplifiedResNet()

# Forward pass
vgg_output = vgg.features(input_tensor)
resnet_output = resnet(input_tensor)

print(f"VGG Input shape: {input_tensor.shape}")
print(f"VGG Output shape: {vgg_output.shape}")
print(f"ResNet Input shape: {input_tensor.shape}")
print(f"ResNet Output shape: {resnet_output.shape}")

Real-life Example: Image Segmentation

Image segmentation is a task where padding and stride play crucial roles. Let's implement a simple U-Net-like architecture for image segmentation, demonstrating the use of different padding and stride values.

import torch
import torch.nn as nn

class SimpleUNet(nn.Module):
    def __init__(self):
        super(SimpleUNet, self).__init__()
        # Encoder (downsampling)
        self.enc1 = self.conv_block(3, 64, padding=1)
        self.enc2 = self.conv_block(64, 128, padding=1)
        self.pool = nn.MaxPool2d(2, 2)
        
        # Bridge
        self.bridge = self.conv_block(128, 256, padding=1)
        
        # Decoder (upsampling)
        self.upconv1 = nn.ConvTranspose2d(256, 128, kernel_size=2, stride=2)
        self.dec1 = self.conv_block(256, 128, padding=1)
        self.upconv2 = nn.ConvTranspose2d(128, 64, kernel_size=2, stride=2)
        self.dec2 = self.conv_block(128, 64, padding=1)
        
        self.final = nn.Conv2d(64, 1, kernel_size=1)
    
    def conv_block(self, in_ch, out_ch, padding):
        return nn.Sequential(
            nn.Conv2d(in_ch, out_ch, kernel_size=3, padding=padding),
            nn.ReLU(inplace=True),
            nn.Conv2d(out_ch, out_ch, kernel_size=3, padding=padding),
            nn.ReLU(inplace=True)
        )
    
    def forward(self, x):
        # Encoding
        enc1 = self.enc1(x)
        enc2 = self.enc2(self.pool(enc1))
        
        # Bridge
        bridge = self.bridge(self.pool(enc2))
        
        # Decoding
        dec1 = self.dec1(torch.cat([self.upconv1(bridge), enc2], dim=1))
        dec2 = self.dec2(torch.cat([self.upconv2(dec1), enc1], dim=1))
        
        return self.final(dec2)

# Create a sample input
input_tensor = torch.randn(1, 3, 256, 256)

# Instantiate the model
model = SimpleUNet()

# Forward pass
output = model(input_tensor)

print(f"Input shape: {input_tensor.shape}")
print(f"Output shape: {output.shape}")

Choosing Appropriate Padding and Stride

The choice of padding and stride depends on the specific task and network architecture. Here are some general guidelines:

1. Use 'same' padding (padding that keeps the spatial dimensions constant) for deeper networks to prevent rapid reduction in feature map size.
2. Use larger strides in earlier layers to reduce spatial dimensions and computational cost.
3. In tasks requiring fine-grained spatial information (e.g., segmentation), use smaller strides and maintain spatial dimensions.
4. For classification tasks, gradual reduction of spatial dimensions is often beneficial.

def calculate_output_size(input_size, kernel_size, stride, padding):
    return ((input_size + 2 * padding - kernel_size) // stride) + 1

def print_layer_info(layer_name, input_size, kernel_size, stride, padding):
    output_size = calculate_output_size(input_size, kernel_size, stride, padding)
    print(f"{layer_name}: Input={input_size}, Output={output_size}")

# Example network configuration
input_size = 224
print_layer_info("Conv1", input_size, kernel_size=7, stride=2, padding=3)
input_size = calculate_output_size(input_size, 7, 2, 3)
print_layer_info("MaxPool", input_size, kernel_size=3, stride=2, padding=1)
input_size = calculate_output_size(input_size, 3, 2, 1)
print_layer_info("Conv2", input_size, kernel_size=3, stride=1, padding=1)
input_size = calculate_output_size(input_size, 3, 1, 1)
print_layer_info("Conv3", input_size, kernel_size=3, stride=2, padding=1)

Optimizing Padding and Stride for Performance

Proper use of padding and stride can significantly impact the performance and efficiency of CNNs:

1. Reduced spatial dimensions (larger strides) decrease computational complexity but may lose fine-grained information.
2. Maintaining spatial dimensions (appropriate padding) allows for deeper networks but increases computational cost.
3. Using padding can help preserve information at the edges of the input, which is crucial for tasks like object detection.
4. Stride can be used as an alternative to pooling layers for downsampling, potentially reducing the number of parameters.

import time

def benchmark_conv(input_size, kernel_size, stride, padding, iterations=1000):
    input_tensor = torch.randn(1, 3, input_size, input_size)
    conv_layer = nn.Conv2d(3, 64, kernel_size=kernel_size, stride=stride, padding=padding)
    
    start_time = time.time()
    for _ in range(iterations):
        _ = conv_layer(input_tensor)
    end_time = time.time()
    
    return end_time - start_time

# Benchmark different configurations
configs = [
    {"name": "No padding, stride 1", "kernel": 3, "stride": 1, "padding": 0},
    {"name": "With padding, stride 1", "kernel": 3, "stride": 1, "padding": 1},
    {"name": "No padding, stride 2", "kernel": 3, "stride": 2, "padding": 0},
    {"name": "With padding, stride 2", "kernel": 3, "stride": 2, "padding": 1},
]

for config in configs:
    time_taken = benchmark_conv(224, config["kernel"], config["stride"], config["padding"])
    print(f"{config['name']}: {time_taken:.4f} seconds")

Additional Resources

For more in-depth information on image padding and kernel stride in Convolutional Neural Networks, consider exploring these resources:

1. ArXiv paper: "A guide to convolution arithmetic for deep learning" by Vincent Dumoulin and Francesco Visin (https://arxiv.org/abs/1603.07285)
2. ArXiv paper: "Deconvolution and Checkerboard Artifacts" by Augustus Odena, Vincent Dumoulin, and Chris Olah (https://arxiv.org/abs/1611.07308)
3. Deep Learning book by Ian Goodfellow, Yoshua Bengio, and Aaron Courville, particularly Chapter 9 on Convolutional Networks (https://www.deeplearningbook.org/)

These resources provide comprehensive explanations and mathematical foundations for the concepts discussed in this presentation.