xbe.at

Computer vision

Mastering Convolutional Neural Networks with Python

12 min read, 15 sections

Contents
  1. Introduction to Convolutional Neural Networks (CNNs)
  2. CNN Architecture Overview
  3. Convolutional Layers
  4. Activation Functions in CNNs
  5. Pooling Layers
  6. Fully Connected Layers
  7. Data Preprocessing for CNNs
  8. Training a CNN
  9. Transfer Learning with CNNs
  10. Visualizing CNN Features
  11. Real-Life Example: Image Classification
  12. Real-Life Example: Object Detection
  13. Handling Overfitting in CNNs
  14. Interpreting CNN Decisions
  15. Additional Resources

Introduction to Convolutional Neural Networks (CNNs)

Convolutional Neural Networks are a class of deep learning models designed to process grid-like data, such as images. They're particularly effective for tasks like image classification, object detection, and facial recognition. CNNs use specialized layers that apply convolution operations to extract features from input data.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import load_sample_image

# Load a sample image
china = load_sample_image("china.jpg")
plt.imshow(china)
plt.axis('off')
plt.title("Sample Image for CNN Processing")
plt.show()

CNN Architecture Overview

A typical CNN architecture consists of several key components: convolutional layers, pooling layers, and fully connected layers. The convolutional layers apply filters to the input, pooling layers reduce spatial dimensions, and fully connected layers make the final predictions.

import tensorflow as tf

model = tf.keras.Sequential([
    tf.keras.layers.Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1)),
    tf.keras.layers.MaxPooling2D((2, 2)),
    tf.keras.layers.Conv2D(64, (3, 3), activation='relu'),
    tf.keras.layers.MaxPooling2D((2, 2)),
    tf.keras.layers.Conv2D(64, (3, 3), activation='relu'),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dense(10, activation='softmax')
])

model.summary()

Convolutional Layers

Convolutional layers are the core building blocks of CNNs. They apply a set of learnable filters to the input, creating feature maps that highlight important characteristics of the data. Each filter slides across the input, performing element-wise multiplication and summing the results.

import numpy as np
import matplotlib.pyplot as plt

# Create a simple 5x5 image
image = np.array([
    [0, 0, 0, 0, 0],
    [0, 1, 1, 1, 0],
    [0, 1, 1, 1, 0],
    [0, 1, 1, 1, 0],
    [0, 0, 0, 0, 0]
])

# Define a 3x3 filter
filter = np.array([
    [1, 0, -1],
    [1, 0, -1],
    [1, 0, -1]
])

# Perform convolution
output = np.zeros((3, 3))
for i in range(3):
    for j in range(3):
        output[i, j] = np.sum(image[i:i+3, j:j+3] * filter)

# Visualize the results
fig, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize=(15, 5))
ax1.imshow(image, cmap='gray')
ax1.set_title('Input Image')
ax2.imshow(filter, cmap='gray')
ax2.set_title('Filter')
ax3.imshow(output, cmap='gray')
ax3.set_title('Output Feature Map')
plt.show()

Activation Functions in CNNs

Activation functions introduce non-linearity into the network, allowing it to learn complex patterns. The Rectified Linear Unit (ReLU) is a popular choice for CNNs due to its simplicity and effectiveness in mitigating the vanishing gradient problem.

import numpy as np
import matplotlib.pyplot as plt

def relu(x):
    return np.maximum(0, x)

def leaky_relu(x, alpha=0.01):
    return np.where(x > 0, x, alpha * x)

x = np.linspace(-10, 10, 100)

plt.figure(figsize=(12, 4))
plt.plot(x, relu(x), label='ReLU')
plt.plot(x, leaky_relu(x), label='Leaky ReLU')
plt.title('ReLU and Leaky ReLU Activation Functions')
plt.xlabel('Input')
plt.ylabel('Output')
plt.legend()
plt.grid(True)
plt.show()

Pooling Layers

Pooling layers reduce the spatial dimensions of the feature maps, decreasing computational complexity and helping to achieve spatial invariance. Max pooling is commonly used, which takes the maximum value in each pooling window.

import numpy as np
import matplotlib.pyplot as plt

def max_pooling(input_array, pool_size):
    input_height, input_width = input_array.shape
    pool_height, pool_width = pool_size
    
    output_height = input_height // pool_height
    output_width = input_width // pool_width
    
    output = np.zeros((output_height, output_width))
    
    for i in range(output_height):
        for j in range(output_width):
            output[i, j] = np.max(input_array[i*pool_height:(i+1)*pool_height, 
                                              j*pool_width:(j+1)*pool_width])
    
    return output

# Create a sample 4x4 input
input_array = np.array([
    [1, 2, 3, 4],
    [5, 6, 7, 8],
    [9, 10, 11, 12],
    [13, 14, 15, 16]
])

# Apply max pooling with a 2x2 window
pooled = max_pooling(input_array, (2, 2))

# Visualize the results
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(10, 5))
ax1.imshow(input_array, cmap='viridis')
ax1.set_title('Input Array')
ax2.imshow(pooled, cmap='viridis')
ax2.set_title('After Max Pooling (2x2)')
plt.show()

Fully Connected Layers

Fully connected layers come after the convolutional and pooling layers. They take the flattened output from the previous layers and perform the final classification or regression task. These layers learn global patterns in the feature space.

import tensorflow as tf

# Create a simple CNN model
model = tf.keras.Sequential([
    tf.keras.layers.Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1)),
    tf.keras.layers.MaxPooling2D((2, 2)),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dense(10, activation='softmax')
])

# Visualize the model architecture
tf.keras.utils.plot_model(model, show_shapes=True, show_layer_names=True)

Data Preprocessing for CNNs

Proper data preprocessing is crucial for effective CNN training. This includes resizing images, normalizing pixel values, and data augmentation to increase the diversity of the training set.

import tensorflow as tf
from tensorflow.keras.preprocessing.image import ImageDataGenerator

# Create an ImageDataGenerator for data augmentation
datagen = ImageDataGenerator(
    rotation_range=20,
    width_shift_range=0.2,
    height_shift_range=0.2,
    horizontal_flip=True,
    zoom_range=0.2,
    rescale=1./255
)

# Load and preprocess a sample image
img = tf.keras.preprocessing.image.load_img('sample_image.jpg', target_size=(150, 150))
x = tf.keras.preprocessing.image.img_to_array(img)
x = x.reshape((1,) + x.shape)

# Generate augmented images
i = 0
for batch in datagen.flow(x, batch_size=1):
    plt.figure()
    imgplot = plt.imshow(tf.keras.preprocessing.image.array_to_img(batch[0]))
    plt.axis('off')
    i += 1
    if i % 5 == 0:
        break

plt.show()

Training a CNN

Training a CNN involves forward propagation, loss calculation, backpropagation, and parameter updates. We use optimization algorithms like Stochastic Gradient Descent (SGD) or Adam to minimize the loss function.

import tensorflow as tf
from tensorflow.keras.datasets import mnist

# Load and preprocess the MNIST dataset
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train = x_train.reshape(-1, 28, 28, 1).astype('float32') / 255
x_test = x_test.reshape(-1, 28, 28, 1).astype('float32') / 255

# Create a simple CNN model
model = tf.keras.Sequential([
    tf.keras.layers.Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1)),
    tf.keras.layers.MaxPooling2D((2, 2)),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dense(10, activation='softmax')
])

# Compile and train the model
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
history = model.fit(x_train, y_train, epochs=5, validation_split=0.2)

# Plot the training history
plt.figure(figsize=(12, 4))
plt.subplot(1, 2, 1)
plt.plot(history.history['accuracy'], label='Training Accuracy')
plt.plot(history.history['val_accuracy'], label='Validation Accuracy')
plt.title('Model Accuracy')
plt.xlabel('Epoch')
plt.ylabel('Accuracy')
plt.legend()

plt.subplot(1, 2, 2)
plt.plot(history.history['loss'], label='Training Loss')
plt.plot(history.history['val_loss'], label='Validation Loss')
plt.title('Model Loss')
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.legend()

plt.tight_layout()
plt.show()

Transfer Learning with CNNs

Transfer learning allows us to leverage pre-trained models on large datasets to improve performance on smaller, related tasks. We can use popular architectures like VGG, ResNet, or Inception as feature extractors or fine-tune them for specific tasks.

import tensorflow as tf

# Load a pre-trained VGG16 model
base_model = tf.keras.applications.VGG16(weights='imagenet', include_top=False, input_shape=(224, 224, 3))

# Freeze the base model layers
for layer in base_model.layers:
    layer.trainable = False

# Add custom layers for our specific task
model = tf.keras.Sequential([
    base_model,
    tf.keras.layers.GlobalAveragePooling2D(),
    tf.keras.layers.Dense(256, activation='relu'),
    tf.keras.layers.Dense(10, activation='softmax')
])

# Compile the model
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

# Display the model summary
model.summary()

Visualizing CNN Features

Visualizing the features learned by CNNs helps us understand what the network is focusing on. We can use techniques like activation maximization or gradient-based methods to generate images that maximize the activation of specific neurons.

import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt

def visualize_filters(model, layer_name, filter_index):
    layer = model.get_layer(layer_name)
    
    # Create a model that maps the input to the activations of the target layer
    feature_extractor = tf.keras.Model(inputs=model.inputs, outputs=layer.output)
    
    # Start from a gray image with some noise
    input_img_data = np.random.random((1, 224, 224, 3)) * 20 + 128.
    
    # Define the loss as the mean activation of a specific filter
    loss = tf.reduce_mean(feature_extractor(input_img_data)[:, :, :, filter_index])
    
    # Compute the gradient of the input picture with respect to this loss
    grads = tf.GradientTape().gradient(loss, input_img_data)
    
    # Normalization trick: we normalize the gradient
    grads /= (tf.sqrt(tf.reduce_mean(tf.square(grads))) + 1e-5)
    
    # Perform gradient ascent
    input_img_data += grads * 20
    
    img = input_img_data[0].astype(np.uint8)
    return img

# Assuming we have a pre-trained VGG16 model
model = tf.keras.applications.VGG16(weights='imagenet', include_top=False)

# Visualize the first 4 filters of the first convolutional layer
fig, axes = plt.subplots(2, 2, figsize=(10, 10))
for i, ax in enumerate(axes.flat):
    img = visualize_filters(model, 'block1_conv1', i)
    ax.imshow(img)
    ax.axis('off')
    ax.set_title(f'Filter {i}')

plt.tight_layout()
plt.show()

Real-Life Example: Image Classification

Image classification is a common application of CNNs. Let's use a pre-trained MobileNetV2 model to classify images from the popular CIFAR-10 dataset, which contains 60,000 32x32 color images in 10 classes.

import tensorflow as tf
from tensorflow.keras.applications import MobileNetV2
from tensorflow.keras.datasets import cifar10
import numpy as np
import matplotlib.pyplot as plt

# Load CIFAR-10 dataset
(x_train, y_train), (x_test, y_test) = cifar10.load_data()

# Preprocess the data
x_test = tf.keras.applications.mobilenet_v2.preprocess_input(x_test)

# Load pre-trained MobileNetV2 model
base_model = MobileNetV2(weights='imagenet', include_top=False, input_shape=(32, 32, 3))
model = tf.keras.Sequential([
    base_model,
    tf.keras.layers.GlobalAveragePooling2D(),
    tf.keras.layers.Dense(10, activation='softmax')
])

# Compile the model
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

# Make predictions
predictions = model.predict(x_test)

# Display some predictions
class_names = ['airplane', 'automobile', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck']

plt.figure(figsize=(10, 10))
for i in range(9):
    plt.subplot(3, 3, i+1)
    plt.imshow(x_test[i])
    plt.title(f"Actual: {class_names[y_test[i][0]]}\nPredicted: {class_names[np.argmax(predictions[i])]}")
    plt.axis('off')
plt.tight_layout()
plt.show()

Real-Life Example: Object Detection

Object detection is another powerful application of CNNs. We'll use a pre-trained YOLO (You Only Look Once) model to detect objects in an image. YOLO is known for its speed and accuracy in real-time object detection.

import cv2
import numpy as np
import matplotlib.pyplot as plt

# Load YOLO
net = cv2.dnn.readNet("yolov3.weights", "yolov3.cfg")
with open("coco.names", "r") as f:
    classes = [line.strip() for line in f.readlines()]

# Load image
image = cv2.imread("sample_image.jpg")
height, width = image.shape[:2]

# Prepare image for YOLO
blob = cv2.dnn.blobFromImage(image, 1/255.0, (416, 416), swapRB=True, crop=False)
net.setInput(blob)

# Get output layer names
output_layers = net.getUnconnectedOutLayersNames()

# Forward pass
outs = net.forward(output_layers)

# Process detections
class_ids = []
confidences = []
boxes = []

for out in outs:
    for detection in out:
        scores = detection[5:]
        class_id = np.argmax(scores)
        confidence = scores[class_id]
        if confidence > 0.5:
            center_x = int(detection[0] * width)
            center_y = int(detection[1] * height)
            w = int(detection[2] * width)
            h = int(detection[3] * height)
            x = int(center_x - w / 2)
            y = int(center_y - h / 2)
            boxes.append([x, y, w, h])
            confidences.append(float(confidence))
            class_ids.append(class_id)

# Apply non-maximum suppression
indices = cv2.dnn.NMSBoxes(boxes, confidences, 0.5, 0.4)

# Draw bounding boxes
for i in indices:
    box = boxes[i]
    x, y, w, h = box
    label = str(classes[class_ids[i]])
    cv2.rectangle(image, (x, y), (x + w, y + h), (0, 255, 0), 2)
    cv2.putText(image, f"{label} {confidences[i]:.2f}", (x, y - 10), 
                cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 2)

# Display the result
plt.figure(figsize=(12, 8))
plt.imshow(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))
plt.axis('off')
plt.title("Object Detection with YOLO")
plt.show()

Handling Overfitting in CNNs

Overfitting occurs when a model performs well on training data but poorly on unseen data. To combat this, we can use techniques like data augmentation, dropout, and regularization.

import tensorflow as tf
from tensorflow.keras.layers import Conv2D, MaxPooling2D, Dense, Flatten, Dropout
from tensorflow.keras.regularizers import l2

model = tf.keras.Sequential([
    Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1), 
           kernel_regularizer=l2(0.01)),
    MaxPooling2D((2, 2)),
    Conv2D(64, (3, 3), activation='relu', kernel_regularizer=l2(0.01)),
    MaxPooling2D((2, 2)),
    Flatten(),
    Dense(64, activation='relu', kernel_regularizer=l2(0.01)),
    Dropout(0.5),
    Dense(10, activation='softmax')
])

model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

# Data augmentation
datagen = tf.keras.preprocessing.image.ImageDataGenerator(
    rotation_range=10,
    width_shift_range=0.1,
    height_shift_range=0.1,
    zoom_range=0.1,
    horizontal_flip=True
)

# Train the model with data augmentation
history = model.fit(datagen.flow(x_train, y_train, batch_size=32),
                    epochs=10, validation_data=(x_test, y_test))

# Plot training history
plt.figure(figsize=(12, 4))
plt.subplot(1, 2, 1)
plt.plot(history.history['accuracy'], label='Training Accuracy')
plt.plot(history.history['val_accuracy'], label='Validation Accuracy')
plt.title('Model Accuracy')
plt.legend()

plt.subplot(1, 2, 2)
plt.plot(history.history['loss'], label='Training Loss')
plt.plot(history.history['val_loss'], label='Validation Loss')
plt.title('Model Loss')
plt.legend()

plt.show()

Interpreting CNN Decisions

Understanding why a CNN makes certain decisions is crucial for building trust in the model and debugging issues. Techniques like Grad-CAM (Gradient-weighted Class Activation Mapping) can help visualize which parts of an image are important for the model's decision.

import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt

def make_gradcam_heatmap(img_array, model, last_conv_layer_name, pred_index=None):
    grad_model = tf.keras.models.Model(
        [model.inputs], [model.get_layer(last_conv_layer_name).output, model.output]
    )

    with tf.GradientTape() as tape:
        last_conv_layer_output, preds = grad_model(img_array)
        if pred_index is None:
            pred_index = tf.argmax(preds[0])
        class_channel = preds[:, pred_index]

    grads = tape.gradient(class_channel, last_conv_layer_output)
    pooled_grads = tf.reduce_mean(grads, axis=(0, 1, 2))

    last_conv_layer_output = last_conv_layer_output[0]
    heatmap = last_conv_layer_output @ pooled_grads[..., tf.newaxis]
    heatmap = tf.squeeze(heatmap)

    heatmap = tf.maximum(heatmap, 0) / tf.math.reduce_max(heatmap)
    return heatmap.numpy()

# Assume we have a pre-trained model and an input image
model = tf.keras.applications.MobileNetV2(weights='imagenet')
img = tf.keras.preprocessing.image.load_img('elephant.jpg', target_size=(224, 224))
img_array = tf.keras.preprocessing.image.img_to_array(img)
img_array = tf.expand_dims(img_array, 0)
img_array = tf.keras.applications.mobilenet_v2.preprocess_input(img_array)

heatmap = make_gradcam_heatmap(img_array, model, 'Conv_1')

plt.matshow(heatmap)
plt.title("Grad-CAM Heatmap")
plt.show()

# Superimpose heatmap on original image
img = tf.keras.preprocessing.image.img_to_array(img)
heatmap = np.uint8(255 * heatmap)
jet = plt.cm.get_cmap("jet")
jet_colors = jet(np.arange(256))[:, :3]
jet_heatmap = jet_colors[heatmap]
jet_heatmap = tf.keras.preprocessing.image.array_to_img(jet_heatmap)
jet_heatmap = jet_heatmap.resize((img.shape[1], img.shape[0]))
jet_heatmap = tf.keras.preprocessing.image.img_to_array(jet_heatmap)

superimposed_img = jet_heatmap * 0.4 + img
superimposed_img = tf.keras.preprocessing.image.array_to_img(superimposed_img)

plt.imshow(superimposed_img)
plt.title("Grad-CAM Result")
plt.axis('off')
plt.show()

Additional Resources

For further exploration of Convolutional Neural Networks, consider the following resources:

  1. "Deep Learning" by Ian Goodfellow, Yoshua Bengio, and Aaron Courville (Available online: http://www.deeplearningbook.org/)
  2. CS231n: Convolutional Neural Networks for Visual Recognition (Stanford University course materials: http://cs231n.stanford.edu/)
  3. ArXiv paper: "A Survey of the Recent Architectures of Deep Convolutional Neural Networks" by Khan et al. (2020) (ArXiv:1901.06032)
  4. ArXiv paper: "Visualizing and Understanding Convolutional Networks" by Zeiler and Fergus (2013) (ArXiv:1311.2901)
  5. TensorFlow and Keras documentation for implementing CNNs (https://www.tensorflow.org/tutorials/images/cnn)

These resources provide a mix of theoretical foundations and practical implementations to deepen your understanding of CNNs.

More in Computer vision

  1. Mastering YOLO Object Detection with Python12 min

    YOLO (You Only Look Once) is a state-of-the-art, real-time object detection system that revolutionized computer vision. It treats object detection as a...

  2. Machine Learning Convolutional Neural Network (CNN) and Transfer Learning using Python4 min

    This outline covers the key concepts and techniques related to Convolutional Neural Networks (CNNs) and Transfer Learning using Python. Each slide includes a...

  3. Object-based Image Classification with Python6 min

    Introduction to Object-based Image Classification

  4. Leveraging Deep Depth Prediction for Monocular Direct Sparse Odometry:12 min

    Monocular Direct Sparse Odometry (DSO) is a visual odometry method that estimates camera motion and scene structure from a single camera. It's a key...

  5. Padding Strides and Pooling in Convolutional Neural Networks9 min

    Convolutional Neural Networks (CNNs) are a class of deep learning models particularly effective for image processing tasks. In CNNs, the size of the output...

  6. Introduction to Convolutional Neural Networks in Python10 min

    Convolutional Neural Networks (CNNs) are a class of deep learning models primarily used for processing grid-like data, such as images. They are designed to...