xbe.at

Deep learning / Neural network fundamentals

Explaining Neural Network Architecture and Training

8 min read, 16 sections

Contents
  1. Introduction to Neural Networks
  2. Basic Architecture - Layers
  3. Activation Functions
  4. Forward Propagation
  5. Backpropagation
  6. Loss Functions
  7. Optimizers
  8. Batch Normalization
  9. Dropout
  10. Learning Rate and Scheduling
  11. Convolutional Neural Networks (CNNs)
  12. Recurrent Neural Networks (RNNs)
  13. Transfer Learning
  14. Real-life Example: Image Classification
  15. Real-life Example: Sentiment Analysis
  16. Additional Resources

Introduction to Neural Networks

Neural networks are computational models inspired by the human brain. They consist of interconnected nodes (neurons) organized in layers, designed to recognize patterns and solve complex problems.

import numpy as np
import matplotlib.pyplot as plt

def simple_neuron(inputs, weights, bias):
    return np.dot(inputs, weights) + bias

# Example
inputs = np.array([1, 2, 3])
weights = np.array([0.2, 0.3, 0.5])
bias = 1

output = simple_neuron(inputs, weights, bias)
print(f"Neuron output: {output}")

Basic Architecture - Layers

Neural networks typically consist of three types of layers: input layer, hidden layers, and output layer. Each layer contains neurons that process and transmit information.

import torch
import torch.nn as nn

class SimpleNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(SimpleNN, self).__init__()
        self.layer1 = nn.Linear(input_size, hidden_size)
        self.layer2 = nn.Linear(hidden_size, output_size)
    
    def forward(self, x):
        x = torch.relu(self.layer1(x))
        x = self.layer2(x)
        return x

model = SimpleNN(input_size=3, hidden_size=5, output_size=2)
print(model)

Activation Functions

Activation functions introduce non-linearity to the network, allowing it to learn complex patterns. Common functions include ReLU, Sigmoid, and Tanh.

import numpy as np
import matplotlib.pyplot as plt

def relu(x):
    return np.maximum(0, x)

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

def tanh(x):
    return np.tanh(x)

x = np.linspace(-5, 5, 100)
plt.figure(figsize=(10, 6))
plt.plot(x, relu(x), label='ReLU')
plt.plot(x, sigmoid(x), label='Sigmoid')
plt.plot(x, tanh(x), label='Tanh')
plt.legend()
plt.title('Activation Functions')
plt.grid(True)
plt.show()

Forward Propagation

Forward propagation is the process of passing input data through the network to generate predictions. It involves matrix multiplication and activation function application at each layer.

import numpy as np

def forward_propagation(X, weights, biases):
    layer1 = np.dot(X, weights[0]) + biases[0]
    activation1 = np.maximum(0, layer1)  # ReLU activation
    layer2 = np.dot(activation1, weights[1]) + biases[1]
    output = sigmoid(layer2)
    return output

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

# Example usage
X = np.array([[1, 2, 3]])
weights = [np.random.randn(3, 4), np.random.randn(4, 1)]
biases = [np.random.randn(1, 4), np.random.randn(1, 1)]

prediction = forward_propagation(X, weights, biases)
print(f"Prediction: {prediction}")

Backpropagation

Backpropagation is the algorithm used to train neural networks. It calculates gradients of the loss function with respect to the network's parameters, allowing for weight updates.

import torch
import torch.nn as nn

class SimpleNN(nn.Module):
    def __init__(self):
        super(SimpleNN, self).__init__()
        self.linear1 = nn.Linear(2, 3)
        self.linear2 = nn.Linear(3, 1)
    
    def forward(self, x):
        x = torch.relu(self.linear1(x))
        x = self.linear2(x)
        return x

model = SimpleNN()
criterion = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

# Training loop (simplified)
for epoch in range(100):
    inputs = torch.randn(10, 2)
    targets = torch.randn(10, 1)
    
    outputs = model(inputs)
    loss = criterion(outputs, targets)
    
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

print("Training complete")

Loss Functions

Loss functions measure the difference between predicted and actual outputs. They guide the network's learning process. Common loss functions include Mean Squared Error (MSE) and Cross-Entropy.

import numpy as np

def mse_loss(y_true, y_pred):
    return np.mean((y_true - y_pred)**2)

def binary_cross_entropy(y_true, y_pred):
    return -np.mean(y_true * np.log(y_pred) + (1 - y_true) * np.log(1 - y_pred))

# Example
y_true = np.array([1, 0, 1, 1])
y_pred = np.array([0.9, 0.1, 0.8, 0.95])

mse = mse_loss(y_true, y_pred)
bce = binary_cross_entropy(y_true, y_pred)

print(f"MSE Loss: {mse}")
print(f"Binary Cross-Entropy Loss: {bce}")

Optimizers

Optimizers update the network's weights based on the calculated gradients. Popular optimizers include Stochastic Gradient Descent (SGD), Adam, and RMSprop.

import torch
import torch.nn as nn
import torch.optim as optim

class SimpleModel(nn.Module):
    def __init__(self):
        super(SimpleModel, self).__init__()
        self.linear = nn.Linear(10, 1)
    
    def forward(self, x):
        return self.linear(x)

model = SimpleModel()

# Different optimizers
sgd_optimizer = optim.SGD(model.parameters(), lr=0.01)
adam_optimizer = optim.Adam(model.parameters(), lr=0.01)
rmsprop_optimizer = optim.RMSprop(model.parameters(), lr=0.01)

print("Optimizers initialized")

Batch Normalization

Batch Normalization normalizes the inputs of each layer, reducing internal covariate shift and accelerating training. It helps in training deeper networks and can act as a regularizer.

import torch
import torch.nn as nn

class BatchNormModel(nn.Module):
    def __init__(self):
        super(BatchNormModel, self).__init__()
        self.fc1 = nn.Linear(10, 20)
        self.bn1 = nn.BatchNorm1d(20)
        self.fc2 = nn.Linear(20, 1)
    
    def forward(self, x):
        x = self.fc1(x)
        x = self.bn1(x)
        x = torch.relu(x)
        x = self.fc2(x)
        return x

model = BatchNormModel()
print(model)

# Example usage
input_data = torch.randn(32, 10)  # Batch of 32 samples
output = model(input_data)
print(f"Output shape: {output.shape}")

Dropout

Dropout is a regularization technique that randomly sets a fraction of input units to 0 at each update during training, which helps prevent overfitting.

import torch
import torch.nn as nn

class DropoutModel(nn.Module):
    def __init__(self, dropout_rate=0.5):
        super(DropoutModel, self).__init__()
        self.fc1 = nn.Linear(100, 50)
        self.dropout = nn.Dropout(dropout_rate)
        self.fc2 = nn.Linear(50, 10)
    
    def forward(self, x):
        x = torch.relu(self.fc1(x))
        x = self.dropout(x)
        x = self.fc2(x)
        return x

model = DropoutModel()
print(model)

# Training mode (dropout active)
model.train()
train_input = torch.randn(10, 100)
train_output = model(train_input)

# Evaluation mode (dropout inactive)
model.eval()
eval_input = torch.randn(10, 100)
eval_output = model(eval_input)

print("Dropout applied during training, not during evaluation")

Learning Rate and Scheduling

The learning rate determines the step size during optimization. Learning rate scheduling adjusts the learning rate during training to improve convergence.

import torch
import torch.optim as optim

# Define a simple model
model = torch.nn.Linear(10, 1)

# Initialize optimizer
optimizer = optim.SGD(model.parameters(), lr=0.1)

# Learning rate scheduler
scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=30, gamma=0.1)

# Simulated training loop
for epoch in range(100):
    optimizer.step()
    scheduler.step()
    
    if epoch % 10 == 0:
        print(f"Epoch {epoch}, Learning Rate: {scheduler.get_last_lr()[0]}")

Convolutional Neural Networks (CNNs)

CNNs are specialized neural networks designed for processing grid-like data, such as images. They use convolutional layers to detect spatial features.

import torch
import torch.nn as nn

class SimpleCNN(nn.Module):
    def __init__(self):
        super(SimpleCNN, self).__init__()
        self.conv1 = nn.Conv2d(3, 16, kernel_size=3, stride=1, padding=1)
        self.pool = nn.MaxPool2d(kernel_size=2, stride=2)
        self.conv2 = nn.Conv2d(16, 32, kernel_size=3, stride=1, padding=1)
        self.fc = nn.Linear(32 * 8 * 8, 10)  # Assuming 32x32 input image
    
    def forward(self, x):
        x = self.pool(torch.relu(self.conv1(x)))
        x = self.pool(torch.relu(self.conv2(x)))
        x = x.view(-1, 32 * 8 * 8)
        x = self.fc(x)
        return x

model = SimpleCNN()
print(model)

# Example usage
input_image = torch.randn(1, 3, 32, 32)  # One 32x32 RGB image
output = model(input_image)
print(f"Output shape: {output.shape}")

Recurrent Neural Networks (RNNs)

RNNs are designed to work with sequential data by maintaining an internal state. They are commonly used in natural language processing and time series analysis.

import torch
import torch.nn as nn

class SimpleRNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(SimpleRNN, self).__init__()
        self.rnn = nn.RNN(input_size, hidden_size, batch_first=True)
        self.fc = nn.Linear(hidden_size, output_size)
    
    def forward(self, x):
        _, hidden = self.rnn(x)
        output = self.fc(hidden.squeeze(0))
        return output

model = SimpleRNN(input_size=10, hidden_size=20, output_size=5)
print(model)

# Example usage
sequence = torch.randn(1, 5, 10)  # Batch size 1, sequence length 5, input size 10
output = model(sequence)
print(f"Output shape: {output.shape}")

Transfer Learning

Transfer learning involves using pre-trained models as a starting point for a new task. It's particularly useful when working with limited data or complex architectures.

import torchvision.models as models
import torch.nn as nn

# Load a pre-trained ResNet model
resnet = models.resnet18(pretrained=True)

# Freeze the parameters
for param in resnet.parameters():
    param.requires_grad = False

# Modify the last layer for a new task (e.g., 10 classes)
num_ftrs = resnet.fc.in_features
resnet.fc = nn.Linear(num_ftrs, 10)

# Now only the last layer is trainable
print("Trainable parameters:")
for name, param in resnet.named_parameters():
    if param.requires_grad:
        print(name)

Real-life Example: Image Classification

Image classification is a common application of neural networks. Here's a simple example using a pre-trained model for classifying everyday objects.

import torch
import torchvision.transforms as transforms
from torchvision.models import resnet50
from PIL import Image

# Load pre-trained ResNet50 model
model = resnet50(pretrained=True)
model.eval()

# Prepare image
transform = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])

# Load and preprocess image (replace with your image path)
img = Image.open("example_image.jpg")
img_tensor = transform(img).unsqueeze(0)

# Make prediction
with torch.no_grad():
    output = model(img_tensor)

# Get top prediction
_, predicted_idx = torch.max(output, 1)

# Load class labels (simplified version)
class_labels = ['airplane', 'automobile', 'bird', 'cat', 'deer', 
                'dog', 'frog', 'horse', 'ship', 'truck']

print(f"Predicted class: {class_labels[predicted_idx.item()]}")

Real-life Example: Sentiment Analysis

Sentiment analysis is another common application of neural networks, particularly in natural language processing. Here's a simple example using a pre-trained model for analyzing sentiment in text.

from transformers import pipeline

# Load pre-trained sentiment analysis model
sentiment_analyzer = pipeline("sentiment-analysis")

# Example texts
texts = [
    "I love this product! It's amazing!",
    "This movie was terrible and a waste of time.",
    "The weather is nice today, but I'm feeling neutral about it."
]

# Analyze sentiment
for text in texts:
    result = sentiment_analyzer(text)[0]
    print(f"Text: {text}")
    print(f"Sentiment: {result['label']}, Score: {result['score']:.4f}\n")

Additional Resources

For further learning about neural networks and deep learning, consider exploring these resources:

  1. "Deep Learning" by Ian Goodfellow, Yoshua Bengio, and Aaron Courville ArXiv: https://arxiv.org/abs/1601.06823
  2. "Neural Networks and Deep Learning" by Michael Nielsen Online book: http://neuralnetworksanddeeplearning.com/
  3. "CS231n: Convolutional Neural Networks for Visual Recognition" Stanford University course materials: http://cs231n.stanford.edu/
  4. "Attention Is All You Need" by Vaswani et al. ArXiv: https://arxiv.org/abs/1706.03762
  5. PyTorch documentation and tutorials: https://pytorch.org/docs/stable/index.html

These resources provide in-depth explanations, practical examples, and cutting-edge research in the field of neural networks and deep learning.

More in Neural network fundamentals

  1. Exploring the Perceptron Architecture and Training6 min

    The perceptron is a fundamental building block of neural networks, representing a binary classifier that maps input features to a binary output through a...

  2. Deep Learning Essentials with Python11 min

    Deep Learning is a subset of machine learning that uses artificial neural networks to model and solve complex problems. It has revolutionized various fields...

  3. Exploring the Perceptron Binary Classification with Deep Learning8 min

    The perceptron is a fundamental building block of neural networks, implementing linear binary classification. It takes input features, applies weights, adds...

  4. Deep Dive into Neural Network Core Concepts13 min

    Neural networks are computational models inspired by the human brain. They consist of interconnected nodes (neurons) organized in layers, designed to...

  5. Feedforward Neural Network Fundamentals9 min

    In feedforward neural networks, information flows unidirectionally from input to output layers through hidden layers. Each neuron receives inputs, applies...

  6. Debunking the Myth! Hidden Layers and Neural Network Performance11 min

    The idea that increasing the number of hidden layers always improves neural network performance is a common misconception. In reality, the relationship...