xbe.at

Deep learning / Neural network fundamentals

Neural Networks Layered Learning for Pattern Recognition

14 min read, 15 sections

Contents
  1. Neural Networks: The Power of Layered Learning
  2. Input Layer: The Gateway to Neural Networks
  3. Hidden Layers: Unveiling Patterns
  4. Output Layer: Making Predictions
  5. Activation Functions: Adding Non-linearity
  6. Backpropagation: Learning from Mistakes
  7. Loss Functions: Measuring Performance
  8. Optimization Algorithms: Finding the Right Path
  9. Regularization: Preventing Overfitting
  10. Convolutional Neural Networks: Image Processing Powerhouse
  11. Recurrent Neural Networks: Handling Sequential Data
  12. Transfer Learning: Leveraging Pre-trained Models
  13. Real-life Example: Image Classification
  14. Real-life Example: Natural Language Processing
  15. Additional Resources

Neural Networks: The Power of Layered Learning

Neural networks are computational models inspired by the human brain, designed to recognize patterns and solve complex problems. They consist of interconnected nodes (neurons) organized in layers, each contributing to the overall learning process.


class NeuralNetwork:
    def __init__(self, input_size, hidden_size, output_size):
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.output_size = output_size
        
        # Initialize weights and biases
        self.W1 = np.random.randn(self.input_size, self.hidden_size)
        self.b1 = np.zeros((1, self.hidden_size))
        self.W2 = np.random.randn(self.hidden_size, self.output_size)
        self.b2 = np.zeros((1, self.output_size))

    def forward(self, X):
        # Forward pass through the network
        self.z1 = np.dot(X, self.W1) + self.b1
        self.a1 = self.sigmoid(self.z1)
        self.z2 = np.dot(self.a1, self.W2) + self.b2
        self.a2 = self.sigmoid(self.z2)
        return self.a2

    def sigmoid(self, x):
        return 1 / (1 + np.exp(-x))

# Create a simple neural network
nn = NeuralNetwork(input_size=2, hidden_size=3, output_size=1)

Input Layer: The Gateway to Neural Networks

The input layer is the first point of contact for data entering the neural network. It receives raw information and prepares it for processing by subsequent layers. Each neuron in this layer represents a feature of the input data.


# Example: Image recognition input layer
image_size = (28, 28)  # 28x28 pixel image
input_neurons = image_size[0] * image_size[1]

# Simulate an input image (random pixel values)
input_image = np.random.rand(*image_size)

# Flatten the image for the input layer
input_data = input_image.flatten()

print(f"Input layer neurons: {input_neurons}")
print(f"Input data shape: {input_data.shape}")

# Visualize the input image
import matplotlib.pyplot as plt
plt.imshow(input_image, cmap='gray')
plt.title("Input Image")
plt.show()

Hidden Layers: Unveiling Patterns

Hidden layers are the core of neural networks, where complex patterns and features are learned. Each neuron in these layers receives inputs from the previous layer, applies weights and biases, and passes the result through an activation function.


def relu(x):
    return np.maximum(0, x)

# Example: A single hidden layer
input_size = 784  # 28x28 image
hidden_size = 128
batch_size = 32

# Initialize weights and biases
W = np.random.randn(input_size, hidden_size) * 0.01
b = np.zeros((1, hidden_size))

# Forward pass through the hidden layer
X = np.random.randn(batch_size, input_size)  # Random input batch
Z = np.dot(X, W) + b
A = relu(Z)

print(f"Hidden layer output shape: {A.shape}")
print(f"Number of activated neurons: {np.sum(A > 0)}")

# Visualize neuron activations
plt.hist(A.flatten(), bins=50)
plt.title("Distribution of Hidden Layer Activations")
plt.xlabel("Activation Value")
plt.ylabel("Frequency")
plt.show()

Output Layer: Making Predictions

The output layer is the final stage of a neural network, where the processed information is transformed into the desired output format. The number and type of neurons in this layer depend on the specific task the network is designed to perform.


def softmax(x):
    exp_x = np.exp(x - np.max(x, axis=1, keepdims=True))
    return exp_x / np.sum(exp_x, axis=1, keepdims=True)

# Example: Multi-class classification output layer
hidden_size = 128
num_classes = 10
batch_size = 32

# Initialize weights and biases
W = np.random.randn(hidden_size, num_classes) * 0.01
b = np.zeros((1, num_classes))

# Forward pass through the output layer
A_prev = np.random.randn(batch_size, hidden_size)  # Output from previous layer
Z = np.dot(A_prev, W) + b
A = softmax(Z)

print(f"Output layer shape: {A.shape}")
print(f"Sample prediction:\n{A[0]}")

# Visualize class probabilities for a sample
plt.bar(range(num_classes), A[0])
plt.title("Class Probabilities for a Sample")
plt.xlabel("Class")
plt.ylabel("Probability")
plt.show()

Activation Functions: Adding Non-linearity

Activation functions introduce non-linearity into neural networks, allowing them to learn complex patterns. Common activation functions include ReLU, Sigmoid, and Tanh. These functions determine whether and to what extent a neuron's output is passed to the next layer.

import matplotlib.pyplot as plt

def relu(x):
    return np.maximum(0, x)

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

def tanh(x):
    return np.tanh(x)

x = np.linspace(-5, 5, 100)

plt.figure(figsize=(12, 4))

plt.subplot(131)
plt.plot(x, relu(x))
plt.title('ReLU')
plt.grid(True)

plt.subplot(132)
plt.plot(x, sigmoid(x))
plt.title('Sigmoid')
plt.grid(True)

plt.subplot(133)
plt.plot(x, tanh(x))
plt.title('Tanh')
plt.grid(True)

plt.tight_layout()
plt.show()

# Example: Apply activation functions
input_data = np.array([-2, -1, 0, 1, 2])
print(f"ReLU: {relu(input_data)}")
print(f"Sigmoid: {sigmoid(input_data)}")
print(f"Tanh: {tanh(input_data)}")

Backpropagation: Learning from Mistakes

Backpropagation is the algorithm used to train neural networks. It calculates the gradient of the loss function with respect to each weight by the chain rule, iterating backwards from the output layer to the input layer. This process allows the network to adjust its weights and improve its predictions.


def sigmoid(x):
    return 1 / (1 + np.exp(-x))

def sigmoid_derivative(x):
    return x * (1 - x)

# Simple 2-layer neural network
X = np.array([[0, 0, 1], [0, 1, 1], [1, 0, 1], [1, 1, 1]])
y = np.array([[0], [1], [1], [0]])

np.random.seed(1)
weights0 = 2 * np.random.random((3, 4)) - 1
weights1 = 2 * np.random.random((4, 1)) - 1

# Training
for _ in range(60000):
    # Forward propagation
    layer0 = X
    layer1 = sigmoid(np.dot(layer0, weights0))
    layer2 = sigmoid(np.dot(layer1, weights1))
    
    # Backpropagation
    layer2_error = y - layer2
    layer2_delta = layer2_error * sigmoid_derivative(layer2)
    
    layer1_error = layer2_delta.dot(weights1.T)
    layer1_delta = layer1_error * sigmoid_derivative(layer1)
    
    # Update weights
    weights1 += layer1.T.dot(layer2_delta)
    weights0 += layer0.T.dot(layer1_delta)

print("Output after training:")
print(layer2)

Loss Functions: Measuring Performance

Loss functions quantify the difference between the predicted output and the true target. They guide the learning process by providing a measure of how well the network is performing. Common loss functions include Mean Squared Error (MSE) for regression and Cross-Entropy for classification tasks.

import matplotlib.pyplot as plt

def mse_loss(y_true, y_pred):
    return np.mean((y_true - y_pred) ** 2)

def binary_cross_entropy(y_true, y_pred):
    return -np.mean(y_true * np.log(y_pred) + (1 - y_true) * np.log(1 - y_pred))

# Generate sample data
y_true = np.array([0, 1, 1, 0, 1])
y_pred_range = np.linspace(0, 1, 100)

mse_losses = [mse_loss(y_true, np.full_like(y_true, pred)) for pred in y_pred_range]
bce_losses = [binary_cross_entropy(y_true, np.full_like(y_true, pred)) for pred in y_pred_range]

plt.figure(figsize=(10, 5))
plt.plot(y_pred_range, mse_losses, label='MSE')
plt.plot(y_pred_range, bce_losses, label='Binary Cross-Entropy')
plt.xlabel('Predicted Value')
plt.ylabel('Loss')
plt.title('Comparison of MSE and Binary Cross-Entropy Loss')
plt.legend()
plt.grid(True)
plt.show()

# Example: Calculate losses for a specific prediction
y_pred_example = np.array([0.1, 0.9, 0.8, 0.2, 0.7])
print(f"MSE Loss: {mse_loss(y_true, y_pred_example):.4f}")
print(f"Binary Cross-Entropy Loss: {binary_cross_entropy(y_true, y_pred_example):.4f}")

Optimization Algorithms: Finding the Right Path

Optimization algorithms are used to minimize the loss function and find the optimal weights for the neural network. Popular algorithms include Stochastic Gradient Descent (SGD), Adam, and RMSprop. These methods determine how the network's weights are updated during training.

import matplotlib.pyplot as plt

def sgd(params, grads, learning_rate):
    return [p - learning_rate * g for p, g in zip(params, grads)]

def adam(params, grads, m, v, t, learning_rate=0.001, beta1=0.9, beta2=0.999, epsilon=1e-8):
    t += 1
    m = [beta1 * m_i + (1 - beta1) * g for m_i, g in zip(m, grads)]
    v = [beta2 * v_i + (1 - beta2) * (g ** 2) for v_i, g in zip(v, grads)]
    m_hat = [m_i / (1 - beta1 ** t) for m_i in m]
    v_hat = [v_i / (1 - beta2 ** t) for v_i in v]
    params = [p - learning_rate * m_h / (np.sqrt(v_h) + epsilon) for p, m_h, v_h in zip(params, m_hat, v_hat)]
    return params, m, v, t

# Simulate optimization on a simple 2D function
def f(x, y):
    return x**2 + y**2

x = np.linspace(-5, 5, 100)
y = np.linspace(-5, 5, 100)
X, Y = np.meshgrid(x, y)
Z = f(X, Y)

# Initialize parameters and hyperparameters
params_sgd = params_adam = [4, 4]
learning_rate = 0.1
m = [0, 0]
v = [0, 0]
t = 0

sgd_path = [params_sgd]
adam_path = [params_adam]

for _ in range(20):
    # Compute gradients
    grads = [2 * params_sgd[0], 2 * params_sgd[1]]
    
    # Update using SGD
    params_sgd = sgd(params_sgd, grads, learning_rate)
    sgd_path.append(params_sgd)
    
    # Update using Adam
    params_adam, m, v, t = adam(params_adam, grads, m, v, t)
    adam_path.append(params_adam)

# Plot the optimization paths
plt.figure(figsize=(12, 5))

plt.subplot(121)
plt.contour(X, Y, Z, levels=20)
plt.plot(*zip(*sgd_path), 'ro-', label='SGD')
plt.title('SGD Optimization Path')
plt.legend()

plt.subplot(122)
plt.contour(X, Y, Z, levels=20)
plt.plot(*zip(*adam_path), 'bo-', label='Adam')
plt.title('Adam Optimization Path')
plt.legend()

plt.tight_layout()
plt.show()

Regularization: Preventing Overfitting

Regularization techniques help prevent overfitting in neural networks by adding constraints to the learning process. Common methods include L1/L2 regularization, dropout, and early stopping. These techniques improve the model's generalization ability and performance on unseen data.

import matplotlib.pyplot as plt

# Generate synthetic data
np.random.seed(42)
X = np.linspace(0, 10, 100).reshape(-1, 1)
y = 0.5 * X + np.sin(X) + np.random.normal(0, 0.1, X.shape)

# Define models
def linear_model(X, w, b):
    return X.dot(w) + b

def complex_model(X, w):
    return np.column_stack([X**i for i in range(len(w))]).dot(w)

# L2 regularization
def l2_regularization(w, lambda_):
    return lambda_ * np.sum(w**2)

# Fit models
def fit_model(X, y, model, regularization=None, lambda_=0):
    X_aug = np.column_stack([X**i for i in range(10)])
    w = np.linalg.inv(X_aug.T.dot(X_aug) + lambda_ * np.eye(10)).dot(X_aug.T).dot(y)
    return w

w_linear = fit_model(X, y, linear_model)
w_complex = fit_model(X, y, complex_model)
w_regularized = fit_model(X, y, complex_model, l2_regularization, lambda_=1)

# Plot results
plt.figure(figsize=(12, 4))

plt.subplot(131)
plt.scatter(X, y, alpha=0.5)
plt.plot(X, linear_model(X, *w_linear), 'r', label='Linear')
plt.title('Linear Model')
plt.legend()

plt.subplot(132)
plt.scatter(X, y, alpha=0.5)
plt.plot(X, complex_model(X, w_complex), 'g', label='Complex')
plt.title('Complex Model (Overfitting)')
plt.legend()

plt.subplot(133)
plt.scatter(X, y, alpha=0.5)
plt.plot(X, complex_model(X, w_regularized), 'b', label='Regularized')
plt.title('Regularized Complex Model')
plt.legend()

plt.tight_layout()
plt.show()

# Print model complexities
print(f"Linear model complexity: {np.sum(np.abs(w_linear) > 1e-6)}")
print(f"Complex model complexity: {np.sum(np.abs(w_complex) > 1e-6)}")
print(f"Regularized model complexity: {np.sum(np.abs(w_regularized) > 1e-6)}")

Convolutional Neural Networks: Image Processing Powerhouse

Convolutional Neural Networks (CNNs) are specialized neural networks designed for processing grid-like data, such as images. They use convolutional layers to automatically learn spatial hierarchies of features, making them highly effective for tasks like image classification and object detection.

import matplotlib.pyplot as plt
from scipy import signal

# Create a simple image
image = np.zeros((8, 8))
image[2:6, 2:6] = 1

# Define a simple edge detection kernel
kernel = np.array([[-1, -1, -1],
                   [-1,  8, -1],
                   [-1, -1, -1]])

# Perform convolution
conv_output = signal.convolve2d(image, kernel, mode='same', boundary='wrap')

# Visualize the process
fig, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize=(15, 5))

ax1.imshow(image, cmap='gray')
ax1.set_title('Original Image')
ax1.axis('off')

ax2.imshow(kernel, cmap='gray')
ax2.set_title('Convolutional Kernel')
ax2.axis('off')

ax3.imshow(conv_output, cmap='gray')
ax3.set_title('Convolution Output')
ax3.axis('off')

plt.tight_layout()
plt.show()

# Print shape information
print(f"Input shape: {image.shape}")
print(f"Kernel shape: {kernel.shape}")
print(f"Output shape: {conv_output.shape}")

Recurrent Neural Networks: Handling Sequential Data

Recurrent Neural Networks (RNNs) are designed to work with sequential data by maintaining an internal state or "memory". This makes them particularly useful for tasks involving time series, natural language processing, and other sequence-based problems.

import matplotlib.pyplot as plt

def simple_rnn_cell(x, h, W_xh, W_hh, b_h):
    return np.tanh(np.dot(x, W_xh) + np.dot(h, W_hh) + b_h)

# Generate a simple sequence
sequence = np.sin(np.linspace(0, 4*np.pi, 100)).reshape(-1, 1)

# Initialize RNN parameters
hidden_size = 16
W_xh = np.random.randn(1, hidden_size) * 0.01
W_hh = np.random.randn(hidden_size, hidden_size) * 0.01
b_h = np.zeros((1, hidden_size))

# Process the sequence
h = np.zeros((1, hidden_size))
hidden_states = []

for x in sequence:
    h = simple_rnn_cell(x, h, W_xh, W_hh, b_h)
    hidden_states.append(h.flatten())

hidden_states = np.array(hidden_states)

# Visualize the results
plt.figure(figsize=(12, 4))
plt.plot(sequence, label='Input Sequence')
plt.plot(hidden_states[:, 0], label='First Hidden Unit')
plt.plot(hidden_states[:, 1], label='Second Hidden Unit')
plt.title('RNN Processing a Simple Sequence')
plt.xlabel('Time Step')
plt.ylabel('Value')
plt.legend()
plt.grid(True)
plt.show()

print(f"Input sequence shape: {sequence.shape}")
print(f"Hidden states shape: {hidden_states.shape}")

Transfer Learning: Leveraging Pre-trained Models

Transfer learning is a technique where a model trained on one task is repurposed for a related task. This approach is particularly useful when you have limited data for your specific problem but can leverage knowledge from a model trained on a larger dataset.

import matplotlib.pyplot as plt
from sklearn.decomposition import PCA

# Simulate a pre-trained model's feature space
np.random.seed(42)
pretrained_features = np.random.randn(1000, 50)

# Simulate new task data
new_task_data = np.random.randn(100, 50) + np.array([2, 2] + [0] * 48)

# Combine data
combined_data = np.vstack([pretrained_features, new_task_data])

# Apply PCA for visualization
pca = PCA(n_components=2)
reduced_data = pca.fit_transform(combined_data)

# Visualize the feature spaces
plt.figure(figsize=(10, 6))
plt.scatter(reduced_data[:1000, 0], reduced_data[:1000, 1], alpha=0.5, label='Pre-trained Features')
plt.scatter(reduced_data[1000:, 0], reduced_data[1000:, 1], alpha=0.5, label='New Task Data')
plt.title('Transfer Learning: Feature Space Visualization')
plt.xlabel('Principal Component 1')
plt.ylabel('Principal Component 2')
plt.legend()
plt.grid(True)
plt.show()

# Simulate transfer learning process
def simple_transfer_model(pretrained_weights, new_task_data, num_epochs=100):
    weights = pretrained_weights.copy()
    for _ in range(num_epochs):
        # Simplified training process
        gradient = np.mean(new_task_data, axis=0) - weights
        weights += 0.01 * gradient
    return weights

pretrained_weights = np.mean(pretrained_features, axis=0)
transferred_weights = simple_transfer_model(pretrained_weights, new_task_data)

print("Euclidean distance between weights:")
print(f"Before transfer learning: {np.linalg.norm(pretrained_weights - np.mean(new_task_data, axis=0)):.4f}")
print(f"After transfer learning: {np.linalg.norm(transferred_weights - np.mean(new_task_data, axis=0)):.4f}")

Real-life Example: Image Classification

Image classification is a common application of neural networks, used in various fields such as medical diagnosis, autonomous vehicles, and facial recognition systems. Let's explore a simple example of classifying handwritten digits using a basic neural network.

import matplotlib.pyplot as plt
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.neural_network import MLPClassifier

# Load the digits dataset
digits = load_digits()
X, y = digits.data, digits.target

# Split the data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Standardize the features
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Create and train the neural network
mlp = MLPClassifier(hidden_layer_sizes=(50,), max_iter=1000, random_state=42)
mlp.fit(X_train_scaled, y_train)

# Evaluate the model
train_accuracy = mlp.score(X_train_scaled, y_train)
test_accuracy = mlp.score(X_test_scaled, y_test)

print(f"Training accuracy: {train_accuracy:.4f}")
print(f"Testing accuracy: {test_accuracy:.4f}")

# Visualize some predictions
fig, axes = plt.subplots(2, 5, figsize=(12, 6))
for i, ax in enumerate(axes.flat):
    ax.imshow(X_test[i].reshape(8, 8), cmap='gray')
    ax.set_title(f"True: {y_test[i]}, Pred: {mlp.predict(X_test_scaled[i].reshape(1, -1))[0]}")
    ax.axis('off')

plt.tight_layout()
plt.show()

Real-life Example: Natural Language Processing

Natural Language Processing (NLP) is another area where neural networks excel. Tasks like sentiment analysis, language translation, and text generation all benefit from neural network architectures. Let's look at a simple sentiment analysis example using a basic neural network.

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPClassifier
from sklearn.metrics import classification_report

# Sample dataset (you would typically use a larger, real-world dataset)
texts = [
    "I love this product, it's amazing!",
    "This is terrible, worst purchase ever.",
    "Neutral opinion, nothing special.",
    "Absolutely fantastic experience!",
    "Disappointing quality, would not recommend.",
    "It's okay, does the job.",
    "Incredible value for money!",
    "Waste of time and money.",
    "Average product, met expectations.",
    "Superb customer service!"
]

labels = [1, 0, 0.5, 1, 0, 0.5, 1, 0, 0.5, 1]  # 1: Positive, 0: Negative, 0.5: Neutral

# Vectorize the text data
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(texts)

# Split the data
X_train, X_test, y_train, y_test = train_test_split(X, labels, test_size=0.2, random_state=42)

# Create and train the neural network
mlp = MLPClassifier(hidden_layer_sizes=(10,), max_iter=1000, random_state=42)
mlp.fit(X_train, y_train)

# Make predictions
y_pred = mlp.predict(X_test)

# Evaluate the model
print(classification_report(y_test, y_pred, target_names=['Negative', 'Neutral', 'Positive']))

# Test on new sentences
new_texts = [
    "This product exceeded my expectations!",
    "I regret buying this, it's useless.",
    "It's neither good nor bad, just average."
]

new_X = vectorizer.transform(new_texts)
new_predictions = mlp.predict(new_X)

for text, prediction in zip(new_texts, new_predictions):
    sentiment = "Positive" if prediction == 1 else "Negative" if prediction == 0 else "Neutral"
    print(f"Text: '{text}'\nPredicted sentiment: {sentiment}\n")

Additional Resources

For those interested in diving deeper into neural networks and deep learning, here are some valuable resources:

  1. ArXiv papers:
  2. Online courses:
    • Deep Learning Specialization by Andrew Ng on Coursera
    • Fast.ai's Practical Deep Learning for Coders
  3. Textbooks:
    • "Deep Learning" by Ian Goodfellow, Yoshua Bengio, and Aaron Courville
    • "Neural Networks and Deep Learning" by Michael Nielsen (free online book)
  4. Frameworks and libraries:

These resources provide a mix of theoretical foundations and practical implementations to further your understanding of neural networks and their applications.

More in Neural network fundamentals

  1. Simple Neural Network Implementation in Python9 min

    Neural networks are computational models inspired by the human brain. They consist of interconnected nodes (neurons) that process and transmit information....

  2. Neural Networks as Universal Function Approximators8 min

    Neural networks fundamentally operate as universal function approximators, capable of learning complex mappings between inputs and outputs through...

  3. Understanding Perceptrons A Python Implementation12 min

    A perceptron is a simple artificial neuron that forms the building block of neural networks. It takes multiple inputs, applies weights to them, sums them up...

  4. Neural Networks and Deep Learning Concepts8 min

    Neural networks are the backbone of deep learning, inspired by the human brain's structure. They consist of interconnected nodes (neurons) organized in...

  5. Understanding the Role of Epochs in Training Neural Networks9 min

    A fundamental concept in deep learning, epochs represent complete iterations through the training dataset. Each epoch allows the model to see every example...

  6. Neural Network Forward Pass in Python7 min

    A forward pass in a neural network is the process of propagating input data through the network to generate an output. This fundamental operation involves a...