xbe.at

Deep learning / Neural network fundamentals

Comprehensive Overview of Deep Learning

9 min read, 20 sections

Contents
  1. What is Deep Learning?
  2. Source Code for What is Deep Learning?
  3. Results for Source Code for What is Deep Learning?
  4. How Deep Learning Relates to Neural Networks
  5. Source Code for How Deep Learning Relates to Neural Networks
  6. Results for Source Code for How Deep Learning Relates to Neural Networks
  7. How a Deep Learning Model Works
  8. Source Code for How a Deep Learning Model Works
  9. Results for Source Code for How a Deep Learning Model Works
  10. Key Deep Learning Models
  11. Source Code for Key Deep Learning Models
  12. Results for Source Code for Key Deep Learning Models
  13. Applications of Deep Learning
  14. Source Code for Applications of Deep Learning
  15. Results for Source Code for Applications of Deep Learning
  16. History of Deep Learning
  17. Factors Leading to the Deep Learning Revolution
  18. Source Code for Factors Leading to the Deep Learning Revolution
  19. Results for Source Code for Factors Leading to the Deep Learning Revolution
  20. Additional Resources

What is Deep Learning?

Deep Learning is a subset of Machine Learning that uses artificial neural networks with multiple layers to learn and extract features from data. It's capable of learning hierarchical representations, allowing it to process complex patterns in data such as images, text, and sound.

Source Code for What is Deep Learning?

import random

class Neuron:
    def __init__(self, inputs):
        self.weights = [random.random() for _ in range(inputs)]
        self.bias = random.random()

    def activate(self, inputs):
        return sum(w * i for w, i in zip(self.weights, inputs)) + self.bias > 0

class DeepNeuralNetwork:
    def __init__(self, layer_sizes):
        self.layers = [[Neuron(inputs) for _ in range(neurons)] 
                       for inputs, neurons in zip(layer_sizes, layer_sizes[1:])]

    def forward(self, inputs):
        for layer in self.layers:
            inputs = [neuron.activate(inputs) for neuron in layer]
        return inputs

# Example usage
nn = DeepNeuralNetwork([3, 4, 2, 1])
input_data = [0.5, 0.3, 0.7]
output = nn.forward(input_data)
print(f"Input: {input_data}")
print(f"Output: {output}")

Results for Source Code for What is Deep Learning?

Input: [0.5, 0.3, 0.7]
Output: [True]

How Deep Learning Relates to Neural Networks

Deep Learning is built upon the foundation of artificial neural networks. While traditional neural networks might have only one or two hidden layers, deep learning models typically have many more, allowing them to learn more complex representations of data.

Source Code for How Deep Learning Relates to Neural Networks

import math

def sigmoid(x):
    return 1 / (1 + math.exp(-x))

class DeepNeuralNetwork:
    def __init__(self, layer_sizes):
        self.layers = []
        for i in range(1, len(layer_sizes)):
            layer = [[random.random() for _ in range(layer_sizes[i-1])] for _ in range(layer_sizes[i])]
            self.layers.append(layer)

    def forward(self, inputs):
        for layer in self.layers:
            new_inputs = []
            for neuron in layer:
                activation = sum(w * i for w, i in zip(neuron, inputs))
                new_inputs.append(sigmoid(activation))
            inputs = new_inputs
        return inputs

# Example usage
shallow_nn = DeepNeuralNetwork([3, 2, 1])
deep_nn = DeepNeuralNetwork([3, 5, 4, 3, 2, 1])

input_data = [0.5, 0.3, 0.7]
shallow_output = shallow_nn.forward(input_data)
deep_output = deep_nn.forward(input_data)

print(f"Shallow NN output: {shallow_output}")
print(f"Deep NN output: {deep_output}")

Results for Source Code for How Deep Learning Relates to Neural Networks

Shallow NN output: [0.5793458318772803]
Deep NN output: [0.5180632983426979]

How a Deep Learning Model Works

A deep learning model processes data through multiple layers of interconnected nodes. Each layer learns to recognize different features of the input data, with deeper layers capturing more abstract and complex patterns. The model adjusts its internal parameters through a process called backpropagation to minimize the difference between its predictions and the actual outcomes.

Source Code for How a Deep Learning Model Works

import math
import random

def sigmoid(x):
    return 1 / (1 + math.exp(-x))

def sigmoid_derivative(x):
    return x * (1 - x)

class DeepNeuralNetwork:
    def __init__(self, layers):
        self.layers = layers
        self.weights = []
        self.biases = []
        for i in range(1, len(layers)):
            self.weights.append([[random.random() for _ in range(layers[i-1])] for _ in range(layers[i])])
            self.biases.append([random.random() for _ in range(layers[i])])

    def forward(self, inputs):
        self.activations = [inputs]
        for i in range(len(self.weights)):
            layer_activations = []
            for j in range(len(self.weights[i])):
                neuron = sum(w * a for w, a in zip(self.weights[i][j], self.activations[-1])) + self.biases[i][j]
                layer_activations.append(sigmoid(neuron))
            self.activations.append(layer_activations)
        return self.activations[-1]

    def backward(self, target, learning_rate):
        error = [t - a for t, a in zip(target, self.activations[-1])]
        for i in reversed(range(len(self.weights))):
            delta = [error[j] * sigmoid_derivative(self.activations[i+1][j]) for j in range(len(error))]
            
            # Update weights and biases
            for j in range(len(self.weights[i])):
                for k in range(len(self.weights[i][j])):
                    self.weights[i][j][k] += learning_rate * delta[j] * self.activations[i][k]
                self.biases[i][j] += learning_rate * delta[j]
            
            # Calculate error for next layer
            if i > 0:
                error = [sum(self.weights[i][j][k] * delta[j] for j in range(len(delta))) 
                         for k in range(len(self.weights[i][0]))]

# Example usage
nn = DeepNeuralNetwork([2, 3, 1])
input_data = [0.5, 0.7]
target = [0.8]

for _ in range(1000):
    output = nn.forward(input_data)
    nn.backward(target, 0.1)

print(f"Input: {input_data}")
print(f"Target: {target}")
print(f"Prediction: {nn.forward(input_data)}")

Results for Source Code for How a Deep Learning Model Works

Input: [0.5, 0.7]
Target: [0.8]
Prediction: [0.7992458526694437]

Key Deep Learning Models

There are several key deep learning models, each designed for specific tasks. Convolutional Neural Networks (CNNs) excel in image processing, Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) networks are suited for sequential data like text or time series, and Transformers have revolutionized natural language processing tasks.

Source Code for Key Deep Learning Models

import math
import random

def sigmoid(x):
    return 1 / (1 + math.exp(-x))

class CNN:
    def __init__(self, input_size, kernel_size, num_filters):
        self.kernel_size = kernel_size
        self.num_filters = num_filters
        self.filters = [[[random.random() for _ in range(kernel_size)] 
                         for _ in range(kernel_size)] 
                        for _ in range(num_filters)]

    def convolve(self, input_data):
        output = []
        for f in self.filters:
            conv = [[sum(input_data[i+x][j+y] * f[x][y] 
                         for x in range(self.kernel_size) 
                         for y in range(self.kernel_size))
                     for j in range(len(input_data[0]) - self.kernel_size + 1)]
                    for i in range(len(input_data) - self.kernel_size + 1)]
            output.append(conv)
        return output

class RNN:
    def __init__(self, input_size, hidden_size, output_size):
        self.hidden_size = hidden_size
        self.Wxh = [[random.random() for _ in range(input_size)] for _ in range(hidden_size)]
        self.Whh = [[random.random() for _ in range(hidden_size)] for _ in range(hidden_size)]
        self.Why = [[random.random() for _ in range(hidden_size)] for _ in range(output_size)]

    def forward(self, inputs):
        h = [0] * self.hidden_size
        outputs = []
        for x in inputs:
            h = [sigmoid(sum(self.Wxh[i][j] * x[j] for j in range(len(x))) + 
                         sum(self.Whh[i][j] * h[j] for j in range(self.hidden_size)))
                 for i in range(self.hidden_size)]
            y = [sigmoid(sum(self.Why[i][j] * h[j] for j in range(self.hidden_size)))
                 for i in range(len(self.Why))]
            outputs.append(y)
        return outputs

# Example usage
cnn = CNN(input_size=5, kernel_size=3, num_filters=2)
rnn = RNN(input_size=3, hidden_size=4, output_size=2)

# CNN example
image = [[random.random() for _ in range(5)] for _ in range(5)]
cnn_output = cnn.convolve(image)

# RNN example
sequence = [[random.random() for _ in range(3)] for _ in range(4)]
rnn_output = rnn.forward(sequence)

print("CNN output shape:", len(cnn_output), "x", len(cnn_output[0]), "x", len(cnn_output[0][0]))
print("RNN output shape:", len(rnn_output), "x", len(rnn_output[0]))

Results for Source Code for Key Deep Learning Models

CNN output shape: 2 x 3 x 3
RNN output shape: 4 x 2

Applications of Deep Learning

Deep Learning has found applications in various fields. In computer vision, it's used for image classification, object detection, and facial recognition. In natural language processing, it powers machine translation, sentiment analysis, and chatbots. Other applications include speech recognition, autonomous vehicles, drug discovery, and game playing AI.

Source Code for Applications of Deep Learning

import random
import math

def sigmoid(x):
    return 1 / (1 + math.exp(-x))

class ImageClassifier:
    def __init__(self, input_size, num_classes):
        self.weights = [[random.random() for _ in range(input_size)] for _ in range(num_classes)]
        self.biases = [random.random() for _ in range(num_classes)]

    def classify(self, image):
        scores = [sum(w * p for w, p in zip(weights, image)) + bias 
                  for weights, bias in zip(self.weights, self.biases)]
        return scores.index(max(scores))

class TextClassifier:
    def __init__(self, vocab_size, num_classes):
        self.weights = [[random.random() for _ in range(vocab_size)] for _ in range(num_classes)]
        self.biases = [random.random() for _ in range(num_classes)]

    def classify(self, text):
        # Assuming text is a list of word indices
        text_vector = [0] * len(self.weights[0])
        for word_idx in text:
            text_vector[word_idx] += 1
        scores = [sigmoid(sum(w * f for w, f in zip(weights, text_vector)) + bias) 
                  for weights, bias in zip(self.weights, self.biases)]
        return scores.index(max(scores))

# Example usage
image_classifier = ImageClassifier(input_size=784, num_classes=10)  # For 28x28 grayscale images
text_classifier = TextClassifier(vocab_size=10000, num_classes=5)  # For sentiment analysis

# Simulate image classification
random_image = [random.random() for _ in range(784)]
image_class = image_classifier.classify(random_image)
print(f"Predicted image class: {image_class}")

# Simulate text classification
random_text = [random.randint(0, 9999) for _ in range(20)]  # 20 random words
sentiment = text_classifier.classify(random_text)
print(f"Predicted sentiment: {sentiment}")

Results for Source Code for Applications of Deep Learning

Predicted image class: 7
Predicted sentiment: 2

History of Deep Learning

The concept of deep learning dates back to the 1940s with the introduction of artificial neurons. However, it wasn't until the 1980s that the backpropagation algorithm was developed, allowing for efficient training of neural networks. The true deep learning revolution began in the 2000s with increased computational power and the availability of large datasets.

Factors Leading to the Deep Learning Revolution

The deep learning revolution was fueled by several factors: the availability of big data, advancements in hardware (particularly GPUs), development of better training algorithms, and open-source software frameworks. These factors combined to make it possible to train larger and more complex models, leading to breakthrough performances in various tasks.

Source Code for Factors Leading to the Deep Learning Revolution

import random
import time

def sigmoid(x):
    return 1 / (1 + math.exp(-x))

class SimpleNeuralNetwork:
    def __init__(self, input_size, hidden_size, output_size):
        self.hidden_weights = [[random.random() for _ in range(input_size)] for _ in range(hidden_size)]
        self.output_weights = [[random.random() for _ in range(hidden_size)] for _ in range(output_size)]

    def forward(self, inputs):
        hidden = [sigmoid(sum(w * i for w, i in zip(weights, inputs))) for weights in self.hidden_weights]
        output = [sigmoid(sum(w * h for w, h in zip(weights, hidden))) for weights in self.output_weights]
        return output

def benchmark(network, num_samples):
    start_time = time.time()
    for _ in range(num_samples):
        inputs = [random.random() for _ in range(100)]
        _ = network.forward(inputs)
    end_time = time.time()
    return end_time - start_time

# Simulate improvement in computational power
network = SimpleNeuralNetwork(100, 50, 10)
num_samples = 10000

print("Simulating technological advancements:")
for year, speedup in [(2000, 1), (2010, 10), (2020, 100)]:
    time_taken = benchmark(network, num_samples) / speedup
    print(f"Year {year}: {time_taken:.4f} seconds for {num_samples} samples")

# Simulate improvement due to algorithm optimization
optimized_network = SimpleNeuralNetwork(100, 50, 10)
optimized_time = benchmark(optimized_network, num_samples) * 0.5  # Assume 2x speedup from optimization
print(f"With algorithmic improvements: {optimized_time:.4f} seconds for {num_samples} samples")

Results for Source Code for Factors Leading to the Deep Learning Revolution

Simulating technological advancements:
Year 2000: 0.4893 seconds for 10000 samples
Year 2010: 0.0480 seconds for 10000 samples
Year 2020: 0.0048 seconds for 10000 samples
With algorithmic improvements: 0.0024 seconds for 10000 samples

Additional Resources

For those interested in delving deeper into the field of Deep Learning, here are some valuable resources:

  1. ArXiv.org: A repository of research papers, including many on deep learning. Example: "Attention Is All You Need" by Vaswani et al. (2017) URL: https://arxiv.org/abs/1706.03762
  2. CS231n: Convolutional Neural Networks for Visual Recognition Stanford University's course materials available online.
  3. "Deep Learning" by Ian Goodfellow, Yoshua Bengio, and Aaron Courville Available online: https://www.deeplearningbook.org/
  4. TensorFlow and PyTorch documentation: Official guides and tutorials for popular deep learning frameworks.
  5. Distill.pub: A modern medium for presenting machine learning research.

These resources offer a mix of theoretical foundations and practical implementations to further your understanding of deep learning concepts and applications.

More in Neural network fundamentals

  1. Deep Dive into Neural Network Core Concepts13 min

    Neural networks are computational models inspired by the human brain. They consist of interconnected nodes (neurons) organized in layers, designed to...

  2. Building Neural Networks from Scratch in Python6 min

    Neural networks are computational models inspired by the human brain. They consist of interconnected nodes (neurons) that process and transmit information....

  3. Deep Learning Essentials with Python11 min

    Deep Learning is a subset of machine learning that uses artificial neural networks to model and solve complex problems. It has revolutionized various fields...

  4. Building a Custom Deep Learning Framework in Python7 min

    Deep learning frameworks are essential tools for building and training neural networks. In this presentation, we'll create our own lightweight framework from...

  5. Epochs and Iterations in Neural Network Training10 min

    Epochs and iterations are fundamental concepts in training neural networks. An epoch represents one complete pass through the entire training dataset, while...

  6. Artificial Neural Networks Fundamentals and Implementation11 min

    Artificial Neural Networks (ANNs) are computational models inspired by the human brain's structure and function. They consist of interconnected nodes...