xbe.at

Miscellany / AI applications and systems

Hopfield Network for Handwritten Digit Reconstruction

8 min read, 15 sections

Contents
  1. Hopfield Network Fundamentals
  2. Mathematical Foundation of Hopfield Networks
  3. Pattern Storage Implementation
  4. Pattern Retrieval and Dynamics
  5. Digit Recognition Implementation - Part 1
  6. Digit Recognition Implementation - Part 2
  7. Adding Noise to Test Pattern Recovery
  8. Energy Landscape Visualization
  9. Real-world Application: Character Recognition System
  10. Pattern Storage Capacity Analysis
  11. Implementation of Asynchronous Updates
  12. Real-world Application: Image Denoising
  13. Performance Metrics and Evaluation
  14. Advanced Pattern Recovery Techniques
  15. Additional Resources

Hopfield Network Fundamentals

A Hopfield Network is a recurrent artificial neural network with binary threshold nodes and symmetric weights between nodes. It functions as a content-addressable memory system capable of converging to stored patterns when presented with noisy or incomplete versions of those patterns.

import numpy as np

class HopfieldNetwork:
    def __init__(self, size):
        self.size = size
        self.weights = np.zeros((size, size))
    
    def train_pattern(self, pattern):
        # Implement Hebbian learning rule
        # W[i,j] = sum(x[i] * x[j]) for all patterns
        pattern = np.array(pattern)
        self.weights += np.outer(pattern, pattern)
        np.fill_diagonal(self.weights, 0)  # No self-connections
        
    def update(self, state):
        # Update rule: x[i] = sign(sum(w[i,j] * x[j]))
        return np.sign(self.weights @ state)

Mathematical Foundation of Hopfield Networks

The energy function of a Hopfield network represents the stability of the network state. The network evolves to minimize this energy function, which is guaranteed to converge to a local minimum due to the symmetric weight matrix.

def energy_function(weights, state):
    """
    Calculate network energy: E = -1/2 * sum(w[i,j] * s[i] * s[j])
    """
    return -0.5 * state @ weights @ state

Pattern Storage Implementation

The network's storage capacity is determined by its ability to maintain distinct attractors. For N neurons, the theoretical maximum number of patterns that can be stored is approximately 0.15N. Pattern storage involves calculating weight matrices using Hebbian learning.

def train_patterns(patterns):
    n_neurons = len(patterns[0])
    weights = np.zeros((n_neurons, n_neurons))
    
    for pattern in patterns:
        pattern = np.array(pattern)
        weights += np.outer(pattern, pattern)
    
    # Normalize and remove self-connections
    weights /= len(patterns)
    np.fill_diagonal(weights, 0)
    return weights

Pattern Retrieval and Dynamics

Pattern retrieval in Hopfield networks occurs through an iterative process where neurons update their states asynchronously or synchronously until convergence. The network eventually settles into a stable state representing the stored pattern.

def retrieve_pattern(weights, initial_state, max_iterations=100):
    current_state = np.array(initial_state)
    
    for _ in range(max_iterations):
        previous_state = current_state.copy()
        # Asynchronous update
        for i in range(len(current_state)):
            h = weights[i] @ current_state
            current_state[i] = 1 if h >= 0 else -1
            
        if np.array_equal(previous_state, current_state):
            break
            
    return current_state

Digit Recognition Implementation - Part 1

Implementation of a Hopfield Network for recognizing handwritten digits. This practical example demonstrates how to preprocess digit images and train the network to memorize digit patterns.

import numpy as np
from PIL import Image

def preprocess_digit(image_path, size=(8, 8)):
    # Load and preprocess digit image
    img = Image.open(image_path).convert('L')
    img = img.resize(size)
    pixel_array = np.array(img)
    # Convert to binary pattern (-1 and 1)
    binary_pattern = np.where(pixel_array > 128, 1, -1)
    return binary_pattern.flatten()

Digit Recognition Implementation - Part 2

The core implementation of digit pattern storage requires careful preprocessing and normalization. This implementation shows how to create a complete digit recognition system using the Hopfield Network architecture.

class DigitHopfieldNetwork:
    def __init__(self, image_size=(8, 8)):
        self.size = image_size[0] * image_size[1]
        self.weights = np.zeros((self.size, self.size))
        
    def train_digits(self, digit_patterns):
        # Train network on multiple digit patterns
        for pattern in digit_patterns:
            pattern = pattern.flatten()
            self.weights += np.outer(pattern, pattern)
        
        # Normalize weights
        self.weights /= len(digit_patterns)
        np.fill_diagonal(self.weights, 0)
        
    def recognize(self, noisy_pattern, max_iter=20):
        state = noisy_pattern.flatten()
        for _ in range(max_iter):
            new_state = np.sign(self.weights @ state)
            if np.array_equal(state, new_state):
                break
            state = new_state
        return state.reshape((int(np.sqrt(self.size)), -1))

Adding Noise to Test Pattern Recovery

The robustness of Hopfield Networks can be tested by adding noise to stored patterns. This implementation demonstrates how to corrupt patterns and measure the network's recovery accuracy.

def add_noise(pattern, noise_level=0.2):
    """
    Add random noise to a pattern
    noise_level: proportion of bits to flip
    """
    noisy = pattern.copy()
    n_flip = int(len(pattern) * noise_level)
    flip_idx = np.random.choice(len(pattern), n_flip, replace=False)
    noisy[flip_idx] *= -1
    return noisy

def measure_recovery_accuracy(original, recovered):
    """
    Calculate recovery accuracy percentage
    """
    return np.mean(original == recovered) * 100

Energy Landscape Visualization

Understanding the energy landscape helps visualize how the network converges to stored patterns. This implementation creates a simplified 2D visualization of the network's energy function.

import matplotlib.pyplot as plt

def plot_energy_landscape(hopfield_net, pattern_size=8):
    x = np.linspace(-1, 1, pattern_size)
    y = np.linspace(-1, 1, pattern_size)
    X, Y = np.meshgrid(x, y)
    
    Z = np.zeros_like(X)
    for i in range(pattern_size):
        for j in range(pattern_size):
            state = np.array([X[i,j], Y[i,j]])
            Z[i,j] = -0.5 * state @ hopfield_net.weights @ state
            
    plt.contour(X, Y, Z)
    plt.colorbar(label='Energy')
    plt.title('Hopfield Network Energy Landscape')
    plt.xlabel('State dimension 1')
    plt.ylabel('State dimension 2')
    return plt.gcf()

Real-world Application: Character Recognition System

Implementation of a complete character recognition system using Hopfield Network. This practical example shows how to handle real character images and recognize them even with distortions.

class CharacterRecognitionSystem:
    def __init__(self):
        self.hn = HopfieldNetwork(64)  # 8x8 characters
        self.char_patterns = {}
        
    def add_character(self, char, image_path):
        pattern = preprocess_digit(image_path)
        self.char_patterns[char] = pattern
        self.hn.train_pattern(pattern)
        
    def recognize_character(self, noisy_image_path):
        test_pattern = preprocess_digit(noisy_image_path)
        recovered = self.hn.update(test_pattern)
        
        # Find best matching stored pattern
        best_match = None
        best_score = -float('inf')
        for char, pattern in self.char_patterns.items():
            score = np.sum(recovered == pattern)
            if score > best_score:
                best_score = score
                best_match = char
                
        return best_match, best_score / len(recovered)

Pattern Storage Capacity Analysis

The storage capacity of a Hopfield Network is limited by the network size and pattern correlation. This implementation analyzes the network's capacity and demonstrates how performance degrades as the number of stored patterns increases.

def analyze_storage_capacity(network_size, max_patterns=50):
    hn = HopfieldNetwork(network_size)
    success_rates = []
    
    for n_patterns in range(1, max_patterns + 1):
        # Generate random patterns
        patterns = [np.random.choice([-1, 1], size=network_size) 
                   for _ in range(n_patterns)]
        
        # Train network
        for p in patterns:
            hn.train_pattern(p)
        
        # Test recovery
        successes = 0
        trials = 100
        for pattern in patterns:
            noisy = add_noise(pattern, 0.2)
            recovered = hn.update(noisy)
            if np.array_equal(recovered, pattern):
                successes += 1
                
        success_rates.append(successes / len(patterns))
        
    return success_rates

Implementation of Asynchronous Updates

Asynchronous updates in Hopfield Networks often lead to more stable convergence. This implementation shows how to perform randomized asynchronous updates and compare them with synchronous updates.

def async_update(weights, state, max_iterations=100):
    current_state = state.copy()
    n_neurons = len(state)
    
    for _ in range(max_iterations):
        old_state = current_state.copy()
        # Random order of neuron updates
        update_order = np.random.permutation(n_neurons)
        
        for i in update_order:
            # Update single neuron
            h = weights[i] @ current_state
            current_state[i] = np.sign(h)
            
        if np.array_equal(old_state, current_state):
            break
            
    return current_state, _

def compare_update_methods(pattern, noise_level=0.2):
    """Compare convergence of sync vs async updates"""
    noisy = add_noise(pattern, noise_level)
    
    sync_result = retrieve_pattern(weights, noisy)
    async_result, async_iters = async_update(weights, noisy)
    
    return {
        'sync_accuracy': measure_recovery_accuracy(pattern, sync_result),
        'async_accuracy': measure_recovery_accuracy(pattern, async_result),
        'async_iterations': async_iters
    }

Real-world Application: Image Denoising

Implementation of an image denoising system using Hopfield Networks. This example demonstrates how to handle grayscale images and remove various types of noise.

class ImageDenoiser:
    def __init__(self, patch_size=8):
        self.patch_size = patch_size
        self.hn = HopfieldNetwork(patch_size * patch_size)
        
    def _extract_patches(self, image):
        h, w = image.shape
        patches = []
        for i in range(0, h - self.patch_size + 1, self.patch_size):
            for j in range(0, w - self.patch_size + 1, self.patch_size):
                patch = image[i:i+self.patch_size, j:j+self.patch_size]
                patches.append(patch.flatten())
        return patches, (h, w)
    
    def train(self, clean_image):
        patches, _ = self._extract_patches(clean_image)
        for patch in patches:
            self.hn.train_pattern(np.sign(patch))
    
    def denoise(self, noisy_image):
        patches, (h, w) = self._extract_patches(noisy_image)
        denoised_patches = []
        
        for patch in patches:
            recovered = self.hn.update(np.sign(patch))
            denoised_patches.append(recovered)
            
        # Reconstruct image
        denoised = np.zeros((h, w))
        patch_idx = 0
        for i in range(0, h - self.patch_size + 1, self.patch_size):
            for j in range(0, w - self.patch_size + 1, self.patch_size):
                denoised[i:i+self.patch_size, j:j+self.patch_size] = \
                    denoised_patches[patch_idx].reshape((self.patch_size, self.patch_size))
                patch_idx += 1
                
        return denoised

Performance Metrics and Evaluation

This implementation provides comprehensive performance metrics for Hopfield Networks, including convergence rate, pattern stability, and noise tolerance analysis. The evaluation framework helps assess network reliability in practical applications.

class HopfieldEvaluator:
    def __init__(self, network):
        self.network = network
        self.metrics = {}
        
    def evaluate_convergence(self, patterns, noise_levels=[0.1, 0.2, 0.3]):
        results = {}
        for noise in noise_levels:
            successes = 0
            iterations = []
            
            for pattern in patterns:
                noisy = add_noise(pattern, noise)
                final_state, iters = self.network.update(noisy, return_iterations=True)
                
                if np.array_equal(final_state, pattern):
                    successes += 1
                iterations.append(iters)
                
            results[noise] = {
                'success_rate': successes / len(patterns),
                'avg_iterations': np.mean(iterations),
                'std_iterations': np.std(iterations)
            }
            
        return results
    
    def analyze_basin_of_attraction(self, pattern, n_samples=100):
        distances = np.linspace(0.1, 0.9, 9)
        attraction_strength = []
        
        for d in distances:
            recoveries = 0
            for _ in range(n_samples):
                corrupted = self._corrupt_pattern(pattern, d)
                recovered = self.network.update(corrupted)
                if np.array_equal(recovered, pattern):
                    recoveries += 1
                    
            attraction_strength.append(recoveries / n_samples)
            
        return distances, attraction_strength
    
    @staticmethod
    def _corrupt_pattern(pattern, distance):
        n_flip = int(len(pattern) * distance)
        corrupted = pattern.copy()
        flip_idx = np.random.choice(len(pattern), n_flip, replace=False)
        corrupted[flip_idx] *= -1
        return corrupted

Advanced Pattern Recovery Techniques

Implementation of sophisticated pattern recovery methods including temperature-based annealing and probabilistic updates, which can help escape local minima and improve pattern reconstruction.

class AdvancedHopfieldNetwork:
    def __init__(self, size):
        self.size = size
        self.weights = np.zeros((size, size))
        
    def update_with_annealing(self, state, T_start=1.0, T_min=0.01, cooling_rate=0.95):
        current_state = state.copy()
        T = T_start
        
        while T > T_min:
            for i in range(self.size):
                h = self.weights[i] @ current_state
                # Probabilistic update using temperature
                p = 1 / (1 + np.exp(-2 * h / T))
                current_state[i] = 1 if np.random.random() < p else -1
                
            T *= cooling_rate
            
        return current_state
    
    def update_with_momentum(self, state, alpha=0.9, max_iter=100):
        current_state = state.copy()
        velocity = np.zeros_like(state)
        
        for _ in range(max_iter):
            old_state = current_state.copy()
            
            # Update with momentum
            h = self.weights @ current_state
            velocity = alpha * velocity + (1 - alpha) * h
            current_state = np.sign(velocity)
            
            if np.array_equal(old_state, current_state):
                break
                
        return current_state

Additional Resources

More in AI applications and systems

  1. Introducing Foundation Agents The Next Frontier in AI Decision-Making9 min

    Foundation Agents are an emerging paradigm in AI that combines large language models (LLMs) with planning and decision-making capabilities. They aim to...

  2. Fun with Sparse PyTorch Models via Hadamard Parametrization10 min

    Sparsity in neural networks refers to the concept of having many zero-valued parameters. This can lead to more efficient models in terms of computation and...

  3. Introduction to Activate Functions6 min

    Activation functions are crucial components in neural networks, introducing non-linearity to the model and enabling it to learn complex patterns. They...

  4. Exploring the Building Blocks of Modern AI12 min

    A Multi-layer Perceptron forms the foundation of modern neural networks, implementing forward and backward propagation to learn patterns in data through...

  5. John McCarthy - Father of Artificial Intelligence9 min

    John McCarthy was a pioneering computer scientist and cognitive scientist who played a crucial role in shaping the field of Artificial Intelligence (AI). His...

  6. Explainable AI Demystifying Complex Models8 min

    SHAP (SHapley Additive exPlanations) values provide a unified measure of feature importance based on game theoretic principles. These values represent the...