xbe.at

Python / Python for ML and AI

Understanding LSTMs with Python

10 min read, 16 sections

Contents
  1. Introduction to LSTMs
  2. LSTM Cell Structure
  3. The Forget Gate
  4. The Input Gate
  5. Updating the Cell State
  6. The Output Gate
  7. Implementing a Simple LSTM Layer
  8. Preparing Data for LSTM
  9. Training an LSTM Model
  10. Making Predictions with LSTM
  11. Real-Life Example: Weather Forecasting
  12. Real-Life Example: Text Generation
  13. LSTM Variants: Bidirectional and Stacked LSTMs
  14. LSTM Hyperparameter Tuning
  15. Handling Long Sequences: Truncated Backpropagation Through Time
  16. Additional Resources

Introduction to LSTMs

Long Short-Term Memory (LSTM) networks are a type of recurrent neural network designed to overcome the vanishing gradient problem in traditional RNNs. LSTMs are particularly effective for processing and predicting time series data, making them ideal for tasks like natural language processing, speech recognition, and time series forecasting.

import numpy as np
import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, Dense

# Basic LSTM model structure
model = Sequential([
    LSTM(50, activation='relu', input_shape=(sequence_length, features)),
    Dense(1)
])
model.compile(optimizer='adam', loss='mse')

LSTM Cell Structure

An LSTM cell consists of three main components: the forget gate, the input gate, and the output gate. These gates control the flow of information through the cell, allowing it to selectively remember or forget information over long sequences.

import matplotlib.pyplot as plt
import numpy as np

def draw_lstm_cell():
    fig, ax = plt.subplots(figsize=(10, 8))
    
    # Cell state line
    ax.plot([0, 10], [5, 5], 'k-', linewidth=2)
    
    # Gates
    gates = ['Forget', 'Input', 'Output']
    positions = [2, 5, 8]
    for gate, pos in zip(gates, positions):
        ax.add_patch(plt.Circle((pos, 3), 0.5, fill=False))
        ax.text(pos, 2, gate, ha='center')
    
    # Sigmoid and tanh activations
    ax.text(5, 7, 'tanh', ha='center')
    ax.text(3.5, 4, 'σ', ha='center')
    ax.text(6.5, 4, 'σ', ha='center')
    
    ax.axis('off')
    plt.title('LSTM Cell Structure')
    plt.show()

draw_lstm_cell()

The Forget Gate

The forget gate decides what information to discard from the cell state. It takes the previous hidden state and current input, passes them through a sigmoid function, and outputs values between 0 and 1 for each number in the cell state.

def forget_gate(previous_hidden_state, current_input):
    combined_input = np.concatenate([previous_hidden_state, current_input])
    forget_weights = np.random.randn(len(combined_input))
    forget_bias = np.random.randn()
    
    forget_output = 1 / (1 + np.exp(-(np.dot(forget_weights, combined_input) + forget_bias)))
    return forget_output

# Example usage
prev_hidden = np.array([0.1, 0.2, 0.3])
curr_input = np.array([0.4, 0.5])
forget_output = forget_gate(prev_hidden, curr_input)
print(f"Forget gate output: {forget_output}")

The Input Gate

The input gate determines what new information to store in the cell state. It consists of two parts: a sigmoid layer that decides which values to update, and a tanh layer that creates new candidate values to be added to the state.

def input_gate(previous_hidden_state, current_input):
    combined_input = np.concatenate([previous_hidden_state, current_input])
    
    # Sigmoid layer
    input_weights_sigmoid = np.random.randn(len(combined_input))
    input_bias_sigmoid = np.random.randn()
    input_gate_sigmoid = 1 / (1 + np.exp(-(np.dot(input_weights_sigmoid, combined_input) + input_bias_sigmoid)))
    
    # Tanh layer
    input_weights_tanh = np.random.randn(len(combined_input))
    input_bias_tanh = np.random.randn()
    input_gate_tanh = np.tanh(np.dot(input_weights_tanh, combined_input) + input_bias_tanh)
    
    return input_gate_sigmoid, input_gate_tanh

# Example usage
prev_hidden = np.array([0.1, 0.2, 0.3])
curr_input = np.array([0.4, 0.5])
input_sigmoid, input_tanh = input_gate(prev_hidden, curr_input)
print(f"Input gate sigmoid output: {input_sigmoid}")
print(f"Input gate tanh output: {input_tanh}")

Updating the Cell State

The cell state is updated by combining the outputs of the forget gate and input gate. We multiply the old cell state by the forget gate output, then add the product of the input gate sigmoid and tanh outputs.

def update_cell_state(old_cell_state, forget_output, input_sigmoid, input_tanh):
    new_cell_state = old_cell_state * forget_output + input_sigmoid * input_tanh
    return new_cell_state

# Example usage
old_cell_state = np.array([0.5, 0.6, 0.7])
forget_output = np.array([0.4, 0.5, 0.6])
input_sigmoid = np.array([0.7, 0.8, 0.9])
input_tanh = np.array([0.1, 0.2, 0.3])

new_cell_state = update_cell_state(old_cell_state, forget_output, input_sigmoid, input_tanh)
print(f"Updated cell state: {new_cell_state}")

The Output Gate

The output gate determines what information from the cell state will be output. It uses a sigmoid layer to decide which parts of the cell state to output, and then multiplies this by a tanh of the cell state.

def output_gate(previous_hidden_state, current_input, new_cell_state):
    combined_input = np.concatenate([previous_hidden_state, current_input])
    
    # Sigmoid layer
    output_weights = np.random.randn(len(combined_input))
    output_bias = np.random.randn()
    output_gate = 1 / (1 + np.exp(-(np.dot(output_weights, combined_input) + output_bias)))
    
    # Final output
    final_output = output_gate * np.tanh(new_cell_state)
    return final_output

# Example usage
prev_hidden = np.array([0.1, 0.2, 0.3])
curr_input = np.array([0.4, 0.5])
new_cell_state = np.array([0.6, 0.7, 0.8])

final_output = output_gate(prev_hidden, curr_input, new_cell_state)
print(f"Final output: {final_output}")

Implementing a Simple LSTM Layer

Let's implement a basic LSTM layer using TensorFlow and Keras. This example shows how to create an LSTM layer and add it to a sequential model.

import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, Dense

# Define the model
model = Sequential([
    LSTM(64, input_shape=(10, 1), return_sequences=True),
    LSTM(32),
    Dense(1)
])

# Compile the model
model.compile(optimizer='adam', loss='mse')

# Print model summary
model.summary()

Preparing Data for LSTM

Before feeding data into an LSTM, we need to reshape it into a 3D array with dimensions (samples, time steps, features). Here's an example of how to prepare time series data for an LSTM.

import numpy as np

def prepare_data(data, n_steps):
    X, y = [], []
    for i in range(len(data) - n_steps):
        X.append(data[i:i+n_steps])
        y.append(data[i+n_steps])
    return np.array(X), np.array(y)

# Example usage
time_series = np.sin(np.linspace(0, 20, 200)) + np.random.normal(0, 0.1, 200)
n_steps = 10

X, y = prepare_data(time_series, n_steps)
print(f"Input shape: {X.shape}")
print(f"Output shape: {y.shape}")

# Reshape for LSTM input (samples, time steps, features)
X = X.reshape((X.shape[0], X.shape[1], 1))
print(f"Reshaped input: {X.shape}")

Training an LSTM Model

Now that we have prepared our data, let's train an LSTM model to predict the next value in our time series.

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, Dense
from sklearn.model_selection import train_test_split

# Prepare data (using the previous slide's code)
time_series = np.sin(np.linspace(0, 20, 200)) + np.random.normal(0, 0.1, 200)
n_steps = 10
X, y = prepare_data(time_series, n_steps)
X = X.reshape((X.shape[0], X.shape[1], 1))

# Split data into train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Define and compile the model
model = Sequential([
    LSTM(50, activation='relu', input_shape=(n_steps, 1)),
    Dense(1)
])
model.compile(optimizer='adam', loss='mse')

# Train the model
history = model.fit(X_train, y_train, epochs=50, batch_size=32, validation_split=0.1, verbose=0)

# Evaluate the model
test_loss = model.evaluate(X_test, y_test)
print(f"Test loss: {test_loss}")

Making Predictions with LSTM

After training our LSTM model, we can use it to make predictions on new data. Here's how to generate predictions and visualize the results.

import matplotlib.pyplot as plt

# Make predictions
train_predictions = model.predict(X_train)
test_predictions = model.predict(X_test)

# Prepare data for plotting
train_dates = range(n_steps, len(y_train) + n_steps)
test_dates = range(len(y_train) + n_steps, len(time_series))

# Plot results
plt.figure(figsize=(12, 6))
plt.plot(time_series, label='Actual')
plt.plot(train_dates, train_predictions, label='Train Predictions')
plt.plot(test_dates, test_predictions, label='Test Predictions')
plt.legend()
plt.title('LSTM Time Series Prediction')
plt.xlabel('Time')
plt.ylabel('Value')
plt.show()

Real-Life Example: Weather Forecasting

LSTMs can be used for weather forecasting by predicting future temperature values based on historical data. Here's a simplified example using synthetic weather data.

import numpy as np
import pandas as pd
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, Dense

# Generate synthetic weather data
dates = pd.date_range(start='2022-01-01', end='2023-12-31', freq='D')
temperatures = 20 + 10 * np.sin(np.arange(len(dates)) * 2 * np.pi / 365) + np.random.normal(0, 2, len(dates))
df = pd.DataFrame({'date': dates, 'temperature': temperatures})

# Prepare data for LSTM
X, y = prepare_data(df['temperature'].values, n_steps=30)
X = X.reshape((X.shape[0], X.shape[1], 1))

# Split data
train_size = int(len(X) * 0.8)
X_train, X_test = X[:train_size], X[train_size:]
y_train, y_test = y[:train_size], y[train_size:]

# Create and train model
model = Sequential([
    LSTM(50, activation='relu', input_shape=(30, 1)),
    Dense(1)
])
model.compile(optimizer='adam', loss='mse')
model.fit(X_train, y_train, epochs=50, batch_size=32, validation_split=0.1, verbose=0)

# Make predictions
predictions = model.predict(X_test)

# Plot results
plt.figure(figsize=(12, 6))
plt.plot(df['date'][30:], df['temperature'][30:], label='Actual')
plt.plot(df['date'][train_size+30:], predictions, label='Predicted')
plt.legend()
plt.title('LSTM Weather Forecast')
plt.xlabel('Date')
plt.ylabel('Temperature (°C)')
plt.show()

Real-Life Example: Text Generation

LSTMs are commonly used in natural language processing tasks, including text generation. Here's a simple example of how to use an LSTM for character-level text generation.

import numpy as np
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, Dense
from tensorflow.keras.utils import to_categorical

# Sample text
text = "Hello, world! This is a simple example of text generation using LSTM."
chars = sorted(list(set(text)))
char_to_int = {c: i for i, c in enumerate(chars)}
int_to_char = {i: c for c, i in char_to_int.items()}

# Prepare sequences
seq_length = 10
X = []
y = []
for i in range(len(text) - seq_length):
    X.append([char_to_int[c] for c in text[i:i+seq_length]])
    y.append(char_to_int[text[i+seq_length]])

X = np.array(X)
y = to_categorical(y)

# Create and train model
model = Sequential([
    LSTM(128, input_shape=(seq_length, 1)),
    Dense(len(chars), activation='softmax')
])
model.compile(loss='categorical_crossentropy', optimizer='adam')
model.fit(X.reshape(X.shape[0], X.shape[1], 1), y, epochs=100, batch_size=32, verbose=0)

# Generate text
start = np.random.randint(0, len(X)-1)
pattern = X[start]
generated_text = ''.join([int_to_char[value] for value in pattern])

for _ in range(100):
    x = np.reshape(pattern, (1, len(pattern), 1))
    prediction = model.predict(x, verbose=0)
    index = np.argmax(prediction)
    result = int_to_char[index]
    generated_text += result
    pattern = np.append(pattern[1:], index)

print("Generated text:")
print(generated_text)

LSTM Variants: Bidirectional and Stacked LSTMs

LSTMs can be modified to create more powerful architectures. Bidirectional LSTMs process sequences in both forward and backward directions, while stacked LSTMs use multiple LSTM layers for increased complexity.

from tensorflow.keras.layers import Bidirectional

# Bidirectional LSTM
bi_lstm_model = Sequential([
    Bidirectional(LSTM(64, return_sequences=True), input_shape=(sequence_length, features)),
    Bidirectional(LSTM(32)),
    Dense(1)
])

# Stacked LSTM
stacked_lstm_model = Sequential([
    LSTM(64, return_sequences=True, input_shape=(sequence_length, features)),
    LSTM(32, return_sequences=True),
    LSTM(16),
    Dense(1)
])

# Print model summaries
bi_lstm_model.summary()
stacked_lstm_model.summary()

LSTM Hyperparameter Tuning

Optimizing LSTM performance often involves tuning hyperparameters. Key parameters include the number of LSTM units, number of layers, dropout rate, and learning rate. Here's an example using Keras Tuner to find optimal hyperparameters:

import keras_tuner as kt
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, Dense, Dropout

def build_model(hp):
    model = Sequential()
    model.add(LSTM(units=hp.Int('units', min_value=32, max_value=512, step=32),
                   input_shape=(sequence_length, features)))
    model.add(Dropout(hp.Float('dropout', min_value=0.0, max_value=0.5, step=0.1)))
    model.add(Dense(1))
    
    model.compile(optimizer=tf.keras.optimizers.Adam(hp.Float('learning_rate', min_value=1e-4, max_value=1e-2, sampling='log')),
                  loss='mse')
    return model

tuner = kt.Hyperband(build_model, objective='val_loss', max_epochs=50, factor=3, directory='my_dir', project_name='lstm_tuning')

tuner.search(X_train, y_train, epochs=50, validation_split=0.2)

best_model = tuner.get_best_models(num_models=1)[0]
best_hps = tuner.get_best_hyperparameters(num_trials=1)[0]

print(f"Best hyperparameters: {best_hps.values}")

Handling Long Sequences: Truncated Backpropagation Through Time

For very long sequences, training LSTMs can be computationally expensive. Truncated Backpropagation Through Time (TBPTT) is a technique to make training more efficient by limiting the number of time steps in each backward pass.

def tbptt_model(input_shape, output_size, truncated_length):
    model = Sequential([
        LSTM(64, return_sequences=True, stateful=True, batch_input_shape=input_shape),
        LSTM(32, return_sequences=True, stateful=True),
        TimeDistributed(Dense(output_size))
    ])
    model.compile(optimizer='adam', loss='mse')
    return model

# Pseudo-code for TBPTT training
for epoch in range(num_epochs):
    model.reset_states()
    for batch in batches:
        for t in range(0, sequence_length, truncated_length):
            x_batch = batch[:, t:t+truncated_length, :]
            y_batch = batch[:, t+1:t+truncated_length+1, :]
            model.train_on_batch(x_batch, y_batch)

Additional Resources

For more in-depth understanding of LSTMs and their applications, consider exploring these peer-reviewed papers:

  1. "Long Short-Term Memory" by Hochreiter and Schmidhuber (1997) ArXiv: https://arxiv.org/abs/1409.1259v2
  2. "LSTM: A Search Space Odyssey" by Greff et al. (2017) ArXiv: https://arxiv.org/abs/1503.04069
  3. "Visualizing and Understanding Recurrent Networks" by Karpathy et al. (2015) ArXiv: https://arxiv.org/abs/1506.02078

These papers provide foundational knowledge and advanced insights into LSTM architectures and their applications in various domains.

More in Python for ML and AI

  1. Understanding Robust Scaler with Python8 min

    Robust Scaler is a preprocessing technique used in machine learning to scale features that may contain outliers. It uses statistics that are robust to...

  2. Understanding Keras Model with Python Examples6 min

    Keras is a high-level neural network API, written in Python and capable of running on top of TensorFlow, CNTK, or Theano. It was developed with a focus on...

  3. Understanding the Curse of Dimensionality with Python8 min

    The curse of dimensionality refers to various phenomena that arise when analyzing and organizing data in high-dimensional spaces. As the number of dimensions...

  4. Ultimate Guide to Detecting and Removing Outliers in Python10 min

    Outliers are data points that significantly differ from other observations in a dataset. They can occur due to various reasons such as measurement errors...

  5. Univariate Analysis of Continuous Variables in Python7 min

    Univariate analysis is a fundamental statistical method that focuses on examining and describing a single variable at a time. For continuous variables, this...

  6. Transforming Document Summarization with Python8 min

    Document summarization is the process of distilling the most important information from a text while maintaining its core meaning. This technique is crucial...