LSTM for Time Series Forecasting in Python

Introduction to Long Short-Term Memory (LSTM)

Long Short-Term Memory (LSTM) is a type of recurrent neural network architecture designed to address the vanishing gradient problem in traditional RNNs. LSTMs are particularly effective for time series analysis and prediction due to their ability to capture long-term dependencies in sequential data.

import numpy as np
import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, Dense

# Create a simple LSTM model
model = Sequential([
    LSTM(50, activation='relu', input_shape=(10, 1)),
    Dense(1)
])
model.compile(optimizer='adam', loss='mse')

LSTM Cell Structure

The LSTM cell consists of three gates: forget gate, input gate, and output gate. These gates control the flow of information through the cell, allowing it to selectively remember or forget information over long sequences.

import matplotlib.pyplot as plt

def plot_lstm_cell():
    fig, ax = plt.subplots(figsize=(10, 6))
    ax.axis('off')
    
    # Cell state
    ax.add_patch(plt.Rectangle((0.1, 0.4), 0.8, 0.2, fill=False))
    ax.text(0.5, 0.5, 'Cell State', ha='center', va='center')
    
    # Gates
    gates = ['Forget Gate', 'Input Gate', 'Output Gate']
    for i, gate in enumerate(gates):
        ax.add_patch(plt.Circle((0.3 + i*0.2, 0.2), 0.1, fill=False))
        ax.text(0.3 + i*0.2, 0.2, gate, ha='center', va='center')
    
    plt.show()

plot_lstm_cell()

Forget Gate

The forget gate decides what information to discard from the cell state. It takes the previous hidden state and current input, passing them through a sigmoid function to output values between 0 and 1.

def forget_gate(prev_hidden_state, current_input):
    combined_input = np.concatenate([prev_hidden_state, current_input])
    forget_weights = np.random.rand(combined_input.shape[0])
    forget_bias = np.random.rand()
    
    forget_output = 1 / (1 + np.exp(-(np.dot(forget_weights, combined_input) + forget_bias)))
    return forget_output

# Example usage
prev_hidden = np.random.rand(10)
current_input = np.random.rand(5)
forget_output = forget_gate(prev_hidden, current_input)
print(f"Forget gate output: {forget_output}")

Input Gate

The input gate determines what new information to store in the cell state. It consists of two parts: a sigmoid layer to decide which values to update and a tanh layer to create candidate values.

def input_gate(prev_hidden_state, current_input):
    combined_input = np.concatenate([prev_hidden_state, current_input])
    
    # Sigmoid layer
    input_weights_sigmoid = np.random.rand(combined_input.shape[0])
    input_bias_sigmoid = np.random.rand()
    input_gate_output = 1 / (1 + np.exp(-(np.dot(input_weights_sigmoid, combined_input) + input_bias_sigmoid)))
    
    # Tanh layer
    input_weights_tanh = np.random.rand(combined_input.shape[0])
    input_bias_tanh = np.random.rand()
    candidate_values = np.tanh(np.dot(input_weights_tanh, combined_input) + input_bias_tanh)
    
    return input_gate_output, candidate_values

# Example usage
prev_hidden = np.random.rand(10)
current_input = np.random.rand(5)
input_output, candidates = input_gate(prev_hidden, current_input)
print(f"Input gate output: {input_output}")
print(f"Candidate values: {candidates}")

Cell State Update

The cell state is updated by combining the forget gate and input gate outputs. The forget gate "forgets" information from the previous cell state, while the input gate adds new information.

def update_cell_state(prev_cell_state, forget_output, input_output, candidates):
    # Forget part of the previous cell state
    forgotten_state = prev_cell_state * forget_output
    
    # Add new information to the cell state
    new_info = input_output * candidates
    
    # Update the cell state
    new_cell_state = forgotten_state + new_info
    
    return new_cell_state

# Example usage
prev_cell_state = np.random.rand(10)
forget_output = np.random.rand(10)
input_output = np.random.rand(10)
candidates = np.random.rand(10)

new_cell_state = update_cell_state(prev_cell_state, forget_output, input_output, candidates)
print(f"Updated cell state: {new_cell_state}")

Output Gate

The output gate determines what information from the cell state will be output as the hidden state. It uses a sigmoid function to decide which parts of the cell state to output and applies tanh to the cell state.

def output_gate(prev_hidden_state, current_input, new_cell_state):
    combined_input = np.concatenate([prev_hidden_state, current_input])
    
    # Output gate
    output_weights = np.random.rand(combined_input.shape[0])
    output_bias = np.random.rand()
    output_gate = 1 / (1 + np.exp(-(np.dot(output_weights, combined_input) + output_bias)))
    
    # New hidden state
    new_hidden_state = output_gate * np.tanh(new_cell_state)
    
    return new_hidden_state

# Example usage
prev_hidden = np.random.rand(10)
current_input = np.random.rand(5)
new_cell_state = np.random.rand(10)

new_hidden_state = output_gate(prev_hidden, current_input, new_cell_state)
print(f"New hidden state: {new_hidden_state}")

Preparing Time Series Data for LSTM

To use LSTM for time series analysis, we need to prepare our data in a specific format. This involves creating sequences of input data and corresponding target values.

import numpy as np

def create_sequences(data, seq_length):
    sequences = []
    targets = []
    for i in range(len(data) - seq_length):
        seq = data[i:i+seq_length]
        target = data[i+seq_length]
        sequences.append(seq)
        targets.append(target)
    return np.array(sequences), np.array(targets)

# Example usage
time_series_data = np.sin(np.linspace(0, 10, 100))
seq_length = 10

X, y = create_sequences(time_series_data, seq_length)
print(f"Input shape: {X.shape}")
print(f"Target shape: {y.shape}")

Building an LSTM Model for Time Series Prediction

Now, let's build a simple LSTM model for time series prediction using TensorFlow and Keras.

import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, Dense

def build_lstm_model(seq_length):
    model = Sequential([
        LSTM(50, activation='relu', input_shape=(seq_length, 1)),
        Dense(1)
    ])
    model.compile(optimizer='adam', loss='mse')
    return model

# Example usage
seq_length = 10
model = build_lstm_model(seq_length)
model.summary()

Training the LSTM Model

With our model built, we can now train it on our prepared time series data.

import numpy as np
import matplotlib.pyplot as plt

# Generate sample data
time_series_data = np.sin(np.linspace(0, 10, 1000))
X, y = create_sequences(time_series_data, seq_length)

# Reshape input data for LSTM (samples, time steps, features)
X = X.reshape((X.shape[0], X.shape[1], 1))

# Train the model
history = model.fit(X, y, epochs=50, batch_size=32, validation_split=0.2, verbose=0)

# Plot training history
plt.plot(history.history['loss'], label='Training Loss')
plt.plot(history.history['val_loss'], label='Validation Loss')
plt.legend()
plt.show()

Making Predictions with the Trained LSTM Model

After training, we can use our LSTM model to make predictions on new data.

def predict_next_value(model, sequence):
    sequence = sequence.reshape((1, sequence.shape[0], 1))
    prediction = model.predict(sequence)
    return prediction[0][0]

# Example usage
test_sequence = time_series_data[-seq_length:]
next_value = predict_next_value(model, test_sequence)
print(f"Predicted next value: {next_value}")

# Plot the results
plt.plot(range(len(time_series_data)), time_series_data, label='Actual')
plt.plot(len(time_series_data), next_value, 'ro', label='Prediction')
plt.legend()
plt.show()

Real-Life Example: Weather Forecasting

LSTMs can be applied to weather forecasting by predicting future temperatures based on historical data. This example demonstrates how to prepare weather data and train an LSTM model for temperature prediction.

import numpy as np
import pandas as pd
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, Dense

# Generate sample weather data
dates = pd.date_range(start='2020-01-01', end='2022-12-31', freq='D')
temperatures = np.sin(np.arange(len(dates)) * 2 * np.pi / 365) * 10 + 20  # Simulated seasonal temperature
weather_data = pd.DataFrame({'date': dates, 'temperature': temperatures})

# Prepare sequences
sequence_length = 30
X, y = create_sequences(weather_data['temperature'].values, sequence_length)
X = X.reshape((X.shape[0], X.shape[1], 1))

# Build and train the model
model = Sequential([
    LSTM(50, activation='relu', input_shape=(sequence_length, 1)),
    Dense(1)
])
model.compile(optimizer='adam', loss='mse')
model.fit(X, y, epochs=50, batch_size=32, validation_split=0.2, verbose=0)

# Make predictions
last_sequence = weather_data['temperature'].values[-sequence_length:]
next_temp = predict_next_value(model, last_sequence)
print(f"Predicted temperature for the next day: {next_temp:.2f}°C")

Real-Life Example: Energy Consumption Forecasting

LSTMs can be used to forecast energy consumption patterns, helping utility companies optimize resource allocation and plan for peak demand periods.

import numpy as np
import pandas as pd
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, Dense

# Generate sample energy consumption data
dates = pd.date_range(start='2020-01-01', end='2022-12-31', freq='H')
consumption = np.sin(np.arange(len(dates)) * 2 * np.pi / 24) * 100 + 200  # Simulated daily consumption pattern
noise = np.random.normal(0, 20, len(dates))
energy_data = pd.DataFrame({'date': dates, 'consumption': consumption + noise})

# Prepare sequences
sequence_length = 24  # Use 24 hours of data to predict the next hour
X, y = create_sequences(energy_data['consumption'].values, sequence_length)
X = X.reshape((X.shape[0], X.shape[1], 1))

# Build and train the model
model = Sequential([
    LSTM(50, activation='relu', input_shape=(sequence_length, 1)),
    Dense(1)
])
model.compile(optimizer='adam', loss='mse')
model.fit(X, y, epochs=50, batch_size=32, validation_split=0.2, verbose=0)

# Make predictions
last_sequence = energy_data['consumption'].values[-sequence_length:]
next_consumption = predict_next_value(model, last_sequence)
print(f"Predicted energy consumption for the next hour: {next_consumption:.2f} kWh")

Hyperparameter Tuning for LSTM Models

Optimizing LSTM performance often requires tuning various hyperparameters. This example demonstrates how to use Keras Tuner to find the best hyperparameters for an LSTM model.

import keras_tuner as kt
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, Dense

def build_model(hp):
    model = Sequential()
    model.add(LSTM(
        units=hp.Int('units', min_value=32, max_value=512, step=32),
        activation='relu',
        input_shape=(sequence_length, 1)
    ))
    model.add(Dense(1))
    model.compile(
        optimizer=tf.keras.optimizers.Adam(
            hp.Float('learning_rate', min_value=1e-4, max_value=1e-2, sampling='log')
        ),
        loss='mse'
    )
    return model

tuner = kt.Hyperband(
    build_model,
    objective='val_loss',
    max_epochs=50,
    factor=3,
    directory='my_dir',
    project_name='lstm_tuning'
)

# Assuming X and y are already prepared
tuner.search(X, y, epochs=50, validation_split=0.2)

best_hps = tuner.get_best_hyperparameters(num_trials=1)[0]
print(f"Best number of units: {best_hps.get('units')}")
print(f"Best learning rate: {best_hps.get('learning_rate')}")

Handling Multiple Features in LSTM Models

In real-world scenarios, time series often involve multiple features. This example shows how to prepare multi-feature data and build an LSTM model to handle it.

import numpy as np
import pandas as pd
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, Dense

# Generate sample multi-feature time series data
dates = pd.date_range(start='2020-01-01', end='2022-12-31', freq='D')
temperature = np.sin(np.arange(len(dates)) * 2 * np.pi / 365) * 10 + 20
humidity = np.cos(np.arange(len(dates)) * 2 * np.pi / 365) * 20 + 60
wind_speed = np.random.normal(10, 3, len(dates))

weather_data = pd.DataFrame({
    'date': dates,
    'temperature': temperature,
    'humidity': humidity,
    'wind_speed': wind_speed
})

# Prepare sequences
sequence_length = 30
features = ['temperature', 'humidity', 'wind_speed']

def create_multifeature_sequences(data, sequence_length, features):
    X, y = [], []
    for i in range(len(data) - sequence_length):
        X.append(data[features].values[i:i+sequence_length])
        y.append(data['temperature'].values[i+sequence_length])
    return np.array(X), np.array(y)

X, y = create_multifeature_sequences(weather_data, sequence_length, features)

# Build and train the model
model = Sequential([
    LSTM(50, activation='relu', input_shape=(sequence_length, len(features))),
    Dense(1)
])
model.compile(optimizer='adam', loss='mse')
model.fit(X, y, epochs=50, batch_size=32, validation_split=0.2, verbose=0)

# Make predictions
last_sequence = weather_data[features].values[-sequence_length:]
next_temp = model.predict(last_sequence.reshape(1, sequence_length, len(features)))
print(f"Predicted temperature for the next day: {next_temp[0][0]:.2f}°C")

Stacked LSTM for Complex Time Series

For more complex time series problems, we can use stacked LSTM layers to capture hierarchical patterns in the data.

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, Dense

def build_stacked_lstm(input_shape, num_layers=2, units=50):
    model = Sequential()
    
    for i in range(num_layers - 1):
        model.add(LSTM(units, activation='relu', return_sequences=True, input_shape=input_shape))
    
    model.add(LSTM(units, activation='relu'))
    model.add(Dense(1))
    
    model.compile(optimizer='adam', loss='mse')
    return model

# Assuming X and y are prepared as in the previous example
input_shape = (X.shape[1], X.shape[2])
stacked_model = build_stacked_lstm(input_shape, num_layers=3, units=64)
stacked_model.fit(X, y, epochs=50, batch_size=32, validation_split=0.2, verbose=0)

# Make predictions
last_sequence = X[-1:]
next_value = stacked_model.predict(last_sequence)
print(f"Predicted value: {next_value[0][0]:.2f}")

Bidirectional LSTM for Time Series

Bidirectional LSTMs can capture patterns from both past and future contexts in a time series, which can be useful for certain types of data.

from tensorflow.keras.layers import Bidirectional

def build_bidirectional_lstm(input_shape, units=50):
    model = Sequential([
        Bidirectional(LSTM(units, activation='relu'), input_shape=input_shape),
        Dense(1)
    ])
    model.compile(optimizer='adam', loss='mse')
    return model

# Assuming X and y are prepared as in the previous examples
input_shape = (X.shape[1], X.shape[2])
bi_lstm_model = build_bidirectional_lstm(input_shape, units=64)
bi_lstm_model.fit(X, y, epochs=50, batch_size=32, validation_split=0.2, verbose=0)

# Make predictions
last_sequence = X[-1:]
next_value = bi_lstm_model.predict(last_sequence)
print(f"Predicted value: {next_value[0][0]:.2f}")

Additional Resources

For those interested in diving deeper into LSTM networks and their applications in time series analysis, here are some valuable resources:

1. "Long Short-Term Memory" by Sepp Hochreiter and Jürgen Schmidhuber (1997) - The original paper introducing LSTM networks. Available at: https://www.bioinf.jku.at/publications/older/2604.pdf
2. "LSTM Networks for Sentiment Analysis" by Christopher Olah - A comprehensive blog post explaining LSTM networks with visualizations. Available at: https://colah.github.io/posts/2015-08-Understanding-LSTMs/
3. "Sequence Models" course by Andrew Ng on Coursera - Part of the Deep Learning Specialization, this course covers RNNs and LSTMs in depth.
4. "Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow" by Aurélien Géron - A practical guide to implementing machine learning algorithms, including LSTMs for time series.
5. ArXiv paper: "A Review of Long Short-Term Memory Networks" by Lipton et al. (2015) - A comprehensive review of LSTM networks and their applications. ArXiv URL: https://arxiv.org/abs/1503.04069

These resources provide a mix of theoretical foundations and practical implementations to help deepen your understanding of LSTM networks and their applications in time series analysis.