xbe.at

Deep learning / RNN LSTM and sequence

Bidirectional RNN in Deep Learning with Python

12 min read, 15 sections

Contents
  1. Introduction to Bidirectional RNNs
  2. Architecture of Bidirectional RNNs
  3. Advantages of Bidirectional RNNs
  4. Implementing BiRNN with TensorFlow
  5. Training the BiRNN Model
  6. BiRNN vs Unidirectional RNN: Performance Comparison
  7. Real-Life Example: Named Entity Recognition
  8. Training and Using the NER Model
  9. Real-Life Example: Sentiment Analysis
  10. Using the Sentiment Analysis Model
  11. Handling Long-Term Dependencies in BiRNNs
  12. Visualizing Attention Weights
  13. Comparison of BiRNN Architectures
  14. BiRNN for Time Series Forecasting
  15. Additional Resources

Introduction to Bidirectional RNNs

Bidirectional Recurrent Neural Networks (BiRNNs) are an extension of traditional RNNs that process sequences in both forward and backward directions. This allows the network to capture context from both past and future states, leading to improved performance in many sequence-based tasks.

import tensorflow as tf

# Creating a simple Bidirectional RNN
bidirectional_rnn = tf.keras.layers.Bidirectional(
    tf.keras.layers.SimpleRNN(64, return_sequences=True)
)

Architecture of Bidirectional RNNs

BiRNNs consist of two separate RNNs: one processes the input sequence from left to right (forward), while the other processes it from right to left (backward). The outputs of both RNNs are typically concatenated or summed to produce the final output.

import numpy as np
import matplotlib.pyplot as plt

def visualize_birnn():
    fig, ax = plt.subplots(figsize=(10, 5))
    ax.set_xlim(0, 10)
    ax.set_ylim(0, 5)
    ax.axis('off')
    
    # Draw forward and backward arrows
    ax.arrow(1, 2.5, 8, 0, head_width=0.2, head_length=0.3, fc='b', ec='b')
    ax.arrow(9, 1.5, -8, 0, head_width=0.2, head_length=0.3, fc='r', ec='r')
    
    # Add text labels
    ax.text(5, 3, 'Forward RNN', ha='center', fontsize=12)
    ax.text(5, 1, 'Backward RNN', ha='center', fontsize=12)
    ax.text(0.5, 2, 'Input\nSequence', ha='center', va='center', fontsize=10)
    ax.text(9.5, 2, 'Output', ha='center', va='center', fontsize=10)
    
    plt.show()

visualize_birnn()

Advantages of Bidirectional RNNs

BiRNNs offer several advantages over unidirectional RNNs. They can capture both past and future context, which is crucial for tasks like named entity recognition, machine translation, and sentiment analysis. This bidirectional processing often leads to more accurate predictions and better handling of long-range dependencies.

import tensorflow as tf

# Compare unidirectional and bidirectional RNNs
uni_rnn = tf.keras.layers.SimpleRNN(64, return_sequences=True)
bi_rnn = tf.keras.layers.Bidirectional(tf.keras.layers.SimpleRNN(64, return_sequences=True))

# Example input
x = tf.random.normal((1, 10, 32))  # (batch_size, time_steps, features)

uni_output = uni_rnn(x)
bi_output = bi_rnn(x)

print(f"Unidirectional output shape: {uni_output.shape}")
print(f"Bidirectional output shape: {bi_output.shape}")

Implementing BiRNN with TensorFlow

Let's implement a simple BiRNN model using TensorFlow for a text classification task. We'll use the IMDB movie review dataset as an example.

import tensorflow as tf
from tensorflow.keras.datasets import imdb
from tensorflow.keras.preprocessing.sequence import pad_sequences

# Load and preprocess the data
max_features = 10000
maxlen = 200

(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=max_features)
x_train = pad_sequences(x_train, maxlen=maxlen)
x_test = pad_sequences(x_test, maxlen=maxlen)

# Build the model
model = tf.keras.Sequential([
    tf.keras.layers.Embedding(max_features, 128, input_length=maxlen),
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64, return_sequences=True)),
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(32)),
    tf.keras.layers.Dense(1, activation='sigmoid')
])

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
model.summary()

Training the BiRNN Model

Now that we have defined our BiRNN model, let's train it on the IMDB dataset and evaluate its performance.

# Train the model
history = model.fit(x_train, y_train,
                    epochs=5,
                    batch_size=128,
                    validation_split=0.2,
                    verbose=1)

# Evaluate the model
test_loss, test_acc = model.evaluate(x_test, y_test, verbose=0)
print(f"Test accuracy: {test_acc:.4f}")

# Plot training history
plt.figure(figsize=(12, 4))
plt.subplot(1, 2, 1)
plt.plot(history.history['accuracy'], label='Train')
plt.plot(history.history['val_accuracy'], label='Validation')
plt.title('Model Accuracy')
plt.xlabel('Epoch')
plt.ylabel('Accuracy')
plt.legend()

plt.subplot(1, 2, 2)
plt.plot(history.history['loss'], label='Train')
plt.plot(history.history['val_loss'], label='Validation')
plt.title('Model Loss')
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.legend()

plt.tight_layout()
plt.show()

BiRNN vs Unidirectional RNN: Performance Comparison

To better understand the advantages of BiRNNs, let's compare its performance with a unidirectional RNN on the same task.

# Unidirectional RNN model
uni_model = tf.keras.Sequential([
    tf.keras.layers.Embedding(max_features, 128, input_length=maxlen),
    tf.keras.layers.LSTM(64, return_sequences=True),
    tf.keras.layers.LSTM(32),
    tf.keras.layers.Dense(1, activation='sigmoid')
])

uni_model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

# Train unidirectional model
uni_history = uni_model.fit(x_train, y_train,
                            epochs=5,
                            batch_size=128,
                            validation_split=0.2,
                            verbose=0)

# Evaluate unidirectional model
uni_test_loss, uni_test_acc = uni_model.evaluate(x_test, y_test, verbose=0)

print(f"BiRNN Test accuracy: {test_acc:.4f}")
print(f"Unidirectional RNN Test accuracy: {uni_test_acc:.4f}")

# Plot comparison
plt.figure(figsize=(10, 5))
plt.plot(history.history['val_accuracy'], label='BiRNN')
plt.plot(uni_history.history['val_accuracy'], label='Unidirectional RNN')
plt.title('Validation Accuracy: BiRNN vs Unidirectional RNN')
plt.xlabel('Epoch')
plt.ylabel('Accuracy')
plt.legend()
plt.show()

Real-Life Example: Named Entity Recognition

Named Entity Recognition (NER) is a task where BiRNNs excel due to their ability to capture context from both directions. Let's implement a simple BiRNN-based NER model.

import numpy as np
from tensorflow.keras.preprocessing.sequence import pad_sequences

# Simulated NER data
sentences = [
    "John lives in New York",
    "Apple is headquartered in Cupertino",
    "The Eiffel Tower is in Paris"
]
labels = [
    ["B-PER", "O", "O", "B-LOC", "I-LOC"],
    ["B-ORG", "O", "O", "O", "B-LOC"],
    ["B-LOC", "I-LOC", "I-LOC", "O", "O", "B-LOC"]
]

# Create vocabularies
words = set([word for sentence in sentences for word in sentence.split()])
word_to_index = {word: index for index, word in enumerate(words)}
tag_to_index = {"O": 0, "B-PER": 1, "I-PER": 2, "B-LOC": 3, "I-LOC": 4, "B-ORG": 5, "I-ORG": 6}

# Encode sentences and labels
X = [[word_to_index[word] for word in sentence.split()] for sentence in sentences]
y = [[tag_to_index[tag] for tag in sentence_labels] for sentence_labels in labels]

# Pad sequences
max_len = max(len(sentence) for sentence in X)
X_padded = pad_sequences(X, maxlen=max_len, padding='post')
y_padded = pad_sequences(y, maxlen=max_len, padding='post', value=tag_to_index["O"])

# Convert to one-hot encoded labels
y_one_hot = tf.keras.utils.to_categorical(y_padded)

# Build the BiRNN model for NER
ner_model = tf.keras.Sequential([
    tf.keras.layers.Embedding(len(word_to_index), 64, input_length=max_len),
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(32, return_sequences=True)),
    tf.keras.layers.TimeDistributed(tf.keras.layers.Dense(len(tag_to_index), activation='softmax'))
])

ner_model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
ner_model.summary()

Training and Using the NER Model

Now let's train our NER model and use it to make predictions on new sentences.

# Train the NER model
ner_history = ner_model.fit(X_padded, y_one_hot, epochs=50, batch_size=2, verbose=0)

# Function to predict entities in a new sentence
def predict_entities(sentence):
    words = sentence.split()
    X_new = pad_sequences([[word_to_index.get(word, 0) for word in words]], maxlen=max_len, padding='post')
    y_pred = ner_model.predict(X_new)[0]
    pred_tags = [list(tag_to_index.keys())[np.argmax(pred)] for pred in y_pred[:len(words)]]
    return list(zip(words, pred_tags))

# Test the model
test_sentence = "Emma visited London last summer"
result = predict_entities(test_sentence)
print("Predicted entities:")
for word, tag in result:
    print(f"{word}: {tag}")

# Plot training history
plt.figure(figsize=(10, 5))
plt.plot(ner_history.history['accuracy'], label='Accuracy')
plt.title('NER Model Training Accuracy')
plt.xlabel('Epoch')
plt.ylabel('Accuracy')
plt.legend()
plt.show()

Real-Life Example: Sentiment Analysis

Sentiment analysis is another task where BiRNNs can be particularly effective. Let's implement a BiRNN-based sentiment analysis model using a different dataset.

import tensorflow_datasets as tfds

# Load the IMDB reviews dataset
(train_data, test_data), info = tfds.load(
    'imdb_reviews/subwords8k',
    split=(tfds.Split.TRAIN, tfds.Split.TEST),
    as_supervised=True,
    with_info=True)

encoder = info.features['text'].encoder

# Prepare the data
BUFFER_SIZE = 10000
BATCH_SIZE = 64

train_dataset = train_data.shuffle(BUFFER_SIZE)
train_dataset = train_dataset.padded_batch(BATCH_SIZE)
test_dataset = test_data.padded_batch(BATCH_SIZE)

# Build the sentiment analysis model
model = tf.keras.Sequential([
    tf.keras.layers.Embedding(encoder.vocab_size, 64),
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64, return_sequences=True)),
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(32)),
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dense(1, activation='sigmoid')
])

model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
model.summary()

# Train the model
history = model.fit(train_dataset, epochs=10, validation_data=test_dataset, validation_steps=30)

# Plot training history
plt.figure(figsize=(12, 4))
plt.subplot(1, 2, 1)
plt.plot(history.history['accuracy'], label='Train')
plt.plot(history.history['val_accuracy'], label='Validation')
plt.title('Model Accuracy')
plt.xlabel('Epoch')
plt.ylabel('Accuracy')
plt.legend()

plt.subplot(1, 2, 2)
plt.plot(history.history['loss'], label='Train')
plt.plot(history.history['val_loss'], label='Validation')
plt.title('Model Loss')
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.legend()

plt.tight_layout()
plt.show()

Using the Sentiment Analysis Model

Now that we have trained our sentiment analysis model, let's use it to predict the sentiment of some example reviews.

def predict_sentiment(text):
    encoded_text = encoder.encode(text)
    padded_text = tf.keras.preprocessing.sequence.pad_sequences([encoded_text], maxlen=200, padding='post')
    prediction = model.predict(padded_text)[0][0]
    return "Positive" if prediction > 0.5 else "Negative", prediction

# Example reviews
reviews = [
    "This movie was fantastic! I really enjoyed every minute of it.",
    "Terrible acting, poor plot, and awful cinematography. Don't waste your time.",
    "The film had its moments, but overall it was just average.",
    "I can't decide if I liked it or not. It was quite confusing."
]

for review in reviews:
    sentiment, score = predict_sentiment(review)
    print(f"Review: {review}")
    print(f"Predicted sentiment: {sentiment} (Score: {score:.2f})")
    print()

# Visualize sentiment scores
sentiments, scores = zip(*[predict_sentiment(review) for review in reviews])
plt.figure(figsize=(10, 5))
plt.bar(range(len(reviews)), scores, color=['g' if s == "Positive" else 'r' for s in sentiments])
plt.axhline(y=0.5, color='b', linestyle='--')
plt.title('Sentiment Scores for Example Reviews')
plt.xlabel('Review')
plt.ylabel('Sentiment Score')
plt.ylim(0, 1)
plt.xticks(range(len(reviews)), [f"Review {i+1}" for i in range(len(reviews))], rotation=45)
plt.tight_layout()
plt.show()

Handling Long-Term Dependencies in BiRNNs

While BiRNNs are better at capturing context than unidirectional RNNs, they can still struggle with very long sequences. Attention mechanisms can help address this issue by allowing the model to focus on relevant parts of the input sequence.

import tensorflow as tf

class AttentionLayer(tf.keras.layers.Layer):
    def __init__(self, units):
        super(AttentionLayer, self).__init__()
        self.W1 = tf.keras.layers.Dense(units)
        self.W2 = tf.keras.layers.Dense(units)
        self.V = tf.keras.layers.Dense(1)

    def call(self, query, values):
        query_with_time_axis = tf.expand_dims(query, 1)
        score = self.V(tf.nn.tanh(self.W1(query_with_time_axis) + self.W2(values)))
        attention_weights = tf.nn.softmax(score, axis=1)
        context_vector = attention_weights * values
        context_vector = tf.reduce_sum(context_vector, axis=1)
        return context_vector, attention_weights

# Usage in a model
inputs = tf.keras.Input(shape=(max_length,))
embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim)(inputs)
bi_lstm = tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64, return_sequences=True))(embedding)
query = tf.keras.layers.GlobalAveragePooling1D()(bi_lstm)
attn_out, _ = AttentionLayer(64)(query, bi_lstm)
outputs = tf.keras.layers.Dense(1, activation='sigmoid')(attn_out)
model = tf.keras.Model(inputs=inputs, outputs=outputs)

Visualizing Attention Weights

Attention mechanisms not only improve performance but also provide interpretability. Let's visualize attention weights to understand which parts of the input sequence the model focuses on.

import matplotlib.pyplot as plt

def visualize_attention(sentence, attention_weights):
    fig = plt.figure(figsize=(10, 5))
    ax = fig.add_subplot(1, 1, 1)
    ax.matshow(attention_weights, cmap='viridis')
    
    ax.set_xticklabels([''] + sentence, rotation=90)
    ax.set_yticklabels([''])
    
    plt.title("Attention Weights Visualization")
    plt.xlabel("Input Sequence")
    plt.show()

# Example usage (assuming we have a trained model with attention)
sentence = "The movie was great but the ending could have been better"
input_seq = tokenizer.texts_to_sequences([sentence])
input_seq = tf.keras.preprocessing.sequence.pad_sequences(input_seq, maxlen=max_length, padding='post')

_, attention_weights = model(input_seq)
visualize_attention(sentence.split(), attention_weights[0])

Comparison of BiRNN Architectures

Different types of RNN cells can be used in BiRNNs. Let's compare the performance of BiLSTM and BiGRU on a text classification task.

import tensorflow as tf
from tensorflow.keras.datasets import imdb
from tensorflow.keras.preprocessing.sequence import pad_sequences

# Load and preprocess data
max_features = 10000
maxlen = 200
(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=max_features)
x_train = pad_sequences(x_train, maxlen=maxlen)
x_test = pad_sequences(x_test, maxlen=maxlen)

# BiLSTM model
bilstm_model = tf.keras.Sequential([
    tf.keras.layers.Embedding(max_features, 128, input_length=maxlen),
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64)),
    tf.keras.layers.Dense(1, activation='sigmoid')
])

# BiGRU model
bigru_model = tf.keras.Sequential([
    tf.keras.layers.Embedding(max_features, 128, input_length=maxlen),
    tf.keras.layers.Bidirectional(tf.keras.layers.GRU(64)),
    tf.keras.layers.Dense(1, activation='sigmoid')
])

# Compile and train both models
bilstm_model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
bigru_model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

bilstm_history = bilstm_model.fit(x_train, y_train, epochs=5, batch_size=128, validation_split=0.2, verbose=0)
bigru_history = bigru_model.fit(x_train, y_train, epochs=5, batch_size=128, validation_split=0.2, verbose=0)

# Plot comparison
plt.figure(figsize=(10, 5))
plt.plot(bilstm_history.history['val_accuracy'], label='BiLSTM')
plt.plot(bigru_history.history['val_accuracy'], label='BiGRU')
plt.title('Validation Accuracy: BiLSTM vs BiGRU')
plt.xlabel('Epoch')
plt.ylabel('Accuracy')
plt.legend()
plt.show()

BiRNN for Time Series Forecasting

BiRNNs can also be applied to time series forecasting tasks. Let's implement a simple BiRNN model for predicting future values in a time series.

import numpy as np
import tensorflow as tf
import matplotlib.pyplot as plt

# Generate sample time series data
def generate_time_series(batch_size, n_steps):
    freq1, freq2, offsets1, offsets2 = np.random.rand(4, batch_size, 1)
    time = np.linspace(0, 1, n_steps)
    series = 0.5 * np.sin((time - offsets1) * (freq1 * 10 + 10))
    series += 0.2 * np.sin((time - offsets2) * (freq2 * 20 + 20))
    series += 0.1 * (np.random.rand(batch_size, n_steps) - 0.5)
    return series[..., np.newaxis]

n_steps = 50
series = generate_time_series(10000, n_steps + 1)
X_train, y_train = series[:7000, :n_steps], series[:7000, -1]
X_valid, y_valid = series[7000:9000, :n_steps], series[7000:9000, -1]
X_test, y_test = series[9000:, :n_steps], series[9000:, -1]

# Build and train the model
model = tf.keras.Sequential([
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(20, return_sequences=True), input_shape=[None, 1]),
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(20)),
    tf.keras.layers.Dense(1)
])

model.compile(loss="mse", optimizer="adam")
history = model.fit(X_train, y_train, epochs=20, validation_data=(X_valid, y_valid))

# Plot results
plt.figure(figsize=(10, 6))
plt.plot(history.history['loss'], label='Training Loss')
plt.plot(history.history['val_loss'], label='Validation Loss')
plt.title('Model Loss')
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.legend()
plt.show()

# Make predictions
X_new = generate_time_series(1, n_steps)
y_pred = model.predict(X_new)

plt.figure(figsize=(10, 6))
plt.plot(np.arange(n_steps), X_new[0, :, 0], ".-b", label="Input")
plt.plot(n_steps, y_pred[0, 0], "ro", markersize=10, label="Prediction")
plt.plot(n_steps, X_new[0, -1, 0], "go", markersize=10, label="Target")
plt.legend()
plt.title("Time Series Prediction")
plt.xlabel("Time")
plt.ylabel("Value")
plt.show()

Additional Resources

For those interested in diving deeper into Bidirectional RNNs and their applications, here are some valuable resources:

  1. "Bidirectional Recurrent Neural Networks" by Mike Schuster and Kuldip K. Paliwal (1997) ArXiv link: https://arxiv.org/abs/cs/9705103
  2. "Neural Machine Translation by Jointly Learning to Align and Translate" by Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio (2014) ArXiv link: https://arxiv.org/abs/1409.0473
  3. "Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling" by Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio (2014) ArXiv link: https://arxiv.org/abs/1412.3555

These papers provide foundational knowledge and advanced applications of Bidirectional RNNs in various domains of deep learning and natural language processing.

More in RNN LSTM and sequence

  1. Comparing ANN and RNN Architectures and Forward Propagation in RNNs13 min

    Artificial Neural Networks (ANNs) and Recurrent Neural Networks (RNNs) are two fundamental architectures in deep learning. ANNs are designed for static data...

  2. Architecture of Recurrent Neural Networks in Python11 min

    Recurrent Neural Networks are a class of artificial neural networks designed to work with sequential data. Unlike feedforward networks, RNNs have loops that...

  3. Deep Recurrent Neural Networks Mastering Sequential Data12 min

    Deep Recurrent Neural Networks (RNNs) are advanced neural architectures designed to process sequential data by maintaining an internal state or 'memory'....

  4. Deep Recurrent Neural Networks with Keras and Python12 min

    Deep Recurrent Neural Networks (RNNs) are a class of neural networks designed to process sequential data. They have loops that allow information to persist...

  5. Explaining LSTM Networks in Deep Learning with Python12 min

    Long Short-Term Memory (LSTM) networks are a type of recurrent neural network (RNN) designed to address the vanishing gradient problem in traditional RNNs....

  6. Exploring 5 Key Assumptions in Linear Regression12 min

    Recurrent Neural Networks (RNNs) are powerful models for processing sequential data, but they face significant challenges. This slideshow explores these...