xbe.at

LLMs and generative AI / LLM architecture and theory

Understanding Encoders, Decoders, and Encoder-Decoder LLMs

9 min read, 14 sections

Contents
  1. Introduction to Encoders, Decoders, and Encoder-Decoder LLMs
  2. Encoders in Detail
  3. Decoders Explained
  4. Encoder-Decoder Architecture
  5. Attention Mechanism
  6. Transformer Architecture
  7. BERT: Bidirectional Encoder Representations from Transformers
  8. GPT: Generative Pre-trained Transformer
  9. Real-life Example: Machine Translation
  10. Real-life Example: Text Summarization
  11. Comparing Encoder-only, Decoder-only, and Encoder-Decoder Models
  12. Fine-tuning Pre-trained Models
  13. Challenges and Future Directions
  14. Additional Resources

Introduction to Encoders, Decoders, and Encoder-Decoder LLMs

Encoders, decoders, and encoder-decoder models are fundamental components in natural language processing and machine learning. These architectures form the basis for many language models, including Large Language Models (LLMs). Let's explore their unique characteristics and applications using Python examples.

import torch
import torch.nn as nn

# Simple encoder class
class Encoder(nn.Module):
    def __init__(self, input_size, hidden_size):
        super(Encoder, self).__init__()
        self.hidden_size = hidden_size
        self.embedding = nn.Embedding(input_size, hidden_size)
        self.gru = nn.GRU(hidden_size, hidden_size)

    def forward(self, input):
        embedded = self.embedding(input).view(1, 1, -1)
        output, hidden = self.gru(embedded)
        return output, hidden

# Usage
input_size = 10
hidden_size = 20
encoder = Encoder(input_size, hidden_size)
input_tensor = torch.tensor([5])  # Example input
output, hidden = encoder(input_tensor)
print(f"Encoder output shape: {output.shape}")
print(f"Encoder hidden state shape: {hidden.shape}")

Encoders in Detail

Encoders are neural network components that transform input data into a compact representation or "encoding." They capture essential features of the input, reducing dimensionality while preserving important information. In natural language processing, encoders often process sequences of words or tokens.

import torch
import torch.nn as nn

class EncoderRNN(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers=1):
        super(EncoderRNN, self).__init__()
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        
        self.embedding = nn.Embedding(input_size, hidden_size)
        self.gru = nn.GRU(hidden_size, hidden_size, num_layers, batch_first=True)
    
    def forward(self, input):
        embedded = self.embedding(input)
        output, hidden = self.gru(embedded)
        return output, hidden

# Example usage
vocab_size = 10000
hidden_size = 256
encoder = EncoderRNN(vocab_size, hidden_size)

# Simulate a batch of 3 sentences, each with 5 words
input_seq = torch.randint(0, vocab_size, (3, 5))
output, hidden = encoder(input_seq)

print(f"Encoder output shape: {output.shape}")
print(f"Encoder hidden state shape: {hidden.shape}")

Decoders Explained

Decoders take the encoded representation and generate output sequences. They are crucial in tasks like machine translation, text summarization, and image captioning. Decoders often use attention mechanisms to focus on relevant parts of the input during generation.

import torch
import torch.nn as nn
import torch.nn.functional as F

class DecoderRNN(nn.Module):
    def __init__(self, hidden_size, output_size):
        super(DecoderRNN, self).__init__()
        self.hidden_size = hidden_size
        
        self.embedding = nn.Embedding(output_size, hidden_size)
        self.gru = nn.GRU(hidden_size, hidden_size)
        self.out = nn.Linear(hidden_size, output_size)
    
    def forward(self, input, hidden):
        output = self.embedding(input).view(1, 1, -1)
        output = F.relu(output)
        output, hidden = self.gru(output, hidden)
        output = self.out(output[0])
        return output, hidden

# Example usage
hidden_size = 256
vocab_size = 10000
decoder = DecoderRNN(hidden_size, vocab_size)

# Simulate decoding a single token
input_token = torch.tensor([[42]])  # Example token ID
hidden_state = torch.randn(1, 1, hidden_size)  # Initial hidden state

output, new_hidden = decoder(input_token, hidden_state)
print(f"Decoder output shape: {output.shape}")
print(f"Decoder new hidden state shape: {new_hidden.shape}")

Encoder-Decoder Architecture

The encoder-decoder architecture combines both components to handle sequence-to-sequence tasks. The encoder processes the input sequence, and the decoder generates the output sequence based on the encoder's representation. This architecture is the foundation for many advanced language models.

import torch
import torch.nn as nn

class EncoderDecoder(nn.Module):
    def __init__(self, encoder, decoder):
        super(EncoderDecoder, self).__init__()
        self.encoder = encoder
        self.decoder = decoder
    
    def forward(self, src, tgt, teacher_forcing_ratio=0.5):
        batch_size = src.size(0)
        max_len = tgt.size(1)
        vocab_size = self.decoder.output_size
        
        outputs = torch.zeros(batch_size, max_len, vocab_size).to(src.device)
        
        encoder_output, hidden = self.encoder(src)
        
        decoder_input = tgt[:, 0]
        
        for t in range(1, max_len):
            output, hidden = self.decoder(decoder_input, hidden)
            outputs[:, t] = output
            teacher_force = torch.rand(1).item() < teacher_forcing_ratio
            decoder_input = tgt[:, t] if teacher_force else output.argmax(1)
        
        return outputs

# Example usage
encoder = EncoderRNN(input_vocab_size, hidden_size)
decoder = DecoderRNN(hidden_size, output_vocab_size)
model = EncoderDecoder(encoder, decoder)

src_seq = torch.randint(0, input_vocab_size, (batch_size, src_len))
tgt_seq = torch.randint(0, output_vocab_size, (batch_size, tgt_len))

output = model(src_seq, tgt_seq)
print(f"Output shape: {output.shape}")

Attention Mechanism

Attention mechanisms allow decoder models to focus on different parts of the input sequence when generating each output token. This significantly improves the performance of encoder-decoder models, especially for long sequences.

import torch
import torch.nn as nn
import torch.nn.functional as F

class Attention(nn.Module):
    def __init__(self, hidden_size):
        super(Attention, self).__init__()
        self.hidden_size = hidden_size
        self.attn = nn.Linear(hidden_size * 2, hidden_size)
        self.v = nn.Parameter(torch.rand(hidden_size))

    def forward(self, hidden, encoder_outputs):
        batch_size = encoder_outputs.size(0)
        seq_len = encoder_outputs.size(1)
        
        hidden = hidden.repeat(seq_len, 1, 1).transpose(0, 1)
        energy = torch.tanh(self.attn(torch.cat((hidden, encoder_outputs), dim=2)))
        
        v = self.v.repeat(batch_size, 1).unsqueeze(1)
        attention = torch.bmm(v, energy.transpose(1, 2)).squeeze(1)
        
        return F.softmax(attention, dim=1)

# Usage example
hidden_size = 256
attention = Attention(hidden_size)

hidden = torch.randn(1, 1, hidden_size)
encoder_outputs = torch.randn(1, 10, hidden_size)

attn_weights = attention(hidden, encoder_outputs)
print(f"Attention weights shape: {attn_weights.shape}")

Transformer Architecture

Transformers, introduced in the "Attention is All You Need" paper, revolutionized NLP by replacing recurrent layers with self-attention mechanisms. They form the basis for many modern LLMs, including GPT and BERT.

import torch
import torch.nn as nn

class TransformerEncoder(nn.Module):
    def __init__(self, vocab_size, d_model, nhead, num_layers):
        super(TransformerEncoder, self).__init__()
        self.embedding = nn.Embedding(vocab_size, d_model)
        self.pos_encoder = PositionalEncoding(d_model)
        encoder_layers = nn.TransformerEncoderLayer(d_model, nhead)
        self.transformer_encoder = nn.TransformerEncoder(encoder_layers, num_layers)
        
    def forward(self, src):
        src = self.embedding(src) * math.sqrt(self.d_model)
        src = self.pos_encoder(src)
        output = self.transformer_encoder(src)
        return output

# Note: PositionalEncoding class is not implemented here for brevity
# Usage example
vocab_size = 10000
d_model = 512
nhead = 8
num_layers = 6

model = TransformerEncoder(vocab_size, d_model, nhead, num_layers)
src = torch.randint(0, vocab_size, (20, 32))  # (seq_len, batch_size)
output = model(src)
print(f"Transformer output shape: {output.shape}")

BERT: Bidirectional Encoder Representations from Transformers

BERT is a transformer-based model that uses bidirectional training of transformer, allowing it to learn contextual relations between words in both directions. It's pre-trained on a large corpus of unlabeled text and can be fine-tuned for various NLP tasks.

from transformers import BertTokenizer, BertModel
import torch

# Load pre-trained model and tokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained('bert-base-uncased')

# Prepare input
text = "Hello, how are you?"
encoded_input = tokenizer(text, return_tensors='pt')

# Forward pass
with torch.no_grad():
    output = model(**encoded_input)

# Access the output
last_hidden_states = output.last_hidden_state
print(f"BERT output shape: {last_hidden_states.shape}")

GPT: Generative Pre-trained Transformer

GPT models are autoregressive language models that use transformer decoder architecture. They are trained to predict the next token given the previous tokens and have shown remarkable performance in various language tasks.

from transformers import GPT2LMHeadModel, GPT2Tokenizer
import torch

# Load pre-trained model and tokenizer
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2LMHeadModel.from_pretrained('gpt2')

# Prepare input
text = "Once upon a time"
input_ids = tokenizer.encode(text, return_tensors='pt')

# Generate text
output = model.generate(input_ids, max_length=50, num_return_sequences=1)

# Decode and print the generated text
generated_text = tokenizer.decode(output[0], skip_special_tokens=True)
print(f"Generated text: {generated_text}")

Real-life Example: Machine Translation

Machine translation is a common application of encoder-decoder models. Let's implement a simple English to French translation model using PyTorch.

import torch
import torch.nn as nn
import torch.optim as optim

class Seq2Seq(nn.Module):
    def __init__(self, encoder, decoder):
        super().__init__()
        self.encoder = encoder
        self.decoder = decoder
    
    def forward(self, src, tgt, teacher_forcing_ratio=0.5):
        batch_size = src.shape[1]
        tgt_len = tgt.shape[0]
        tgt_vocab_size = self.decoder.output_dim
        
        outputs = torch.zeros(tgt_len, batch_size, tgt_vocab_size).to(src.device)
        encoder_outputs, hidden = self.encoder(src)
        
        input = tgt[0,:]
        for t in range(1, tgt_len):
            output, hidden = self.decoder(input, hidden, encoder_outputs)
            outputs[t] = output
            teacher_force = torch.rand(1).item() < teacher_forcing_ratio
            top1 = output.argmax(1)
            input = tgt[t] if teacher_force else top1
        
        return outputs

# Usage (pseudo-code, as actual implementation would require more setup)
# encoder = Encoder(input_dim, emb_dim, hid_dim, n_layers, dropout)
# decoder = Decoder(output_dim, emb_dim, hid_dim, n_layers, dropout)
# model = Seq2Seq(encoder, decoder)

# Train the model
# criterion = nn.CrossEntropyLoss()
# optimizer = optim.Adam(model.parameters())

# for epoch in range(num_epochs):
#     for src, tgt in data_loader:
#         optimizer.zero_grad()
#         output = model(src, tgt)
#         loss = criterion(output[1:].view(-1, output.shape[-1]), tgt[1:].view(-1))
#         loss.backward()
#         optimizer.step()

# Translate
# src_sentence = "Hello, how are you?"
# translated = model.translate(src_sentence)
# print(f"Translation: {translated}")

Real-life Example: Text Summarization

Text summarization is another application where encoder-decoder models excel. Here's a simple example using the transformers library to perform extractive summarization.

from transformers import pipeline

def summarize_text(text, max_length=150):
    summarizer = pipeline("summarization")
    summary = summarizer(text, max_length=max_length, min_length=30, do_sample=False)
    return summary[0]['summary_text']

# Example usage
long_text = """
Climate change is one of the most pressing issues facing our planet today. It refers to long-term shifts in temperatures and weather patterns, mainly caused by human activities, especially the burning of fossil fuels. These activities release greenhouse gases into the atmosphere, trapping heat and leading to global warming. The effects of climate change are far-reaching and include rising sea levels, more frequent and severe weather events, and disruptions to ecosystems and biodiversity. Addressing climate change requires a global effort to reduce greenhouse gas emissions and transition to sustainable energy sources.
"""

summary = summarize_text(long_text)
print(f"Summary: {summary}")

Comparing Encoder-only, Decoder-only, and Encoder-Decoder Models

Let's compare the characteristics and use cases of different model architectures:

import matplotlib.pyplot as plt
import numpy as np

model_types = ['Encoder-only', 'Decoder-only', 'Encoder-Decoder']
tasks = ['Classification', 'Generation', 'Translation']

performance = np.array([
    [0.9, 0.6, 0.7],  # Encoder-only
    [0.7, 0.9, 0.8],  # Decoder-only
    [0.8, 0.8, 0.9]   # Encoder-Decoder
])

fig, ax = plt.subplots(figsize=(10, 6))
x = np.arange(len(tasks))
width = 0.25

for i in range(len(model_types)):
    ax.bar(x + i*width, performance[i], width, label=model_types[i])

ax.set_ylabel('Performance Score')
ax.set_title('Model Performance Comparison')
ax.set_xticks(x + width)
ax.set_xticklabels(tasks)
ax.legend()

plt.tight_layout()
plt.show()

Fine-tuning Pre-trained Models

Fine-tuning allows us to adapt pre-trained models to specific tasks. Here's an example of fine-tuning a BERT model for sentiment analysis:

from transformers import BertForSequenceClassification, AdamW
from torch.utils.data import DataLoader

# Assume we have a dataset of movie reviews and sentiment labels
# train_dataset = MovieReviewDataset(reviews, labels)
# train_loader = DataLoader(train_dataset, batch_size=16, shuffle=True)

model = BertForSequenceClassification.from_pretrained('bert-base-uncased', num_labels=2)
optimizer = AdamW(model.parameters(), lr=2e-5)

# Training loop
for epoch in range(3):
    for batch in train_loader:
        inputs = batch['input_ids'].to(device)
        labels = batch['labels'].to(device)
        
        outputs = model(inputs, labels=labels)
        loss = outputs.loss
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()

# Use the fine-tuned model for sentiment analysis
# review = "This movie was fantastic!"
# inputs = tokenizer(review, return_tensors='pt').to(device)
# outputs = model(**inputs)
# prediction = torch.argmax(outputs.logits, dim=1)
# print(f"Sentiment: {'Positive' if prediction == 1 else 'Negative'}")

Challenges and Future Directions

As LLMs continue to evolve, researchers are addressing challenges such as:

  1. Reducing computational resources and energy consumption
  2. Improving model interpretability and reducing bias
  3. Enhancing few-shot and zero-shot learning capabilities
  4. Developing more efficient training techniques

Future directions include multimodal models that can process various types of data, and models that can perform complex reasoning tasks.

# Visualization of model size vs. performance trade-off
import matplotlib.pyplot as plt
import numpy as np

model_sizes = np.array([0.1, 1, 10, 100, 1000])  # in billions of parameters
performance = np.array([70, 80, 85, 90, 92])  # hypothetical performance scores

plt.figure(figsize=(10, 6))
plt.semilogx(model_sizes, performance, marker='o')
plt.xlabel('Model Size (Billion Parameters)')
plt.ylabel('Performance Score')
plt.title('Model Size vs. Performance Trade-off')
plt.grid(True)
plt.show()

Additional Resources

For those interested in diving deeper into the world of LLMs and their architectures, here are some valuable resources:

  1. "Attention Is All You Need" paper (Vaswani et al., 2017): https://arxiv.org/abs/1706.03762
  2. "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" (Devlin et al., 2018): https://arxiv.org/abs/1810.04805
  3. "Language Models are Few-Shot Learners" (GPT-3 paper, Brown et al., 2020): https://arxiv.org/abs/2005.14165
  4. "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer" (T5 paper, Raffel et al., 2019): https://arxiv.org/abs/1910.10683

These papers provide in-depth insights into the architectures and techniques that have shaped modern LLMs.

More in LLM architecture and theory

  1. Understanding LLM Benchmarks with Python10 min

    Large Language Models (LLMs) have revolutionized natural language processing. To evaluate their performance, we use benchmarks. This presentation explores...

  2. Transformers and Attention Mechanisms in Large Language Models9 min

    The attention mechanism enables neural networks to selectively focus on specific parts of the input sequence when generating output. This fundamental concept...

  3. Understanding the Transformer Architecture of Large Language Models11 min

    The transformer architecture forms the backbone of modern LLMs, utilizing self-attention mechanisms to process sequential data. This implementation...

  4. Transformer Architecture for Efficient LLM Processing13 min

    The self-attention mechanism calculates attention scores between all pairs of input tokens, enabling the model to weigh the importance of different parts of...

  5. Unlocking Efficiency in Machine Learning A Guide to MLflow and LLMs11 min

    MLflow is an open-source platform designed to manage the entire machine learning lifecycle, from experimentation to deployment. Large Language Models (LLMs)...

  6. The Exponential Growth of Large Language Models12 min

    Neural network pruning aims to reduce model size by systematically removing weights based on their importance. The gradient-based approach evaluates each...