xbe.at

NLP and transformers / Transformer architecture

Transformer Encoder Explained

6 min read, 21 sections

Contents
  1. What's the Encoder?
  2. Source Code for What's the Encoder?
  3. Input Embedding
  4. Source Code for Input Embedding
  5. Results for Input Embedding
  6. Positional Encoding
  7. Source Code for Positional Encoding
  8. Results for Positional Encoding
  9. Multi-Headed Self-Attention
  10. Source Code for Multi-Headed Self-Attention
  11. Results for Multi-Headed Self-Attention
  12. Feed-Forward Neural Network
  13. Source Code for Feed-Forward Neural Network
  14. Results for Feed-Forward Neural Network
  15. Real-Life Example: Sentiment Analysis
  16. Source Code for Sentiment Analysis
  17. Results for Sentiment Analysis
  18. Real-Life Example: Machine Translation
  19. Source Code for Machine Translation
  20. Results for Machine Translation
  21. Additional Resources

What's the Encoder?

The Encoder is a crucial component of the Transformer architecture, responsible for processing input tokens and generating context-aware representations. It consists of multiple layers that apply self-attention and feed-forward neural networks to refine the input representations iteratively.

Source Code for What's the Encoder?

import math

class Encoder:
    def __init__(self, num_layers, d_model, num_heads, d_ff):
        self.layers = [EncoderLayer(d_model, num_heads, d_ff) for _ in range(num_layers)]
    
    def forward(self, x):
        for layer in self.layers:
            x = layer(x)
        return x

class EncoderLayer:
    def __init__(self, d_model, num_heads, d_ff):
        self.self_attention = MultiHeadAttention(d_model, num_heads)
        self.feed_forward = FeedForward(d_model, d_ff)
        self.norm1 = LayerNorm(d_model)
        self.norm2 = LayerNorm(d_model)
    
    def forward(self, x):
        x = x + self.self_attention(self.norm1(x))
        x = x + self.feed_forward(self.norm2(x))
        return x

Input Embedding

The first step in the encoder is to embed every input word into a vector of fixed size (typically 512 dimensions). This embedding process occurs only in the bottom-most encoder layer and transforms discrete tokens into continuous vector representations.

Source Code for Input Embedding

import torch
import torch.nn as nn

class InputEmbedding(nn.Module):
    def __init__(self, vocab_size, d_model):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, d_model)
        self.d_model = d_model
    
    def forward(self, x):
        return self.embedding(x) * math.sqrt(self.d_model)

# Example usage
vocab_size = 10000
d_model = 512
embedding_layer = InputEmbedding(vocab_size, d_model)

input_tokens = torch.tensor([1, 2, 3, 4, 5])
embedded_tokens = embedding_layer(input_tokens)
print(f"Input shape: {input_tokens.shape}")
print(f"Embedded shape: {embedded_tokens.shape}")

Results for Input Embedding

Input shape: torch.Size([5])
Embedded shape: torch.Size([5, 512])

Positional Encoding

Transformers lack recurrence, so they use positional encodings to indicate token positions. A combination of sine and cosine functions allows the model to understand the order of words in a sentence. These encodings are added to the input embeddings to provide positional information.

Source Code for Positional Encoding

import numpy as np

def positional_encoding(max_seq_len, d_model):
    pos_enc = np.zeros((max_seq_len, d_model))
    for pos in range(max_seq_len):
        for i in range(0, d_model, 2):
            pos_enc[pos, i] = np.sin(pos / (10000 ** ((2 * i) / d_model)))
            pos_enc[pos, i + 1] = np.cos(pos / (10000 ** ((2 * (i + 1)) / d_model)))
    return pos_enc

# Example usage
max_seq_len = 10
d_model = 512
pos_enc = positional_encoding(max_seq_len, d_model)
print(f"Positional encoding shape: {pos_enc.shape}")
print(f"First two dimensions for position 0: {pos_enc[0, :2]}")
print(f"First two dimensions for position 5: {pos_enc[5, :2]}")

Results for Positional Encoding

Positional encoding shape: (10, 512)
First two dimensions for position 0: [0.         1.        ]
First two dimensions for position 5: [ 0.84147098 -0.54030231]

Multi-Headed Self-Attention

Self-attention allows the model to relate each word to every other word in the input. The process involves calculating attention scores based on Query, Key, and Value vectors. This operation is performed multiple times in parallel, called "heads," to capture different aspects of the relationships between words.

Source Code for Multi-Headed Self-Attention

import numpy as np

class MultiHeadAttention:
    def __init__(self, d_model, num_heads):
        self.d_model = d_model
        self.num_heads = num_heads
        self.d_k = d_model // num_heads
        
        self.W_q = np.random.randn(d_model, d_model)
        self.W_k = np.random.randn(d_model, d_model)
        self.W_v = np.random.randn(d_model, d_model)
        self.W_o = np.random.randn(d_model, d_model)
    
    def split_heads(self, x):
        batch_size, seq_len, _ = x.shape
        return x.reshape(batch_size, seq_len, self.num_heads, self.d_k).transpose(0, 2, 1, 3)
    
    def attention(self, q, k, v, mask=None):
        scores = np.matmul(q, k.transpose(-2, -1)) / np.sqrt(self.d_k)
        if mask is not None:
            scores = np.where(mask == 0, -1e9, scores)
        attention_weights = np.exp(scores) / np.sum(np.exp(scores), axis=-1, keepdims=True)
        return np.matmul(attention_weights, v)
    
    def forward(self, x, mask=None):
        q = np.matmul(x, self.W_q)
        k = np.matmul(x, self.W_k)
        v = np.matmul(x, self.W_v)
        
        q = self.split_heads(q)
        k = self.split_heads(k)
        v = self.split_heads(v)
        
        attn_output = self.attention(q, k, v, mask)
        attn_output = attn_output.transpose(0, 2, 1, 3).reshape(-1, x.shape[1], self.d_model)
        
        return np.matmul(attn_output, self.W_o)

# Example usage
d_model = 512
num_heads = 8
mha = MultiHeadAttention(d_model, num_heads)

x = np.random.randn(1, 10, d_model)  # Batch size 1, sequence length 10
output = mha.forward(x)
print(f"Input shape: {x.shape}")
print(f"Output shape: {output.shape}")

Results for Multi-Headed Self-Attention

Input shape: (1, 10, 512)
Output shape: (1, 10, 512)

Feed-Forward Neural Network

After the self-attention layer, each position in the sequence is processed independently through a feed-forward neural network. This network typically consists of two linear transformations with a ReLU activation in between, allowing the model to introduce non-linearity and further refine the representations.

Source Code for Feed-Forward Neural Network

import numpy as np

class FeedForward:
    def __init__(self, d_model, d_ff):
        self.w1 = np.random.randn(d_model, d_ff)
        self.b1 = np.zeros(d_ff)
        self.w2 = np.random.randn(d_ff, d_model)
        self.b2 = np.zeros(d_model)
    
    def relu(self, x):
        return np.maximum(0, x)
    
    def forward(self, x):
        hidden = self.relu(np.dot(x, self.w1) + self.b1)
        return np.dot(hidden, self.w2) + self.b2

# Example usage
d_model = 512
d_ff = 2048
ff_layer = FeedForward(d_model, d_ff)

x = np.random.randn(1, 10, d_model)  # Batch size 1, sequence length 10
output = ff_layer.forward(x)
print(f"Input shape: {x.shape}")
print(f"Output shape: {output.shape}")

Results for Feed-Forward Neural Network

Input shape: (1, 10, 512)
Output shape: (1, 10, 512)

Real-Life Example: Sentiment Analysis

Let's consider a practical application of the Transformer encoder for sentiment analysis. In this example, we'll use a simplified encoder to process a movie review and classify its sentiment.

Source Code for Sentiment Analysis

import numpy as np

class SimplifiedEncoder:
    def __init__(self, vocab_size, d_model):
        self.embedding = np.random.randn(vocab_size, d_model)
        self.attention = MultiHeadAttention(d_model, num_heads=4)
        self.ff = FeedForward(d_model, d_ff=1024)
    
    def forward(self, tokens):
        x = self.embedding[tokens]
        x = self.attention.forward(x)
        x = self.ff.forward(x)
        return x.mean(axis=1)  # Average pooling

# Simplified sentiment classifier
def classify_sentiment(encoder_output):
    w = np.random.randn(512, 2)  # Binary classification: positive/negative
    logits = np.dot(encoder_output, w)
    return np.argmax(logits, axis=1)

# Example usage
vocab_size = 10000
d_model = 512
encoder = SimplifiedEncoder(vocab_size, d_model)

# Tokenized movie review (simplified)
review = np.array([42, 1337, 7, 42, 1337, 7, 42, 1337, 7])
encoded_review = encoder.forward(review)
sentiment = classify_sentiment(encoded_review)

print(f"Encoded review shape: {encoded_review.shape}")
print(f"Predicted sentiment: {'Positive' if sentiment[0] == 1 else 'Negative'}")

Results for Sentiment Analysis

Encoded review shape: (1, 512)
Predicted sentiment: Positive

Real-Life Example: Machine Translation

Another common application of the Transformer encoder is machine translation. Here's a simplified example of how the encoder could be used in a translation system.

Source Code for Machine Translation

import numpy as np

class TranslationEncoder(SimplifiedEncoder):
    def __init__(self, vocab_size, d_model):
        super().__init__(vocab_size, d_model)
        self.language_embedding = np.random.randn(2, d_model)  # Source and target language embeddings
    
    def forward(self, tokens, language_id):
        x = self.embedding[tokens] + self.language_embedding[language_id]
        x = self.attention.forward(x)
        x = self.ff.forward(x)
        return x

# Simplified translation decoder (just for demonstration)
def translate(encoder_output):
    # In a real system, this would be another Transformer stack (the decoder)
    w = np.random.randn(512, vocab_size)
    logits = np.dot(encoder_output, w)
    return np.argmax(logits, axis=-1)

# Example usage
vocab_size = 10000
d_model = 512
encoder = TranslationEncoder(vocab_size, d_model)

# Tokenized sentence in source language (simplified)
sentence = np.array([42, 1337, 7, 42, 1337, 7, 42, 1337, 7])
language_id = 0  # 0 for source language, 1 for target language
encoded_sentence = encoder.forward(sentence, language_id)
translation = translate(encoded_sentence)

print(f"Encoded sentence shape: {encoded_sentence.shape}")
print(f"Translated sentence (token IDs): {translation}")

Results for Machine Translation

Encoded sentence shape: (9, 512)
Translated sentence (token IDs): [3845 6201 9321 4562 1078 7890 2345 6789 1234]

Additional Resources

For more in-depth information on the Transformer architecture and its encoder component, consider the following resources:

  1. Original Transformer paper: "Attention Is All You Need" by Vaswani et al. (2017) ArXiv: https://arxiv.org/abs/1706.03762
  2. "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" by Devlin et al. (2018) ArXiv: https://arxiv.org/abs/1810.04805
  3. "The Illustrated Transformer" by Jay Alammar http://jalammar.github.io/illustrated-transformer/

These resources provide comprehensive explanations and visualizations of the Transformer architecture, including detailed discussions of the encoder component.

More in Transformer architecture

  1. Transformer Encoder Processing Input Tokens8 min

    The embedding layer transforms input tokens into dense vector representations of dimension d\model (typically 512). This crucial first step maps discrete...

  2. Transformer Architecture for Natural Language Processing7 min

    The Transformer architecture revolutionized natural language processing by introducing self-attention mechanisms, enabling parallel processing of sequential...

  3. Transformer Model Explained with Python10 min

    The Transformer model, introduced in the "Attention Is All You Need" paper, revolutionized natural language processing. It's a neural network architecture...

  4. Simplifying Transformer Models with Python12 min

    The tokenization process is fundamental to transformer models, converting raw text into numerical tokens that can be processed. This implementation...

  5. Transformer Model Workflow From Instantiation to Optimization Using Python5 min

    Understanding how to instantiate a Transformer model is the first step in working with these powerful architectures. We'll use the Hugging Face Transformers...

  6. Simplifying Transformer Architecture for Beginners12 min

    The transformer architecture revolutionized natural language processing by introducing self-attention mechanisms and parallel processing capabilities. This...