Positional Encoding in Transformer Models

Positional encoding is a crucial component in Transformer models, addressing the inherent lack of sequential information in self-attention mechanisms. It enables these models to understand the order of input elements, which is essential for tasks involving sequential data like natural language processing.

import numpy as np
import matplotlib.pyplot as plt

def positional_encoding(position, d_model):
    angle_rads = np.arange(d_model) / d_model
    angle_rads = np.power(10000, -angle_rads)
    pos_encoding = position * angle_rads
    pos_encoding[:, 0::2] = np.sin(pos_encoding[:, 0::2])
    pos_encoding[:, 1::2] = np.cos(pos_encoding[:, 1::2])
    return pos_encoding

# Generate positional encoding
seq_length, d_model = 100, 512
pos_encoding = positional_encoding(np.arange(seq_length)[:, np.newaxis], d_model)

# Visualize the positional encoding
plt.figure(figsize=(10, 6))
plt.pcolormesh(pos_encoding, cmap='RdBu')
plt.xlabel('Dimension')
plt.ylabel('Position')
plt.colorbar()
plt.title('Positional Encoding')
plt.show()

The Need for Positional Information

Transformer models rely on self-attention mechanisms, which are permutation-invariant. This means they treat input elements independently, disregarding their original order. However, sequence order is often crucial in tasks like language understanding or time series analysis. Positional encoding solves this by injecting position information into the input embeddings.

import torch
import torch.nn as nn

class SelfAttention(nn.Module):
    def __init__(self, embed_size, heads):
        super(SelfAttention, self).__init__()
        self.embed_size = embed_size
        self.heads = heads
        self.head_dim = embed_size // heads

        self.values = nn.Linear(self.head_dim, self.head_dim, bias=False)
        self.keys = nn.Linear(self.head_dim, self.head_dim, bias=False)
        self.queries = nn.Linear(self.head_dim, self.head_dim, bias=False)
        self.fc_out = nn.Linear(heads * self.head_dim, embed_size)

    def forward(self, values, keys, query, mask):
        N = query.shape[0]
        value_len, key_len, query_len = values.shape[1], keys.shape[1], query.shape[1]

        # Split the embedding into self.heads pieces
        values = values.reshape(N, value_len, self.heads, self.head_dim)
        keys = keys.reshape(N, key_len, self.heads, self.head_dim)
        query = query.reshape(N, query_len, self.heads, self.head_dim)

        values = self.values(values)
        keys = self.keys(keys)
        queries = self.queries(query)

        # Attention mechanism
        energy = torch.einsum("nqhd,nkhd->nhqk", [queries, keys])
        if mask is not None:
            energy = energy.masked_fill(mask == 0, float("-1e20"))

        attention = torch.softmax(energy / (self.embed_size ** (1/2)), dim=3)
        out = torch.einsum("nhql,nlhd->nqhd", [attention, values]).reshape(
            N, query_len, self.heads * self.head_dim
        )

        out = self.fc_out(out)
        return out

# Example usage
embed_size = 256
heads = 8
model = SelfAttention(embed_size, heads)
x = torch.randn((32, 10, embed_size))  # (batch_size, seq_length, embed_size)
output = model(x, x, x, mask=None)
print(output.shape)  # torch.Size([32, 10, 256])

Sinusoidal Positional Encoding

One popular method for positional encoding is the sinusoidal approach. It uses sine and cosine functions of different frequencies to create unique encodings for each position. This method has several advantages, including the ability to extrapolate to sequence lengths longer than those seen during training.

import numpy as np
import matplotlib.pyplot as plt

def get_positional_encoding(seq_len, d_model):
    position = np.arange(seq_len)[:, np.newaxis]
    div_term = np.exp(np.arange(0, d_model, 2) * -(np.log(10000.0) / d_model))
    pos_encoding = np.zeros((seq_len, d_model))
    pos_encoding[:, 0::2] = np.sin(position * div_term)
    pos_encoding[:, 1::2] = np.cos(position * div_term)
    return pos_encoding

# Generate and visualize positional encoding
seq_len, d_model = 100, 64
pos_encoding = get_positional_encoding(seq_len, d_model)

plt.figure(figsize=(10, 6))
plt.imshow(pos_encoding, cmap='viridis', aspect='auto')
plt.colorbar()
plt.title('Sinusoidal Positional Encoding')
plt.xlabel('Dimension')
plt.ylabel('Position')
plt.show()

# Plot specific dimensions
plt.figure(figsize=(10, 6))
for i in range(0, d_model, d_model//4):
    plt.plot(pos_encoding[:, i], label=f'Dim {i}')
plt.legend()
plt.title('Positional Encoding Values for Specific Dimensions')
plt.xlabel('Position')
plt.ylabel('Value')
plt.show()

Learned Positional Encoding

An alternative to fixed sinusoidal encoding is learned positional encoding. In this approach, the model learns the optimal positional representations during training. This can potentially adapt better to specific tasks but may struggle with extrapolation to unseen sequence lengths.

import torch
import torch.nn as nn

class LearnedPositionalEncoding(nn.Module):
    def __init__(self, max_seq_len, d_model):
        super().__init__()
        self.embedding = nn.Embedding(max_seq_len, d_model)
    
    def forward(self, x):
        seq_len = x.size(1)
        positions = torch.arange(seq_len, device=x.device)
        return self.embedding(positions).unsqueeze(0)

# Example usage
max_seq_len, d_model = 1000, 512
learned_pe = LearnedPositionalEncoding(max_seq_len, d_model)

# Simulate input tensor
x = torch.randn(1, 100, d_model)  # Batch size 1, sequence length 100

# Get learned positional encoding
pos_encoding = learned_pe(x)

print("Input shape:", x.shape)
print("Positional encoding shape:", pos_encoding.shape)

# Visualize learned positional encoding
plt.figure(figsize=(10, 6))
plt.imshow(pos_encoding.squeeze().detach().numpy(), cmap='viridis', aspect='auto')
plt.colorbar()
plt.title('Learned Positional Encoding')
plt.xlabel('Dimension')
plt.ylabel('Position')
plt.show()

Combining Positional Encoding with Input Embeddings

In Transformer models, positional encodings are typically added to the input embeddings. This allows the model to retain both the semantic information from the embeddings and the positional information. The combined representation is then used as input to the self-attention layers.

import torch
import torch.nn as nn

class PositionalEmbedding(nn.Module):
    def __init__(self, d_model, max_seq_len):
        super().__init__()
        self.embedding = nn.Embedding(max_seq_len, d_model)
        self.d_model = d_model

    def forward(self, x):
        seq_len = x.size(1)
        positions = torch.arange(seq_len, device=x.device)
        pos_embedding = self.embedding(positions)
        return pos_embedding * (self.d_model ** 0.5)

class TransformerEmbedding(nn.Module):
    def __init__(self, vocab_size, d_model, max_seq_len):
        super().__init__()
        self.token_embedding = nn.Embedding(vocab_size, d_model)
        self.positional_embedding = PositionalEmbedding(d_model, max_seq_len)

    def forward(self, x):
        token_embedding = self.token_embedding(x)
        positional_embedding = self.positional_embedding(x)
        return token_embedding + positional_embedding

# Example usage
vocab_size, d_model, max_seq_len = 10000, 512, 1000
transformer_embedding = TransformerEmbedding(vocab_size, d_model, max_seq_len)

# Simulate input tensor (batch_size=2, seq_len=10)
x = torch.randint(0, vocab_size, (2, 10))

# Get combined embedding
combined_embedding = transformer_embedding(x)

print("Input shape:", x.shape)
print("Combined embedding shape:", combined_embedding.shape)

# Visualize combined embedding for the first sequence in the batch
plt.figure(figsize=(10, 6))
plt.imshow(combined_embedding[0].detach().numpy(), cmap='viridis', aspect='auto')
plt.colorbar()
plt.title('Combined Token and Positional Embedding')
plt.xlabel('Dimension')
plt.ylabel('Position')
plt.show()

Relative Positional Encoding

Relative positional encoding is an alternative approach that encodes the relative distances between tokens rather than absolute positions. This can be particularly useful for tasks where the relative order of elements is more important than their absolute positions.

import torch
import torch.nn as nn
import torch.nn.functional as F

class RelativePositionalEncoding(nn.Module):
    def __init__(self, d_model, max_len):
        super().__init__()
        self.d_model = d_model
        self.max_len = max_len
        self.embeddings = nn.Parameter(torch.randn(max_len * 2 - 1, d_model))

    def forward(self, length):
        positions = torch.arange(length, dtype=torch.long).to(self.embeddings.device)
        positions = positions[None, :] - positions[:, None]
        positions += self.max_len - 1  # shift to positive indices
        return self.embeddings[positions]

class RelativeAttention(nn.Module):
    def __init__(self, d_model, num_heads, max_len):
        super().__init__()
        self.d_model = d_model
        self.num_heads = num_heads
        self.max_len = max_len

        self.q_linear = nn.Linear(d_model, d_model)
        self.k_linear = nn.Linear(d_model, d_model)
        self.v_linear = nn.Linear(d_model, d_model)
        self.pos_encoding = RelativePositionalEncoding(d_model // num_heads, max_len)

    def forward(self, x):
        B, L, _ = x.shape
        H = self.num_heads

        q = self.q_linear(x).view(B, L, H, -1).transpose(1, 2)
        k = self.k_linear(x).view(B, L, H, -1).transpose(1, 2)
        v = self.v_linear(x).view(B, L, H, -1).transpose(1, 2)

        pos_emb = self.pos_encoding(L)
        pos_emb = pos_emb.view(L, L, H, -1).permute(2, 0, 1, 3)

        content_attention = torch.matmul(q, k.transpose(-1, -2))
        pos_attention = torch.matmul(q, pos_emb.transpose(-1, -2))
        attention = content_attention + pos_attention

        attention = F.softmax(attention / (self.d_model // H) ** 0.5, dim=-1)
        out = torch.matmul(attention, v)
        out = out.transpose(1, 2).contiguous().view(B, L, -1)

        return out

# Example usage
d_model, num_heads, max_len = 512, 8, 100
relative_attention = RelativeAttention(d_model, num_heads, max_len)

# Simulate input tensor
x = torch.randn(2, 50, d_model)  # Batch size 2, sequence length 50

# Apply relative attention
output = relative_attention(x)

print("Input shape:", x.shape)
print("Output shape:", output.shape)

Positional Encoding in Vision Transformers

Vision Transformers (ViT) adapt the Transformer architecture for image processing tasks. They use a special form of positional encoding to represent the 2D structure of images. This encoding helps the model understand spatial relationships between image patches.

import torch
import torch.nn as nn
import matplotlib.pyplot as plt

class VisionTransformerPositionalEncoding(nn.Module):
    def __init__(self, d_model, max_h=64, max_w=64):
        super().__init__()
        self.d_model = d_model
        self.max_h = max_h
        self.max_w = max_w
        
        pe = torch.zeros(max_h, max_w, d_model)
        y_pos = torch.arange(0, max_h).unsqueeze(1).expand(-1, max_w).unsqueeze(2)
        x_pos = torch.arange(0, max_w).unsqueeze(0).expand(max_h, -1).unsqueeze(2)
        
        div_term = torch.exp(torch.arange(0, d_model, 2) * -(math.log(10000.0) / d_model))
        pe[:, :, 0::2] = torch.sin(torch.cat([y_pos, x_pos], dim=2) * div_term)
        pe[:, :, 1::2] = torch.cos(torch.cat([y_pos, x_pos], dim=2) * div_term)
        
        self.register_buffer('pe', pe)

    def forward(self, x):
        """
        x: Tensor, shape [batch_size, height, width, channels]
        """
        h, w = x.size(1), x.size(2)
        return self.pe[:h, :w, :].unsqueeze(0)

# Example usage
d_model, max_h, max_w = 256, 32, 32
vit_pe = VisionTransformerPositionalEncoding(d_model, max_h, max_w)

# Simulate input tensor (batch_size=1, height=24, width=24, channels=256)
x = torch.randn(1, 24, 24, d_model)

# Get positional encoding
pos_encoding = vit_pe(x)

print("Input shape:", x.shape)
print("Positional encoding shape:", pos_encoding.shape)

# Visualize positional encoding
plt.figure(figsize=(12, 4))
for i in range(4):  # Visualize first 4 dimensions
    plt.subplot(1, 4, i+1)
    plt.imshow(pos_encoding[0, :, :, i].detach().numpy(), cmap='viridis')
    plt.title(f'Dimension {i}')
    plt.axis('off')
plt.tight_layout()
plt.show()

Impact of Positional Encoding on Model Performance

Positional encoding significantly impacts the performance of Transformer models. Without it, these models struggle to capture sequential information, leading to poor performance on tasks that require understanding of order. Let's demonstrate this with a simple experiment comparing a Transformer model with and without positional encoding.

import torch
import torch.nn as nn
import matplotlib.pyplot as plt

class SimpleTransformer(nn.Module):
    def __init__(self, d_model, nhead, num_layers, use_pos_encoding=True):
        super().__init__()
        self.embedding = nn.Embedding(10, d_model)
        self.pos_encoder = nn.Embedding(100, d_model) if use_pos_encoding else None
        self.transformer = nn.TransformerEncoder(
            nn.TransformerEncoderLayer(d_model, nhead), num_layers)
        self.fc = nn.Linear(d_model, 10)

    def forward(self, x):
        x = self.embedding(x)
        if self.pos_encoder:
            positions = torch.arange(x.size(1), device=x.device).unsqueeze(0)
            x = x + self.pos_encoder(positions)
        x = self.transformer(x)
        return self.fc(x.mean(dim=1))

# Training function (pseudocode)
def train_and_evaluate(model, train_data, test_data):
    # Train the model
    # Evaluate on test data
    return test_accuracy

# Example usage
d_model, nhead, num_layers = 64, 4, 2
model_with_pe = SimpleTransformer(d_model, nhead, num_layers, use_pos_encoding=True)
model_without_pe = SimpleTransformer(d_model, nhead, num_layers, use_pos_encoding=False)

# Simulate training and evaluation
accuracy_with_pe = train_and_evaluate(model_with_pe, train_data, test_data)
accuracy_without_pe = train_and_evaluate(model_without_pe, train_data, test_data)

# Plot results
plt.bar(['With PE', 'Without PE'], [accuracy_with_pe, accuracy_without_pe])
plt.title('Impact of Positional Encoding on Model Accuracy')
plt.ylabel('Test Accuracy')
plt.show()

Positional Encoding in Different Languages

Positional encoding is language-agnostic, making it versatile for various natural language processing tasks across different languages. However, the way it interacts with different linguistic structures can vary. Let's explore how positional encoding works with languages that have different word orders.

import torch
import torch.nn as nn
import matplotlib.pyplot as plt

class MultilingualTransformer(nn.Module):
    def __init__(self, vocab_size, d_model, nhead, num_layers):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, d_model)
        self.pos_encoder = nn.Embedding(100, d_model)
        self.transformer = nn.TransformerEncoder(
            nn.TransformerEncoderLayer(d_model, nhead), num_layers)
        self.fc = nn.Linear(d_model, vocab_size)

    def forward(self, x):
        x = self.embedding(x)
        positions = torch.arange(x.size(1), device=x.device).unsqueeze(0)
        x = x + self.pos_encoder(positions)
        x = self.transformer(x)
        return self.fc(x)

# Example sentences in different languages
sentences = {
    'English': "The cat sits on the mat",
    'Japanese': "猫が マットの上に 座っています",
    'German': "Die Katze sitzt auf der Matte"
}

# Tokenize sentences (simplified)
vocab = set(''.join(sentences.values()))
char_to_idx = {char: idx for idx, char in enumerate(vocab)}
tokenized = {lang: torch.tensor([char_to_idx[c] for c in sent]) for lang, sent in sentences.items()}

# Create and apply model
model = MultilingualTransformer(len(vocab), d_model=64, nhead=4, num_layers=2)
outputs = {lang: model(sent.unsqueeze(0)) for lang, sent in tokenized.items()}

# Visualize attention patterns (pseudocode)
def plot_attention(outputs):
    # Extract attention weights
    # Plot heatmaps for each language
    pass

plot_attention(outputs)

Positional Encoding in Long Sequences

Handling long sequences is a challenge for Transformer models, partly due to the limitations of positional encoding. As sequence length increases, the effectiveness of standard positional encoding may decrease. Let's explore some techniques to address this issue.

import torch
import torch.nn as nn
import math

class SinusoidalPositionalEncoding(nn.Module):
    def __init__(self, d_model, max_len=5000):
        super().__init__()
        pe = torch.zeros(max_len, d_model)
        position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)
        div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))
        pe[:, 0::2] = torch.sin(position * div_term)
        pe[:, 1::2] = torch.cos(position * div_term)
        self.register_buffer('pe', pe)

    def forward(self, x):
        return self.pe[:x.size(1), :]

class CompressedPositionalEncoding(nn.Module):
    def __init__(self, d_model, max_len=5000, compress_factor=10):
        super().__init__()
        self.base_pe = SinusoidalPositionalEncoding(d_model, max_len // compress_factor)
        self.compress_factor = compress_factor

    def forward(self, x):
        positions = torch.arange(0, x.size(1), device=x.device) // self.compress_factor
        return self.base_pe.pe[positions, :]

# Example usage
seq_len, d_model = 10000, 512
x = torch.randn(1, seq_len, d_model)

standard_pe = SinusoidalPositionalEncoding(d_model)
compressed_pe = CompressedPositionalEncoding(d_model)

standard_encoding = standard_pe(x)
compressed_encoding = compressed_pe(x)

print("Standard PE shape:", standard_encoding.shape)
print("Compressed PE shape:", compressed_encoding.shape)

# Visualize encodings (pseudocode)
def plot_encodings(standard, compressed):
    # Plot standard and compressed encodings
    pass

plot_encodings(standard_encoding, compressed_encoding)

Adaptive Positional Encoding

Adaptive positional encoding techniques aim to dynamically adjust the encoding based on the input sequence. This approach can be particularly useful for tasks with varying sequence lengths or when dealing with very long sequences.

import torch
import torch.nn as nn

class AdaptivePositionalEncoding(nn.Module):
    def __init__(self, d_model, max_len=5000):
        super().__init__()
        self.d_model = d_model
        self.max_len = max_len
        self.adaptor = nn.Linear(2, d_model)

    def forward(self, x):
        seq_len = x.size(1)
        positions = torch.arange(seq_len, dtype=torch.float, device=x.device).unsqueeze(-1)
        scaled_positions = positions / self.max_len
        relative_positions = positions / seq_len
        pos_info = torch.cat([scaled_positions, relative_positions], dim=-1)
        return self.adaptor(pos_info)

# Example usage
d_model, max_len = 512, 5000
adaptive_pe = AdaptivePositionalEncoding(d_model, max_len)

# Test with different sequence lengths
seq_lengths = [100, 1000, 5000]
for seq_len in seq_lengths:
    x = torch.randn(1, seq_len, d_model)
    pe = adaptive_pe(x)
    print(f"Sequence length: {seq_len}, PE shape: {pe.shape}")

# Visualize adaptive encoding (pseudocode)
def plot_adaptive_encoding(adaptive_pe, seq_lengths):
    # Generate and plot adaptive encodings for different sequence lengths
    pass

plot_adaptive_encoding(adaptive_pe, seq_lengths)

Positional Encoding in Multimodal Transformers

Multimodal Transformers process different types of data simultaneously, such as text and images. Positional encoding in these models must account for the different structures of each modality. Let's explore a simple example of positional encoding in a text-image Transformer.

import torch
import torch.nn as nn

class MultimodalPositionalEncoding(nn.Module):
    def __init__(self, d_model, max_text_len=1000, max_img_size=(224, 224)):
        super().__init__()
        self.text_pe = nn.Embedding(max_text_len, d_model)
        self.img_pe_h = nn.Embedding(max_img_size[0], d_model // 2)
        self.img_pe_w = nn.Embedding(max_img_size[1], d_model // 2)

    def forward(self, text_x, img_x):
        # Text positional encoding
        text_positions = torch.arange(text_x.size(1), device=text_x.device)
        text_pe = self.text_pe(text_positions)

        # Image positional encoding
        h, w = img_x.shape[-2:]
        img_pos_h = torch.arange(h, device=img_x.device).unsqueeze(1).expand(-1, w)
        img_pos_w = torch.arange(w, device=img_x.device).unsqueeze(0).expand(h, -1)
        img_pe = torch.cat([self.img_pe_h(img_pos_h), self.img_pe_w(img_pos_w)], dim=-1)

        return text_pe, img_pe

# Example usage
d_model, max_text_len, max_img_size = 512, 1000, (224, 224)
multimodal_pe = MultimodalPositionalEncoding(d_model, max_text_len, max_img_size)

# Simulate inputs
text_input = torch.randn(1, 50, d_model)  # Batch size 1, sequence length 50
img_input = torch.randn(1, 3, 224, 224)  # Batch size 1, 3 channels, 224x224 image

text_pe, img_pe = multimodal_pe(text_input, img_input)

print("Text PE shape:", text_pe.shape)
print("Image PE shape:", img_pe.shape)

# Visualize multimodal positional encoding (pseudocode)
def plot_multimodal_pe(text_pe, img_pe):
    # Plot text and image positional encodings
    pass

plot_multimodal_pe(text_pe, img_pe)

Real-life Examples of Positional Encoding Applications

Positional encoding plays a crucial role in various real-world applications of Transformer models. Let's explore two common use cases: machine translation and image captioning.

Machine Translation: In machine translation, positional encoding helps the model understand the structure of sentences in different languages, preserving word order information.

import torch
import torch.nn as nn

class SimpleTranslationModel(nn.Module):
    def __init__(self, src_vocab_size, tgt_vocab_size, d_model, nhead, num_layers):
        super().__init__()
        self.src_embedding = nn.Embedding(src_vocab_size, d_model)
        self.tgt_embedding = nn.Embedding(tgt_vocab_size, d_model)
        self.positional_encoding = nn.Embedding(1000, d_model)
        self.transformer = nn.Transformer(d_model, nhead, num_layers)
        self.fc = nn.Linear(d_model, tgt_vocab_size)

    def forward(self, src, tgt):
        src_pos = torch.arange(src.size(1), device=src.device).unsqueeze(0)
        tgt_pos = torch.arange(tgt.size(1), device=tgt.device).unsqueeze(0)

        src_emb = self.src_embedding(src) + self.positional_encoding(src_pos)
        tgt_emb = self.tgt_embedding(tgt) + self.positional_encoding(tgt_pos)

        output = self.transformer(src_emb, tgt_emb)
        return self.fc(output)

# Example usage (pseudocode)
src_sentence = "Hello, how are you?"
tgt_sentence = "Bonjour, comment allez-vous?"

# Tokenize sentences
# Create and train model
# Translate new sentences

Additional Resources

Image Captioning: In image captioning, positional encoding helps the model generate coherent captions by maintaining the order of words while considering image features.

import torch
import torch.nn as nn
import torchvision.models as models

class ImageCaptioningModel(nn.Module):
    def __init__(self, vocab_size, d_model, nhead, num_layers):
        super().__init__()
        self.cnn = models.resnet50(pretrained=True)
        self.cnn.fc = nn.Linear(self.cnn.fc.in_features, d_model)
        
        self.word_embedding = nn.Embedding(vocab_size, d_model)
        self.positional_encoding = nn.Embedding(1000, d_model)
        
        self.transformer_decoder = nn.TransformerDecoder(
            nn.TransformerDecoderLayer(d_model, nhead), num_layers)
        
        self.fc = nn.Linear(d_model, vocab_size)

    def forward(self, image, caption):
        img_features = self.cnn(image).unsqueeze(1)
        
        word_embeddings = self.word_embedding(caption)
        positions = torch.arange(caption.size(1), device=caption.device).unsqueeze(0)
        pos_embeddings = self.positional_encoding(positions)
        
        caption_embeddings = word_embeddings + pos_embeddings
        
        output = self.transformer_decoder(caption_embeddings, img_features)
        return self.fc(output)

# Example usage (pseudocode)
image = load_image("cat.jpg")
caption = "A cat sitting on a mat"

# Preprocess image and caption
# Create and train model
# Generate captions for new images

Additional Resources

For those interested in diving deeper into positional encoding and Transformer models, here are some valuable resources:

"Attention Is All You Need" - The original Transformer paper arXiv:1706.03762 [cs.CL] https://arxiv.org/abs/1706.03762
"On the Positional Encoding in Transformer Models: An Overview" arXiv:2102.01926 [cs.CL] https://arxiv.org/abs/2102.01926
"Rethinking Positional Encoding in Language Pre-training" arXiv:2006.15595 [cs.CL] https://arxiv.org/abs/2006.15595
"Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context" arXiv:1901.02860 [cs.LG] https://arxiv.org/abs/1901.02860

These papers provide in-depth discussions on various aspects of positional encoding and its role in Transformer models. They cover both theoretical foundations and practical implementations, offering insights into the latest advancements in the field.