xbe.at

NLP and transformers / BERT and language models

Understanding BERT A Cornerstone of NLP with Python

12 min read, 15 sections

Contents
  1. Introduction to BERT
  2. BERT's Architecture
  3. Tokenization in BERT
  4. Pre-training BERT
  5. Fine-tuning BERT
  6. Attention Mechanism in BERT
  7. BERT for Text Classification
  8. BERT for Named Entity Recognition (NER)
  9. BERT for Question Answering
  10. BERT for Text Summarization
  11. BERT for Sentence Similarity
  12. BERT for Text Generation
  13. Real-Life Example: Sentiment Analysis for Product Reviews
  14. Real-Life Example: Chatbot with BERT
  15. Additional Resources

Introduction to BERT

BERT (Bidirectional Encoder Representations from Transformers) is a revolutionary natural language processing model that has significantly improved the way machines understand and process human language. Developed by Google in 2018, BERT has become a cornerstone in various NLP tasks, including question answering, sentiment analysis, and text classification. In this presentation, we'll explore BERT's architecture, its key features, and how to implement it using Python.

import torch
from transformers import BertTokenizer, BertModel

# Load pre-trained BERT model and tokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained('bert-base-uncased')

# Example text
text = "BERT is a powerful NLP model."

# Tokenize and encode the text
inputs = tokenizer(text, return_tensors="pt")

# Get BERT embeddings
with torch.no_grad():
    outputs = model(**inputs)

# Access the last hidden state
last_hidden_state = outputs.last_hidden_state
print(f"Shape of last hidden state: {last_hidden_state.shape}")

BERT's Architecture

BERT's architecture is based on the Transformer model, which utilizes self-attention mechanisms to process input sequences. It consists of multiple layers of bidirectional self-attention and feed-forward neural networks. This bidirectional nature allows BERT to consider the entire context of a word by looking at the words that come both before and after it, leading to a more comprehensive understanding of language.

import torch
from transformers import BertConfig, BertModel

# Create a BERT configuration
config = BertConfig(
    vocab_size=30522,  # Size of the vocabulary
    hidden_size=768,   # Dimension of the hidden layers
    num_hidden_layers=12,  # Number of hidden layers
    num_attention_heads=12,  # Number of attention heads
    intermediate_size=3072  # Dimension of the intermediate (feed-forward) layer
)

# Create a BERT model with the specified configuration
model = BertModel(config)

# Print the model architecture
print(model)

Tokenization in BERT

Tokenization is a crucial step in processing text for BERT. It involves breaking down text into smaller units called tokens. BERT uses WordPiece tokenization, which can handle out-of-vocabulary words by breaking them into subwords. This approach allows BERT to understand and process a wide range of words, including rare or complex ones.

from transformers import BertTokenizer

# Load the BERT tokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

# Example text
text = "BERT uses WordPiece tokenization for handling complex words."

# Tokenize the text
tokens = tokenizer.tokenize(text)
print(f"Tokens: {tokens}")

# Convert tokens to input IDs
input_ids = tokenizer.convert_tokens_to_ids(tokens)
print(f"Input IDs: {input_ids}")

# Decode the input IDs back to text
decoded_text = tokenizer.decode(input_ids)
print(f"Decoded text: {decoded_text}")

Pre-training BERT

BERT is pre-trained on a large corpus of unlabeled text using two unsupervised tasks: Masked Language Modeling (MLM) and Next Sentence Prediction (NSP). These tasks help BERT learn contextual representations of words and understand the relationships between sentences. Pre-training allows BERT to capture general language understanding, which can be fine-tuned for specific downstream tasks.

import torch
from transformers import BertForMaskedLM, BertTokenizer

# Load pre-trained BERT model and tokenizer
model = BertForMaskedLM.from_pretrained('bert-base-uncased')
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

# Example sentence with a masked word
text = "The [MASK] is shining brightly in the sky."

# Tokenize and encode the text
inputs = tokenizer(text, return_tensors="pt")

# Get model predictions
with torch.no_grad():
    outputs = model(**inputs)

# Get the predicted token for the masked word
masked_index = torch.where(inputs["input_ids"][0] == tokenizer.mask_token_id)[0]
predicted_token_id = outputs.logits[0, masked_index].argmax(axis=-1)
predicted_token = tokenizer.decode(predicted_token_id)

print(f"Predicted word: {predicted_token}")

Fine-tuning BERT

After pre-training, BERT can be fine-tuned on specific tasks with relatively small amounts of labeled data. This process adapts BERT's general language understanding to perform well on particular NLP tasks such as text classification, named entity recognition, or question answering. Fine-tuning typically involves adding a task-specific layer on top of the pre-trained BERT model and training the entire model on the task-specific dataset.

import torch
from transformers import BertForSequenceClassification, BertTokenizer
from torch.utils.data import DataLoader, TensorDataset
from sklearn.model_selection import train_test_split

# Load pre-trained BERT model and tokenizer
model = BertForSequenceClassification.from_pretrained('bert-base-uncased', num_labels=2)
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

# Example dataset (replace with your own data)
texts = ["I love this movie!", "This book is terrible.", "The weather is nice today."]
labels = [1, 0, 1]  # 1 for positive, 0 for negative

# Tokenize and encode the texts
encodings = tokenizer(texts, truncation=True, padding=True, return_tensors="pt")

# Create dataset and dataloader
dataset = TensorDataset(encodings['input_ids'], encodings['attention_mask'], torch.tensor(labels))
train_dataset, val_dataset = train_test_split(dataset, test_size=0.2)
train_loader = DataLoader(train_dataset, batch_size=2, shuffle=True)

# Fine-tuning loop (simplified)
optimizer = torch.optim.AdamW(model.parameters(), lr=5e-5)

for epoch in range(3):
    for batch in train_loader:
        input_ids, attention_mask, labels = batch
        outputs = model(input_ids, attention_mask=attention_mask, labels=labels)
        loss = outputs.loss
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()

print("Fine-tuning completed")

Attention Mechanism in BERT

The attention mechanism is a key component of BERT's architecture. It allows the model to focus on different parts of the input sequence when processing each word, enabling it to capture complex relationships and dependencies within the text. BERT uses multi-head attention, which applies multiple attention operations in parallel, allowing the model to capture different types of relationships simultaneously.

import torch
import torch.nn as nn

class SelfAttention(nn.Module):
    def __init__(self, hidden_size):
        super(SelfAttention, self).__init__()
        self.hidden_size = hidden_size
        self.query = nn.Linear(hidden_size, hidden_size)
        self.key = nn.Linear(hidden_size, hidden_size)
        self.value = nn.Linear(hidden_size, hidden_size)
        
    def forward(self, x):
        q = self.query(x)
        k = self.key(x)
        v = self.value(x)
        
        scores = torch.matmul(q, k.transpose(-2, -1)) / (self.hidden_size ** 0.5)
        attention_weights = torch.softmax(scores, dim=-1)
        output = torch.matmul(attention_weights, v)
        
        return output

# Example usage
hidden_size = 768
seq_length = 10
batch_size = 2

attention = SelfAttention(hidden_size)
x = torch.randn(batch_size, seq_length, hidden_size)
output = attention(x)

print(f"Input shape: {x.shape}")
print(f"Output shape: {output.shape}")

BERT for Text Classification

One of the most common applications of BERT is text classification. By fine-tuning BERT on a labeled dataset, we can create powerful classifiers for tasks such as sentiment analysis, topic classification, or spam detection. BERT's contextual understanding of language often leads to superior performance compared to traditional methods.

import torch
from transformers import BertForSequenceClassification, BertTokenizer

# Load fine-tuned BERT model and tokenizer
model = BertForSequenceClassification.from_pretrained('bert-base-uncased', num_labels=2)
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

# Example text for classification
text = "This restaurant's food is absolutely delicious!"

# Tokenize and encode the text
inputs = tokenizer(text, return_tensors="pt", truncation=True, padding=True)

# Get model predictions
with torch.no_grad():
    outputs = model(**inputs)

# Get the predicted class (0 for negative, 1 for positive)
predicted_class = torch.argmax(outputs.logits, dim=1).item()
sentiment = "positive" if predicted_class == 1 else "negative"

print(f"Sentiment: {sentiment}")

BERT for Named Entity Recognition (NER)

Named Entity Recognition is another task where BERT excels. By fine-tuning BERT on NER datasets, we can create models that accurately identify and classify named entities such as person names, organizations, and locations in text. BERT's contextual understanding helps it distinguish between different uses of the same word based on its context.

from transformers import BertForTokenClassification, BertTokenizer
import torch

# Load fine-tuned BERT model and tokenizer for NER
model = BertForTokenClassification.from_pretrained('dbmdz/bert-large-cased-finetuned-conll03-english')
tokenizer = BertTokenizer.from_pretrained('bert-base-cased')

# Example text for NER
text = "Apple is looking at buying U.K. startup for $1 billion"

# Tokenize and encode the text
inputs = tokenizer(text, return_tensors="pt", truncation=True, padding=True)

# Get model predictions
with torch.no_grad():
    outputs = model(**inputs)

# Get the predicted labels
predictions = torch.argmax(outputs.logits, dim=2)

# Decode the predictions
tokens = tokenizer.convert_ids_to_tokens(inputs["input_ids"][0])
labels = [model.config.id2label[t.item()] for t in predictions[0]]

# Print the tokens and their corresponding labels
for token, label in zip(tokens, labels):
    if label != "O":  # "O" stands for "Outside" (not a named entity)
        print(f"{token}: {label}")

BERT for Question Answering

BERT has shown remarkable performance in question answering tasks. By fine-tuning BERT on question-answer pairs, we can create models that can extract answers from given contexts. This capability has applications in chatbots, search engines, and information retrieval systems.

from transformers import BertForQuestionAnswering, BertTokenizer
import torch

# Load fine-tuned BERT model and tokenizer for question answering
model = BertForQuestionAnswering.from_pretrained('bert-large-uncased-whole-word-masking-finetuned-squad')
tokenizer = BertTokenizer.from_pretrained('bert-large-uncased-whole-word-masking-finetuned-squad')

# Example context and question
context = "The Eiffel Tower is a wrought-iron lattice tower on the Champ de Mars in Paris, France. It is named after the engineer Gustave Eiffel, whose company designed and built the tower."
question = "Who is the Eiffel Tower named after?"

# Tokenize and encode the context and question
inputs = tokenizer(question, context, return_tensors="pt")

# Get model predictions
with torch.no_grad():
    outputs = model(**inputs)

# Get the most likely beginning and end of the answer
answer_start = torch.argmax(outputs.start_logits)
answer_end = torch.argmax(outputs.end_logits) + 1

# Extract the answer
answer = tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(inputs["input_ids"][0][answer_start:answer_end]))

print(f"Question: {question}")
print(f"Answer: {answer}")

BERT for Text Summarization

While BERT itself is not designed for text summarization, it can be used as a foundation for building effective summarization models. By combining BERT's language understanding capabilities with additional layers or techniques, we can create models that generate concise and meaningful summaries of longer texts. One approach is to use BERT to encode sentences and then apply a classification layer to determine which sentences are most important for the summary.

from transformers import BertTokenizer, BertModel
import torch
import torch.nn as nn

class BertSummarizer(nn.Module):
    def __init__(self, bert_model, hidden_size, num_labels):
        super(BertSummarizer, self).__init__()
        self.bert = bert_model
        self.classifier = nn.Linear(hidden_size, num_labels)
        
    def forward(self, input_ids, attention_mask):
        outputs = self.bert(input_ids=input_ids, attention_mask=attention_mask)
        pooled_output = outputs.pooler_output
        logits = self.classifier(pooled_output)
        return logits

# Load pre-trained BERT model and tokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
bert_model = BertModel.from_pretrained('bert-base-uncased')

# Create summarizer model
summarizer = BertSummarizer(bert_model, hidden_size=768, num_labels=2)

# Example text
text = "BERT has revolutionized NLP. It uses bidirectional training to understand context. This allows it to excel in various tasks."

# Tokenize and encode the text
inputs = tokenizer(text, return_tensors="pt", padding=True, truncation=True, max_length=512)

# Get model predictions
with torch.no_grad():
    logits = summarizer(**inputs)

# In practice, you would use these logits to decide which sentences to include in the summary
print("Logits shape:", logits.shape)

BERT for Sentence Similarity

BERT's ability to generate contextual embeddings makes it excellent for measuring sentence similarity. By encoding sentences using BERT and comparing their embeddings, we can determine how semantically similar two sentences are. This has applications in information retrieval, duplicate detection, and recommendation systems.

from transformers import BertTokenizer, BertModel
import torch
from scipy.spatial.distance import cosine

# Load pre-trained BERT model and tokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained('bert-base-uncased')

def get_bert_embedding(text):
    inputs = tokenizer(text, return_tensors="pt", truncation=True, padding=True)
    with torch.no_grad():
        outputs = model(**inputs)
    return outputs.last_hidden_state.mean(dim=1).squeeze().numpy()

# Example sentences
sentence1 = "The cat sat on the mat."
sentence2 = "A feline rested on the rug."

# Get BERT embeddings for both sentences
embedding1 = get_bert_embedding(sentence1)
embedding2 = get_bert_embedding(sentence2)

# Calculate cosine similarity
similarity = 1 - cosine(embedding1, embedding2)

print(f"Sentence 1: {sentence1}")
print(f"Sentence 2: {sentence2}")
print(f"Similarity: {similarity:.4f}")

BERT for Text Generation

While BERT is primarily designed for understanding and encoding text, it can be adapted for text generation tasks. One approach is to use BERT as an encoder in a sequence-to-sequence model, where a separate decoder is trained to generate text based on BERT's encoded representations. This technique can be applied to various generation tasks such as text completion, dialogue systems, or even simple language translation.

from transformers import BertTokenizer, BertModel, BertLMHeadModel
import torch

# Load pre-trained BERT model and tokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
bert_encoder = BertModel.from_pretrained('bert-base-uncased')
bert_decoder = BertLMHeadModel.from_pretrained('bert-base-uncased')

# Example input text
input_text = "The quick brown"

# Encode input text
input_ids = tokenizer.encode(input_text, return_tensors='pt')

# Get BERT encoder output
with torch.no_grad():
    encoder_output = bert_encoder(input_ids)[0]

# Generate text using BERT decoder
max_length = 20
generated = bert_decoder.generate(
    encoder_output,
    max_length=max_length,
    num_return_sequences=1,
    no_repeat_ngram_size=2
)

# Decode and print the generated text
generated_text = tokenizer.decode(generated[0], skip_special_tokens=True)
print(f"Input: {input_text}")
print(f"Generated: {generated_text}")

Real-Life Example: Sentiment Analysis for Product Reviews

BERT's ability to understand context makes it particularly useful for sentiment analysis in e-commerce platforms. By fine-tuning BERT on a dataset of product reviews, we can create a model that accurately classifies the sentiment of new reviews, helping businesses gain insights into customer satisfaction.

from transformers import BertForSequenceClassification, BertTokenizer
import torch

# Load fine-tuned BERT model and tokenizer
model = BertForSequenceClassification.from_pretrained('bert-base-uncased', num_labels=3)  # 3 labels: negative, neutral, positive
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

# Example product review
review = "This smartphone has an excellent camera and long battery life, but the user interface is a bit confusing."

# Tokenize and encode the review
inputs = tokenizer(review, return_tensors="pt", truncation=True, padding=True)

# Get model predictions
with torch.no_grad():
    outputs = model(**inputs)

# Get the predicted sentiment
predicted_class = torch.argmax(outputs.logits, dim=1).item()
sentiment = ['Negative', 'Neutral', 'Positive'][predicted_class]

print(f"Review: {review}")
print(f"Sentiment: {sentiment}")

Real-Life Example: Chatbot with BERT

BERT can be used to enhance the natural language understanding capabilities of chatbots. By fine-tuning BERT on a dataset of conversations, we can create a chatbot that better understands user queries and provides more contextually appropriate responses.

from transformers import BertForQuestionAnswering, BertTokenizer
import torch

# Load fine-tuned BERT model and tokenizer
model = BertForQuestionAnswering.from_pretrained('bert-large-uncased-whole-word-masking-finetuned-squad')
tokenizer = BertTokenizer.from_pretrained('bert-large-uncased-whole-word-masking-finetuned-squad')

# Chatbot context
context = """
Our company offers cloud storage solutions. We have three main plans:
Basic: 100GB for $5/month
Pro: 1TB for $10/month
Enterprise: 10TB for $50/month
All plans come with 24/7 customer support and end-to-end encryption.
"""

# Function to get chatbot response
def get_response(question):
    inputs = tokenizer(question, context, return_tensors="pt")
    with torch.no_grad():
        outputs = model(**inputs)
    
    answer_start = torch.argmax(outputs.start_logits)
    answer_end = torch.argmax(outputs.end_logits) + 1
    answer = tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(inputs["input_ids"][0][answer_start:answer_end]))
    return answer

# Example conversation
print("Chatbot: Hello! How can I assist you with our cloud storage solutions?")
print("User: What's the price of the Pro plan?")
response = get_response("What's the price of the Pro plan?")
print(f"Chatbot: {response}")
print("User: Do you offer customer support?")
response = get_response("Do you offer customer support?")
print(f"Chatbot: {response}")

Additional Resources

For those interested in diving deeper into BERT and its applications, here are some valuable resources:

  1. Original BERT paper: "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" by Devlin et al. (2018). Available on arXiv: https://arxiv.org/abs/1810.04805
  2. "Attention Is All You Need" by Vaswani et al. (2017), which introduces the Transformer architecture that BERT is based on. Available on arXiv: https://arxiv.org/abs/1706.03762
  3. HuggingFace Transformers library documentation, which provides a comprehensive guide to working with BERT and other transformer models: https://huggingface.co/transformers/
  4. "BERT Fine-Tuning Tutorial with PyTorch" by Chris McCormick and Nick Ryan, offering a practical guide to fine-tuning BERT: https://mccormickml.com/2019/07/22/BERT-fine-tuning/

These resources provide a mix of theoretical foundations and practical implementations to help you master BERT and its applications in natural language processing.

More in BERT and language models

  1. Understanding Perplexity in Language Models with Python9 min

    Perplexity is a crucial metric in evaluating language models, measuring how well a model predicts a sample of text. Lower perplexity indicates better...

  2. The Power of Word Embeddings in Language Models11 min

    Token embeddings form the foundation of modern language models by transforming discrete tokens into continuous vector representations. These dense vectors...

  3. Unified Multimodal Transformer Model in Python11 min

    The concept of a single transformer model capable of unifying multimodal understanding and generation has gained significant attention in recent years. This...

  4. Temperature's Role in Language Model Output8 min

    Temperature is a crucial hyperparameter in language models that controls the randomness of the model's output. It affects the probability distribution of the...

  5. Sequence-to-Sequence Models in Python8 min

    Sequence-to-Sequence (Seq2Seq) models are a type of neural network architecture designed to transform an input sequence into an output sequence. They are...

  6. Sequence-to-Sequence for Unordered Sets in Python7 min

    In traditional sequence-to-sequence models, the order of input elements is crucial. However, when dealing with sets, where order is irrelevant, we need a...