xbe.at

LLMs and generative AI / RAG and retrieval

5 Chunking Strategies for Retrieval-Augmented Generation

5 min read, 23 sections

Contents
  1. Introduction to Chunking Strategies for RAG
  2. Fixed-size Chunking
  3. Source Code for Fixed-size Chunking
  4. Results for: Source Code for Fixed-size Chunking
  5. Semantic Chunking
  6. Source Code for Semantic Chunking
  7. Results for: Source Code for Semantic Chunking
  8. Recursive Chunking
  9. Source Code for Recursive Chunking
  10. Results for: Source Code for Recursive Chunking
  11. Document Structure-based Chunking
  12. Source Code for Document Structure-based Chunking
  13. Section 1
  14. Section 2
  15. Results for: Source Code for Document Structure-based Chunking
  16. Section 1
  17. Section 2
  18. LLM-based Chunking
  19. Source Code for LLM-based Chunking
  20. Results for: Source Code for LLM-based Chunking
  21. Real-life Example: Text Summarization
  22. Real-life Example: Question Answering Systems
  23. Additional Resources

Introduction to Chunking Strategies for RAG

Chunking is a crucial step in Retrieval-Augmented Generation (RAG) systems, where large documents are divided into smaller, manageable pieces. This process ensures that text fits the input size of embedding models and enhances the efficiency and accuracy of retrieval. Let's explore five common chunking strategies and their implementations.

Fixed-size Chunking

Fixed-size chunking splits text into uniform segments of a specified length. While simple, this method may break sentences or ideas mid-stream, potentially distributing important information across multiple chunks.

Source Code for Fixed-size Chunking

def fixed_size_chunking(text, chunk_size):
    return [text[i:i+chunk_size] for i in range(0, len(text), chunk_size)]

text = "This is a sample text for demonstrating fixed-size chunking. It may break sentences."
chunks = fixed_size_chunking(text, 20)
print(chunks)

Results for: Source Code for Fixed-size Chunking

['This is a sample te', 'xt for demonstratin', 'g fixed-size chunki', 'ng. It may break se', 'ntences.']

Semantic Chunking

Semantic chunking segments documents based on meaningful units like sentences or paragraphs. It creates embeddings for each segment and combines them based on cosine similarity until a significant drop is detected, forming a new chunk.

Source Code for Semantic Chunking

import re
from collections import Counter
import math

def cosine_similarity(vec1, vec2):
    intersection = set(vec1.keys()) & set(vec2.keys())
    numerator = sum([vec1[x] * vec2[x] for x in intersection])
    
    sum1 = sum([vec1[x]**2 for x in vec1.keys()])
    sum2 = sum([vec2[x]**2 for x in vec2.keys()])
    denominator = math.sqrt(sum1) * math.sqrt(sum2)
    
    if not denominator:
        return 0.0
    return float(numerator) / denominator

def create_embedding(text):
    words = re.findall(r'\w+', text.lower())
    return Counter(words)

def semantic_chunking(text, similarity_threshold=0.5):
    sentences = re.split(r'(?<=[.!?])\s+', text)
    chunks = []
    current_chunk = sentences[0]
    current_embedding = create_embedding(current_chunk)
    
    for sentence in sentences[1:]:
        sentence_embedding = create_embedding(sentence)
        similarity = cosine_similarity(current_embedding, sentence_embedding)
        
        if similarity >= similarity_threshold:
            current_chunk += " " + sentence
            current_embedding = create_embedding(current_chunk)
        else:
            chunks.append(current_chunk)
            current_chunk = sentence
            current_embedding = sentence_embedding
    
    chunks.append(current_chunk)
    return chunks

text = "This is a sample text. It demonstrates semantic chunking. We split based on meaning. New topics start new chunks."
chunks = semantic_chunking(text)
print(chunks)

Results for: Source Code for Semantic Chunking

['This is a sample text. It demonstrates semantic chunking.', 'We split based on meaning.', 'New topics start new chunks.']

Recursive Chunking

Recursive chunking first divides text based on inherent separators like paragraphs or sections. If any resulting chunk exceeds a predefined size limit, it's further split into smaller chunks.

Source Code for Recursive Chunking

def recursive_chunking(text, max_chunk_size=100, separator='\n\n'):
    chunks = text.split(separator)
    result = []
    
    for chunk in chunks:
        if len(chunk) <= max_chunk_size:
            result.append(chunk)
        else:
            # Recursively split large chunks
            result.extend(recursive_chunking(chunk, max_chunk_size, '. '))
    
    return result

text = """Paragraph 1 is short.

Paragraph 2 is a bit longer and exceeds the maximum chunk size. It will be split into smaller parts based on sentences.

Paragraph 3 is also short."""

chunks = recursive_chunking(text)
for i, chunk in enumerate(chunks, 1):
    print(f"Chunk {i}: {chunk}")

Results for: Source Code for Recursive Chunking

Chunk 1: Paragraph 1 is short.
Chunk 2: Paragraph 2 is a bit longer and exceeds the maximum chunk size.
Chunk 3: It will be split into smaller parts based on sentences.
Chunk 4: Paragraph 3 is also short.

Document Structure-based Chunking

This method utilizes the inherent structure of documents, such as headings, sections, or paragraphs, to define chunk boundaries. It maintains structural integrity by aligning with the document's logical sections but assumes a clear structure exists.

Source Code for Document Structure-based Chunking

import re

def structure_based_chunking(text):
    # Define patterns for different structural elements
    patterns = {
        'heading': r'^#+\s+.*$',
        'paragraph': r'^(?!#+\s+).*(?:\n(?!#+\s+).+)*',
    }
    
    chunks = []
    lines = text.split('\n')
    current_chunk = ''
    
    for line in lines:
        if re.match(patterns['heading'], line):
            if current_chunk:
                chunks.append(current_chunk.strip())
            current_chunk = line + '\n'
        elif re.match(patterns['paragraph'], line):
            current_chunk += line + '\n'
        else:
            if current_chunk:
                chunks.append(current_chunk.strip())
            current_chunk = ''
    
    if current_chunk:
        chunks.append(current_chunk.strip())
    
    return chunks

text = """# Introduction
This is the introduction paragraph.

## Section 1
This is the first section's content.
It spans multiple lines.

## Section 2
This is the second section's content."""

chunks = structure_based_chunking(text)
for i, chunk in enumerate(chunks, 1):
    print(f"Chunk {i}:\n{chunk}\n")

Results for: Source Code for Document Structure-based Chunking

Chunk 1:
# Introduction
This is the introduction paragraph.

Chunk 2:
## Section 1
This is the first section's content.
It spans multiple lines.

Chunk 3:
## Section 2
This is the second section's content.

LLM-based Chunking

LLM-based chunking leverages language models to create semantically isolated and meaningful chunks. While this method ensures high semantic accuracy, it is computationally demanding and may be limited by the LLM's context window.

Source Code for LLM-based Chunking

def simulate_llm_chunking(text, max_chunk_size=100):
    # This is a simplified simulation of LLM-based chunking
    # In practice, you would use an actual LLM API
    
    words = text.split()
    chunks = []
    current_chunk = []
    current_size = 0
    
    for word in words:
        if current_size + len(word) + 1 > max_chunk_size:
            chunks.append(' '.join(current_chunk))
            current_chunk = [word]
            current_size = len(word)
        else:
            current_chunk.append(word)
            current_size += len(word) + 1
    
    if current_chunk:
        chunks.append(' '.join(current_chunk))
    
    return chunks

text = "This is a simulation of LLM-based chunking. In reality, an LLM would understand context and create more semantically meaningful chunks. This method is computationally expensive but potentially more accurate."

chunks = simulate_llm_chunking(text)
for i, chunk in enumerate(chunks, 1):
    print(f"Chunk {i}: {chunk}")

Results for: Source Code for LLM-based Chunking

Chunk 1: This is a simulation of LLM-based chunking. In reality, an LLM would understand context and create more
Chunk 2: semantically meaningful chunks. This method is computationally expensive but potentially more accurate.

Real-life Example: Text Summarization

Chunking strategies are crucial in text summarization tasks. For instance, when summarizing a long research paper, semantic chunking can be used to divide the paper into coherent sections, allowing for more accurate summarization of each part.

Real-life Example: Question Answering Systems

In question answering systems, document structure-based chunking can be employed to break down textbooks or manuals. This allows the system to quickly locate relevant sections when answering specific questions about the content.

Additional Resources

For more information on chunking strategies and their applications in natural language processing, refer to the following ArXiv papers:

  1. "Efficient Document Retrieval by End-to-End Refining and Chunking" (arXiv:2310.14102)
  2. "Retrieval-Augmented Generation for Large Language Models: A Survey" (arXiv:2312.10997)

These papers provide in-depth discussions on various chunking techniques and their impact on retrieval-augmented generation systems.

More in RAG and retrieval

  1. Accelerating RAG with Binary Quantization4 min

    Binary Quantization (BQ) is a technique used to compress high-dimensional vectors into compact binary representations. This process significantly reduces...

  2. Addressing LLM Limitations with RAG Document Q&A16 min

    Retrieval Augmented Generation (RAG) is a powerful technique that combines document retrieval with language model generation. The architecture consists of an...

  3. Agentic RAG Transforming Customer Support with Retrieval-Augmented Generation8 min

    The foundational architecture of Agentic RAG implements a dynamic agent system that coordinates between knowledge retrieval and response generation. This...

  4. Auto-Document Retrieval for Improved RAG Systems10 min

    Auto-Document Retrieval is an advanced technique that enhances Retrieval-Augmented Generation (RAG) systems. It autonomously selects and retrieves relevant...

  5. Chunking Strategies for Improved RAG Performance19 min

    Text chunking is a critical preprocessing step in RAG systems that involves breaking down large documents into smaller, meaningful segments. This process...

  6. Comparing RAG and Fine-Tuned Language Models15 min

    The fundamental architectural differences between RAG and fine-tuned models lie in how they process and generate responses. RAG systems employ a retrieval...