xbe.at

LLMs and generative AI / LLM architecture and theory

Structured Output in LLM Applications

8 min read, 15 sections

Contents
  1. Introduction to Structured Output in LLM Applications
  2. Importance of Structured Output
  3. JSON as a Structured Output Format
  4. Implementing Custom Output Parsers
  5. Templating for Structured Output
  6. Handling Errors and Edge Cases
  7. Structured Output for Question Answering
  8. Generating Structured Lists and Tables
  9. Structured Output for Named Entity Recognition
  10. Structured Output for Sentiment Analysis
  11. Structured Output for Text Summarization
  12. Real-life Example: Structured Output for Recipe Generation
  13. Real-life Example: Structured Output for Weather Forecasting
  14. Challenges and Limitations of Structured Output in LLMs
  15. Additional Resources

Introduction to Structured Output in LLM Applications

Structured output in Large Language Model (LLM) applications refers to the process of generating organized, predictable responses that follow a specific format. This approach enhances the usability and interpretability of LLM outputs, making them more suitable for downstream tasks and integration with other systems.

from transformers import pipeline

# Initialize a text-generation pipeline
generator = pipeline('text-generation', model='gpt2')

# Generate structured output
prompt = "Summarize the plot of 'The Great Gatsby' in JSON format:"
response = generator(prompt, max_length=200, num_return_sequences=1)[0]['generated_text']

# Parse the generated JSON
try:
    structured_output = json.loads(response.split("JSON format:")[1].strip())
    print(json.dumps(structured_output, indent=2))
except json.JSONDecodeError:
    print("Failed to generate valid JSON structure.")

Importance of Structured Output

Structured output is crucial for maintaining consistency, enabling easy parsing, and facilitating integration with other software components. It allows for more efficient data extraction and analysis, making LLM outputs more valuable in practical applications.


def extract_structured_info(text):
    pattern = r'Name: (.*?)\nAge: (\d+)\nOccupation: (.*?)\n'
    matches = re.findall(pattern, text)
    
    structured_data = [
        {"name": name, "age": int(age), "occupation": occupation}
        for name, age, occupation in matches
    ]
    
    return structured_data

# Example usage
text = """
Name: John Doe
Age: 30
Occupation: Software Engineer

Name: Jane Smith
Age: 28
Occupation: Data Scientist
"""

result = extract_structured_info(text)
print(result)

JSON as a Structured Output Format

JSON (JavaScript Object Notation) is a popular format for structured output due to its simplicity, readability, and widespread support across programming languages.

from transformers import pipeline

def generate_structured_response(prompt):
    generator = pipeline('text-generation', model='gpt2')
    response = generator(prompt, max_length=200, num_return_sequences=1)[0]['generated_text']
    
    # Extract JSON string from the response
    json_str = response.split('```json\n')[1].split('\n```')[0]
    
    return json.loads(json_str)

prompt = "Generate a JSON object describing a book with title, author, and publication year:"
result = generate_structured_response(prompt)
print(json.dumps(result, indent=2))

Implementing Custom Output Parsers

Custom output parsers allow for fine-grained control over the structure of LLM outputs, enabling the extraction of specific information in a desired format.


class PersonParser:
    def __init__(self):
        self.pattern = r'Name: (.*?)\nAge: (\d+)\nProfession: (.*?)\n'
    
    def parse(self, text):
        matches = re.findall(self.pattern, text)
        return [
            {"name": name, "age": int(age), "profession": profession}
            for name, age, profession in matches
        ]

# Example usage
parser = PersonParser()
text = """
Name: Alice Johnson
Age: 35
Profession: Teacher

Name: Bob Williams
Age: 42
Profession: Engineer
"""

parsed_data = parser.parse(text)
print(parsed_data)

Templating for Structured Output

Using templates can help enforce a consistent structure in LLM outputs, making them easier to parse and process.


def generate_structured_output(template, **kwargs):
    return Template(template).safe_substitute(**kwargs)

# Example template
person_template = """
{
  "name": "$name",
  "age": $age,
  "occupation": "$occupation",
  "skills": $skills
}
"""

# Generate structured output
person_data = {
    "name": "Emily Brown",
    "age": 28,
    "occupation": "Software Developer",
    "skills": '["Python", "JavaScript", "Machine Learning"]'
}

structured_output = generate_structured_output(person_template, **person_data)
print(structured_output)

Handling Errors and Edge Cases

Robust error handling and validation are crucial when working with structured outputs from LLMs, as the generated content may not always conform to the expected format.


def parse_json_output(text):
    try:
        # Try to extract JSON from the text
        json_start = text.index('{')
        json_end = text.rindex('}') + 1
        json_str = text[json_start:json_end]
        
        # Parse the JSON string
        data = json.loads(json_str)
        
        # Validate the structure
        required_keys = ['name', 'age', 'occupation']
        if all(key in data for key in required_keys):
            return data
        else:
            raise ValueError("Missing required keys in JSON structure")
    
    except (ValueError, json.JSONDecodeError) as e:
        print(f"Error parsing output: {e}")
        return None

# Example usage
valid_output = '{"name": "John Doe", "age": 30, "occupation": "Engineer"}'
invalid_output = '{"name": "Jane Smith", "occupation": "Teacher"}'

print(parse_json_output(valid_output))
print(parse_json_output(invalid_output))

Structured Output for Question Answering

Implementing structured output in question answering systems can improve the clarity and usability of responses.


def structured_qa(question, context):
    qa_pipeline = pipeline("question-answering")
    
    result = qa_pipeline(question=question, context=context)
    
    structured_answer = {
        "question": question,
        "answer": result["answer"],
        "confidence": round(result["score"], 4),
        "start_index": result["start"],
        "end_index": result["end"]
    }
    
    return structured_answer

# Example usage
context = "The Python programming language was created by Guido van Rossum in 1991."
question = "Who created Python?"

answer = structured_qa(question, context)
print(json.dumps(answer, indent=2))

Generating Structured Lists and Tables

LLMs can be used to generate structured lists and tables, which are particularly useful for presenting information in a clear and organized manner.

from transformers import pipeline

def generate_structured_table(prompt):
    generator = pipeline('text-generation', model='gpt2')
    response = generator(prompt, max_length=200, num_return_sequences=1)[0]['generated_text']
    
    # Extract table data from the response
    lines = response.strip().split('\n')
    header = lines[0].split('|')
    data = [line.split('|') for line in lines[2:]]
    
    # Create a pandas DataFrame
    df = pd.DataFrame(data, columns=header)
    return df

prompt = "Generate a table of top 5 programming languages with their creator and year of creation:"
table = generate_structured_table(prompt)
print(table.to_string(index=False))

Structured Output for Named Entity Recognition

Implementing structured output for Named Entity Recognition (NER) tasks can enhance the interpretability and usability of the results.


def structured_ner(text):
    ner_pipeline = pipeline("ner")
    results = ner_pipeline(text)
    
    structured_entities = {}
    for result in results:
        entity_type = result['entity']
        if entity_type not in structured_entities:
            structured_entities[entity_type] = []
        structured_entities[entity_type].append({
            'word': result['word'],
            'score': round(result['score'], 4),
            'start': result['start'],
            'end': result['end']
        })
    
    return structured_entities

# Example usage
text = "Apple Inc. was founded by Steve Jobs and Steve Wozniak in Cupertino, California."
entities = structured_ner(text)
print(json.dumps(entities, indent=2))

Structured Output for Sentiment Analysis

Implementing structured output for sentiment analysis can provide more detailed and actionable insights.


def structured_sentiment_analysis(text):
    sentiment_pipeline = pipeline("sentiment-analysis")
    result = sentiment_pipeline(text)[0]
    
    structured_result = {
        "text": text,
        "sentiment": result["label"],
        "confidence": round(result["score"], 4),
        "polarity": 1 if result["label"] == "POSITIVE" else -1
    }
    
    return structured_result

# Example usage
text = "I really enjoyed the new movie. The plot was engaging and the acting was superb!"
sentiment = structured_sentiment_analysis(text)
print(json.dumps(sentiment, indent=2))

Structured Output for Text Summarization

Implementing structured output for text summarization can provide more context and metadata about the generated summary.


def structured_summarization(text, max_length=150, min_length=50):
    summarizer = pipeline("summarization")
    summary = summarizer(text, max_length=max_length, min_length=min_length, do_sample=False)[0]
    
    structured_summary = {
        "original_text": text,
        "summary": summary["summary_text"],
        "original_length": len(text.split()),
        "summary_length": len(summary["summary_text"].split()),
        "compression_ratio": round(len(summary["summary_text"].split()) / len(text.split()), 2)
    }
    
    return structured_summary

# Example usage
long_text = """
Climate change is one of the most pressing issues facing our planet today. It refers to long-term shifts in temperatures and weather patterns, mainly caused by human activities, especially the burning of fossil fuels. These activities release greenhouse gases into the atmosphere, trapping heat and causing the Earth's average temperature to rise. The effects of climate change are far-reaching and include more frequent and severe weather events, rising sea levels, and disruptions to ecosystems and biodiversity.
"""

summary = structured_summarization(long_text)
print(json.dumps(summary, indent=2))

Real-life Example: Structured Output for Recipe Generation

This example demonstrates how structured output can be used to generate cooking recipes in a format that's easy to follow and integrate into applications.

from transformers import pipeline

def generate_structured_recipe(dish_name):
    generator = pipeline('text-generation', model='gpt2')
    prompt = f"Generate a recipe for {dish_name} in JSON format with ingredients and steps:"
    
    response = generator(prompt, max_length=500, num_return_sequences=1)[0]['generated_text']
    
    # Extract JSON from the response
    json_start = response.index('{')
    json_end = response.rindex('}') + 1
    recipe_json = response[json_start:json_end]
    
    # Parse and structure the recipe
    try:
        recipe = json.loads(recipe_json)
        structured_recipe = {
            "name": recipe.get("name", dish_name),
            "ingredients": recipe.get("ingredients", []),
            "steps": recipe.get("steps", []),
            "prep_time": recipe.get("prep_time", "N/A"),
            "cook_time": recipe.get("cook_time", "N/A"),
            "servings": recipe.get("servings", "N/A")
        }
        return structured_recipe
    except json.JSONDecodeError:
        return {"error": "Failed to generate a valid recipe structure"}

# Example usage
dish = "Vegetarian Lasagna"
recipe = generate_structured_recipe(dish)
print(json.dumps(recipe, indent=2))

Real-life Example: Structured Output for Weather Forecasting

This example shows how structured output can be used to generate weather forecasts in a format that's easy to process and display in various applications.

from transformers import pipeline

def generate_structured_weather_forecast(location, days=5):
    generator = pipeline('text-generation', model='gpt2')
    prompt = f"Generate a {days}-day weather forecast for {location} in JSON format:"
    
    response = generator(prompt, max_length=500, num_return_sequences=1)[0]['generated_text']
    
    # Extract JSON from the response
    json_start = response.index('{')
    json_end = response.rindex('}') + 1
    forecast_json = response[json_start:json_end]
    
    # Parse and structure the forecast
    try:
        forecast = json.loads(forecast_json)
        structured_forecast = {
            "location": forecast.get("location", location),
            "unit": forecast.get("unit", "Celsius"),
            "forecast": [
                {
                    "date": day.get("date", f"Day {i+1}"),
                    "temperature": day.get("temperature", "N/A"),
                    "condition": day.get("condition", "N/A"),
                    "precipitation": day.get("precipitation", "N/A")
                }
                for i, day in enumerate(forecast.get("forecast", []))
            ]
        }
        return structured_forecast
    except json.JSONDecodeError:
        return {"error": "Failed to generate a valid forecast structure"}

# Example usage
location = "New York City"
forecast = generate_structured_weather_forecast(location)
print(json.dumps(forecast, indent=2))

Challenges and Limitations of Structured Output in LLMs

While structured output offers many benefits, it also comes with challenges:

  1. Inconsistency: LLMs may sometimes generate outputs that deviate from the expected structure.
  2. Hallucination: LLMs might include fictitious information in the structured output.
  3. Context limitations: The model's understanding of context can be limited, affecting the accuracy of structured outputs.
  4. Parsing complexity: Complex structures may require sophisticated parsing techniques.

To address these challenges:

from jsonschema import validate, ValidationError

def validate_structured_output(output, schema):
    try:
        # Parse the output as JSON
        parsed_output = json.loads(output)
        
        # Validate against the schema
        validate(instance=parsed_output, schema=schema)
        
        return parsed_output
    except json.JSONDecodeError:
        return {"error": "Invalid JSON format"}
    except ValidationError as e:
        return {"error": f"Schema validation failed: {e.message}"}

# Example schema for a person
person_schema = {
    "type": "object",
    "properties": {
        "name": {"type": "string"},
        "age": {"type": "integer", "minimum": 0},
        "occupation": {"type": "string"}
    },
    "required": ["name", "age", "occupation"]
}

# Test with valid and invalid outputs
valid_output = '{"name": "John Doe", "age": 30, "occupation": "Engineer"}'
invalid_output = '{"name": "Jane Smith", "age": "twenty-eight", "occupation": "Teacher"}'

print(validate_structured_output(valid_output, person_schema))
print(validate_structured_output(invalid_output, person_schema))

Additional Resources

For further exploration of structured output in LLM applications, consider the following resources:

  1. "Structured Prompting: Scaling In-Context Learning to 1,000 Examples" (arXiv:2212.06713) - This paper discusses techniques for improving structured output generation in LLMs.
  2. "Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm" (arXiv:2102.07350) - This work explores advanced prompting techniques that can be applied to generate more reliable structured outputs.
  3. "Language Models are Few-Shot Learners" (arXiv:2005.14165) - While not specifically about structured output, this seminal paper on GPT-3 provides important context for understanding the

More in LLM architecture and theory

  1. The Exponential Growth of Large Language Models12 min

    Neural network pruning aims to reduce model size by systematically removing weights based on their importance. The gradient-based approach evaluates each...

  2. Self-Harmonized Chain of Thought in Python7 min

    Self-Harmonized Chain of Thought (SH-CoT) is an advanced prompting technique that enhances language models' reasoning capabilities. It builds upon the Chain...

  3. Transformer Architecture for Efficient LLM Processing13 min

    The self-attention mechanism calculates attention scores between all pairs of input tokens, enabling the model to weigh the importance of different parts of...

  4. Roadmap for Developing a LLM from Harry Potter Books8 min

    This slide introduces the process of tokenizing text from Harry Potter books. We'll use the NLTK library to tokenize the text into words.

  5. Transformers and Attention Mechanisms in Large Language Models9 min

    The attention mechanism enables neural networks to selectively focus on specific parts of the input sequence when generating output. This fundamental concept...

  6. Rigorous Unit Testing for LLM Outputs with DeepEval7 min

    DeepEval is a Python framework designed specifically for rigorous testing of Large Language Model outputs. It extends traditional unit testing paradigms to...