xbe.at

Python / Data structures and algorithms

Increase Code Efficiency by using Dictionary Comprehensions

7 min read, 15 sections

Contents
  1. Dictionary Comprehension Basics
  2. Conditional Dictionary Comprehension
  3. String Processing with Dictionary Comprehension
  4. Nested Dictionary Comprehension
  5. Dictionary Comprehension with Zip
  6. Real-world Example - Data Preprocessing
  7. Performance Optimization with Dictionary Comprehension
  8. Dictionary Comprehension with Complex Transformations
  9. Error Handling in Dictionary Comprehensions
  10. Memory-Efficient Dictionary Comprehensions
  11. Real-world Example - Time Series Processing
  12. Advanced Pattern Matching with Dictionary Comprehension
  13. Computational Efficiency Analysis
  14. Real-world Example - Data Analysis Pipeline
  15. Additional Resources

Dictionary Comprehension Basics

Dictionary comprehensions provide a concise way to create dictionaries using a single-line expression. They follow a similar syntax to list comprehensions but use curly braces and require both a key and value expression, separated by a colon, making code more readable and maintainable.

# Traditional for loop approach
numbers = range(5)
squares_dict = {}
for num in numbers:
    squares_dict[num] = num ** 2
print(squares_dict)  # Output: {0: 0, 1: 1, 2: 4, 3: 9, 4: 16}

# Dictionary comprehension approach
squares_dict = {num: num ** 2 for num in range(5)}
print(squares_dict)  # Output: {0: 0, 1: 1, 2: 4, 3: 9, 4: 16}

Conditional Dictionary Comprehension

Dictionary comprehensions can incorporate conditional logic to filter key-value pairs based on specific criteria. This allows for more sophisticated dictionary creation while maintaining code elegance and efficiency compared to traditional loop structures.

# Creating a dictionary of even squares only
numbers = range(10)
even_squares = {x: x**2 for x in numbers if x % 2 == 0}
print(even_squares)  # Output: {0: 0, 2: 4, 4: 16, 6: 36, 8: 64}

# Multiple conditions example
complex_dict = {x: x**2 for x in numbers if x % 2 == 0 and x > 4}
print(complex_dict)  # Output: {6: 36, 8: 64}

String Processing with Dictionary Comprehension

Dictionary comprehensions excel at text processing tasks by creating mappings between characters or words and their properties. This technique is particularly useful in natural language processing and text analysis applications.

# Character frequency counter
text = "hello world"
char_freq = {char: text.count(char) for char in set(text)}
print(char_freq)  # Output: {'h': 1, 'e': 1, 'l': 3, 'o': 2, 'w': 1, 'r': 1, 'd': 1, ' ': 1}

# Word length mapping
sentence = "python is awesome"
word_lengths = {word: len(word) for word in sentence.split()}
print(word_lengths)  # Output: {'python': 6, 'is': 2, 'awesome': 7}

Nested Dictionary Comprehension

Nested dictionary comprehensions enable the creation of complex, multi-level dictionary structures in a single expression. This powerful feature allows for sophisticated data transformations while maintaining code clarity.

matrix = {
    i: {j: i*j for j in range(3)}
    for i in range(3)
}
print(matrix)  
# Output: {
#     0: {0: 0, 1: 0, 2: 0}, 
#     1: {0: 0, 1: 1, 2: 2}, 
#     2: {0: 0, 1: 2, 2: 4}
# }

# Transpose matrix using nested comprehension
transposed = {
    i: {j: matrix[j][i] for j in matrix}
    for i in matrix[0]
}
print(transposed)

Dictionary Comprehension with Zip

The zip function combined with dictionary comprehension creates powerful data transformation patterns. This technique is particularly useful when mapping corresponding elements from multiple iterables into a dictionary structure.

# Combining two lists into a dictionary
keys = ['name', 'age', 'city']
values = ['Alice', 25, 'New York']
person = {k: v for k, v in zip(keys, values)}
print(person)  # Output: {'name': 'Alice', 'age': 25, 'city': 'New York'}

# Creating a dictionary from two parallel lists with filtering
filtered_dict = {k: v for k, v in zip(keys, values) if isinstance(v, str)}
print(filtered_dict)  # Output: {'name': 'Alice', 'city': 'New York'}

Real-world Example - Data Preprocessing

Dictionary comprehensions shine in data preprocessing scenarios, offering efficient ways to transform and clean raw data. This example demonstrates cleaning and transforming a dataset of customer information.

# Raw customer data
raw_data = [
    {'customer_id': '001', 'age': '25', 'purchase': '$150.50'},
    {'customer_id': '002', 'age': 'unknown', 'purchase': '$75.25'},
    {'customer_id': '003', 'age': '31', 'purchase': 'NA'}
]

# Clean and transform data using dictionary comprehension
cleaned_data = {
    item['customer_id']: {
        'age': int(item['age']) if item['age'].isdigit() else None,
        'purchase': float(item['purchase'].replace('$', '')) 
            if item['purchase'] != 'NA' else 0.0
    }
    for item in raw_data
}
print(cleaned_data)

Performance Optimization with Dictionary Comprehension

Dictionary comprehensions often outperform traditional loops when creating dictionaries, especially for large datasets. By avoiding repeated method calls and minimizing interpreter overhead, they provide both cleaner code and better performance characteristics.

import time
import random

# Generate test data
data = list(range(1000000))

# Traditional loop approach
start = time.time()
result_loop = {}
for i in data:
    result_loop[i] = i * 2
loop_time = time.time() - start

# Dictionary comprehension approach
start = time.time()
result_comp = {i: i * 2 for i in data}
comp_time = time.time() - start

print(f"Loop time: {loop_time:.4f} seconds")
print(f"Comprehension time: {comp_time:.4f} seconds")
# Output will show comprehension is typically faster

Dictionary Comprehension with Complex Transformations

When working with data that requires sophisticated transformations, dictionary comprehensions can incorporate lambda functions and complex expressions while maintaining readability and performance.

# Complex data transformation
data = {'a': [1, 2, 3], 'b': [4, 5, 6], 'c': [7, 8, 9]}

# Calculate statistics for each key
stats = {
    key: {
        'mean': sum(values) / len(values),
        'max': max(values),
        'min': min(values),
        'range': max(values) - min(values)
    }
    for key, values in data.items()
}
print(stats)

Error Handling in Dictionary Comprehensions

While dictionary comprehensions don't support try-except blocks directly, we can implement error handling using helper functions or conditional expressions to handle potential errors gracefully.

def safe_convert(x):
    try:
        return float(x)
    except (ValueError, TypeError):
        return None

# Raw data with potential conversion issues
raw_values = {'a': '1.23', 'b': 'error', 'c': '4.56', 'd': None}

# Safe conversion using helper function
converted = {
    k: safe_convert(v) 
    for k, v in raw_values.items()
}
print(converted)  # Output: {'a': 1.23, 'b': None, 'c': 4.56, 'd': None}

# Alternative using conditional expression
converted_alt = {
    k: float(v) if isinstance(v, str) and v.replace('.','').isdigit() else None 
    for k, v in raw_values.items()
}
print(converted_alt)

Memory-Efficient Dictionary Comprehensions

When dealing with large datasets, memory efficiency becomes crucial. This example demonstrates how to use generator expressions within dictionary comprehensions to process large datasets efficiently.

from itertools import islice

def generate_large_dataset():
    i = 0
    while True:
        yield f"item_{i}", i * i
        i += 1

# Memory-efficient dictionary creation
large_dict = {
    k: v 
    for k, v in islice(generate_large_dataset(), 1000000)
    if v % 2 == 0  # Only keep even squares
}

# Print first 5 items
print(dict(list(large_dict.items())[:5]))
# Memory usage stays constant regardless of dictionary size

Real-world Example - Time Series Processing

Dictionary comprehensions excel at processing time series data, offering elegant solutions for data transformation and analysis tasks commonly found in financial and scientific applications.

from datetime import datetime, timedelta

# Generate sample time series data
base_date = datetime(2024, 1, 1)
raw_data = {
    (base_date + timedelta(days=i)).strftime('%Y-%m-%d'): 
    round(random.uniform(10, 100), 2)
    for i in range(10)
}

# Calculate moving averages
window_size = 3
moving_averages = {
    date: sum(
        list(raw_data.values())[i:i+window_size]
    ) / window_size
    for i, date in enumerate(raw_data.keys())
    if i <= len(raw_data) - window_size
}

print("Raw data:", raw_data)
print("\nMoving averages:", moving_averages)

Advanced Pattern Matching with Dictionary Comprehension

Dictionary comprehensions can implement sophisticated pattern matching and filtering operations, particularly useful in data cleaning and text analysis applications where complex transformations are required.

import re

# Sample text data with patterns
raw_text = {
    'user1': 'email: [email protected], phone: 123-456-7890',
    'user2': 'email: invalid@, phone: 987-654-3210',
    'user3': 'email: [email protected], phone: invalid'
}

# Extract and validate email and phone patterns
patterns = {
    'email': r'[\w\.-]+@[\w\.-]+\.\w+',
    'phone': r'\d{3}-\d{3}-\d{4}'
}

validated_data = {
    user: {
        field: re.search(pattern, text).group(0)
        for field, pattern in patterns.items()
        if re.search(pattern, text)
    }
    for user, text in raw_text.items()
}

print(validated_data)

Computational Efficiency Analysis

Dictionary comprehensions demonstrate significant performance advantages in numerical computations and data transformations compared to traditional loop-based approaches, especially for mathematical operations.

import timeit

# Test data size
N = 1000000

# Function with traditional loop
def traditional_loop():
    result = {}
    for i in range(N):
        if i % 2 == 0:
            result[i] = (i**2 + 2*i + 1)
    return result

# Function with dictionary comprehension
def dict_comprehension():
    return {i: (i**2 + 2*i + 1) for i in range(N) if i % 2 == 0}

# Performance comparison
loop_time = timeit.timeit(traditional_loop, number=1)
comp_time = timeit.timeit(dict_comprehension, number=1)

print(f"Traditional loop time: {loop_time:.4f} seconds")
print(f"Dictionary comprehension time: {comp_time:.4f} seconds")
print(f"Performance improvement: {((loop_time - comp_time) / loop_time * 100):.2f}%")

Real-world Example - Data Analysis Pipeline

This example demonstrates a complete data analysis pipeline using dictionary comprehensions to process and analyze scientific measurement data, including error handling and statistical calculations.

import statistics
from typing import Dict, List, Union

# Sample experimental data with potential errors
raw_measurements = {
    'experiment_1': ['23.5', '24.1', 'error', '23.8', '24.2'],
    'experiment_2': ['19.8', '20.1', '19.9', 'invalid', '20.3'],
    'experiment_3': ['25.4', '25.7', '25.9', '25.5', 'error']
}

def process_measurements(data: List[str]) -> List[float]:
    return [float(x) for x in data if x.replace('.', '').isdigit()]

# Process and analyze data
analysis_results = {
    exp_name: {
        'valid_measurements': processed_data := process_measurements(measurements),
        'mean': round(statistics.mean(processed_data), 2),
        'std_dev': round(statistics.stdev(processed_data), 2),
        'sample_size': len(processed_data),
        'data_quality': len(processed_data) / len(measurements) * 100
    }
    for exp_name, measurements in raw_measurements.items()
}

print(analysis_results)

Additional Resources

More in Data structures and algorithms

  1. Linear and Binary Search Algorithms in Python8 min

    Linear search is a simple search algorithm used to find an element in a list or array. It works by iterating through the elements one by one, comparing each...

  2. Implementing Stacks in Python11 min

    A stack is a linear data structure that follows the Last-In-First-Out (LIFO) principle, where elements are added and removed from the same end. This...

  3. Lists and List Functions in Python7 min

    Lists serve as fundamental data structures in Python, offering a mutable and ordered sequence of elements. They can store heterogeneous data types, making...

  4. Implementing Python Data Structures Without External Libraries4 min

    The underscore (\) in Python is a versatile character with multiple uses. It serves as a convention to indicate that a value is meant to be ignored or is...

  5. Locale-Aware Sorting in Python11 min

    The locale module provides basic functionality for locale-aware string sorting through the strcoll function, which compares two strings according to the...

  6. Implementing Graph Data Structures in Python9 min

    A graph is a non-linear data structure consisting of vertices (nodes) and edges connecting these vertices. In Python, we can implement a graph using...