xbe.at

Data science

Polars vs Pandas Memory Efficiency and Performance Advantages

7 min read, 14 sections

Contents
  1. Memory Efficiency Through Arrow Memory Format
  2. Parallel Query Execution
  3. Lazy Evaluation Strategy
  4. Vectorized String Operations
  5. Expression-Based API Design
  6. Advanced Time Series Operations
  7. Query Optimization for Large Datasets
  8. Real-world Example - Financial Data Analysis
  9. Memory-Efficient Data Streaming
  10. Results Visualization for Memory Analysis
  11. Real-world Example - IoT Sensor Data Processing
  12. Predicate Pushdown Optimization
  13. Advanced Joins and Aggregations
  14. Additional Resources

Memory Efficiency Through Arrow Memory Format

Polars leverages Apache Arrow's columnar memory format, enabling zero-copy operations and minimizing memory overhead during data processing. This fundamental architectural difference from Pandas results in significantly reduced memory usage when handling large datasets.

import polars as pl
import pandas as pd
import numpy as np
import time

# Create large dataset
n_rows = 1_000_000
data = {
    'id': range(n_rows),
    'values': np.random.randn(n_rows)
}

# Compare memory usage
df_pd = pd.DataFrame(data)
df_pl = pl.DataFrame(data)

print(f"Pandas Memory Usage: {df_pd.memory_usage().sum() / 1024**2:.2f} MB")
print(f"Polars Memory Usage: {df_pl.estimated_size() / 1024**2:.2f} MB")

Parallel Query Execution

Polars automatically parallelizes query operations across available CPU cores, leveraging modern hardware capabilities for data processing tasks. The query optimizer creates efficient execution plans that minimize memory allocations and maximize throughput.

# Comparing execution speed for groupby operations
start_time = time.time()
result_pd = df_pd.groupby('id').agg({'values': ['mean', 'std']})
pd_time = time.time() - start_time

start_time = time.time()
result_pl = df_pl.groupby('id').agg([
    pl.col('values').mean(),
    pl.col('values').std()
])
pl_time = time.time() - start_time

print(f"Pandas execution time: {pd_time:.2f} seconds")
print(f"Polars execution time: {pl_time:.2f} seconds")

Lazy Evaluation Strategy

Polars implements a lazy evaluation system that optimizes query execution by building a computation graph before actual execution. This allows for query optimization and efficient resource utilization compared to Pandas' eager evaluation.

import polars as pl
import numpy as np

# Create large dataset
data = pl.DataFrame({
    'A': np.random.randn(1_000_000),
    'B': np.random.randn(1_000_000)
})

# Define lazy computation
lazy_query = (
    data.lazy()
    .filter(pl.col('A') > 0)
    .groupby(pl.col('A').round(1))
    .agg([
        pl.col('B').mean().alias('B_mean'),
        pl.col('B').std().alias('B_std')
    ])
    .sort('A')
)

# Execute query
result = lazy_query.collect()

Vectorized String Operations

Polars provides highly optimized string operations through a vectorized implementation, resulting in superior performance for text processing tasks compared to Pandas' string operations.

import polars as pl
import pandas as pd
import time

# Create dataset with string operations
n_rows = 1_000_000
data = {
    'text': ['hello_world_' + str(i) for i in range(n_rows)]
}

df_pd = pd.DataFrame(data)
df_pl = pl.DataFrame(data)

# Compare string splitting performance
start_time = time.time()
pd_result = df_pd['text'].str.split('_')
pd_time = time.time() - start_time

start_time = time.time()
pl_result = df_pl['text'].str.split('_')
pl_time = time.time() - start_time

print(f"Pandas string split time: {pd_time:.2f} seconds")
print(f"Polars string split time: {pl_time:.2f} seconds")

Expression-Based API Design

Polars introduces a powerful expression-based API that enables complex data transformations through composable operations. This design allows for more intuitive and maintainable code while maintaining high performance through optimized execution paths.

import polars as pl
import numpy as np

# Create sample dataset
df = pl.DataFrame({
    'date': pl.date_range(
        start=datetime(2023, 1, 1),
        end=datetime(2023, 12, 31),
        interval='1d'
    ),
    'sales': np.random.normal(1000, 100, 365),
    'costs': np.random.normal(800, 50, 365)
})

# Complex transformations using expressions
result = df.select([
    pl.col('date'),
    pl.col('sales').rolling_mean(window_size=7).alias('sales_ma7'),
    (pl.col('sales') - pl.col('costs')).alias('profit'),
    pl.col('sales').pct_change().alias('sales_growth')
]).filter(
    pl.col('profit') > pl.col('profit').mean()
)

Advanced Time Series Operations

Polars excels at time series manipulation through specialized datetime functions and optimized window operations. The framework provides native support for various temporal aggregations and transformations with minimal overhead.

# Time series analytics example
result = df.select([
    pl.col('date'),
    pl.col('sales').rolling_mean(
        window_size='7d',
        by='date',
        closed='right'
    ).alias('weekly_avg'),
    pl.col('sales').rolling_std(
        window_size='30d',
        by='date'
    ).alias('monthly_volatility'),
    pl.col('date').dt.month().alias('month'),
    pl.col('date').dt.year().alias('year')
]).groupby(['year', 'month']).agg([
    pl.col('sales').mean().alias('monthly_avg_sales'),
    pl.col('weekly_avg').last().alias('last_weekly_avg')
])

Query Optimization for Large Datasets

Polars implements sophisticated query optimization techniques including predicate pushdown, projection pushdown, and common subexpression elimination. These optimizations significantly reduce memory usage and computation time for complex queries.

# Example of query optimization benefits
df_large = pl.DataFrame({
    'id': range(10_000_000),
    'value': np.random.randn(10_000_000),
    'category': np.random.choice(['A', 'B', 'C'], 10_000_000)
})

# Complex query with optimization
optimized_query = (
    df_large.lazy()
    .filter(pl.col('value') > 0)
    .groupby('category')
    .agg([
        pl.col('value').mean().alias('avg_value'),
        pl.col('value').quantile(0.95).alias('p95_value')
    ])
    .sort('avg_value', descending=True)
).collect(streaming=True)

Real-world Example - Financial Data Analysis

This example demonstrates Polars' efficiency in processing high-frequency trading data, showcasing its superior performance in handling time-series operations and group-by transformations.

import polars as pl
from datetime import datetime, timedelta

# Generate sample trading data
n_records = 1_000_000
timestamps = [
    datetime(2024, 1, 1) + timedelta(microseconds=i)
    for i in range(n_records)
]

trading_data = pl.DataFrame({
    'timestamp': timestamps,
    'price': np.random.normal(100, 5, n_records),
    'volume': np.random.exponential(1000, n_records),
    'symbol': np.random.choice(['AAPL', 'GOOGL', 'MSFT'], n_records)
})

# Complex financial analysis
analysis_result = (
    trading_data.lazy()
    .with_columns([
        pl.col('timestamp').dt.hour().alias('hour'),
        (pl.col('price') * pl.col('volume')).alias('turnover')
    ])
    .groupby(['symbol', 'hour'])
    .agg([
        pl.col('price').mean().alias('vwap'),
        pl.col('volume').sum().alias('total_volume'),
        pl.col('turnover').sum().alias('total_turnover'),
        pl.col('price').std().alias('price_volatility')
    ])
).collect()

Memory-Efficient Data Streaming

Polars implements streaming capabilities that enable processing of datasets larger than available RAM. This approach maintains constant memory usage regardless of input size by processing data in chunks while preserving query optimization.

import polars as pl
import numpy as np

# Simulate large CSV file creation
def generate_large_csv(filename, n_rows=10_000_000):
    chunk_size = 100_000
    with open(filename, 'w') as f:
        f.write('id,value,category\n')
        for i in range(0, n_rows, chunk_size):
            chunk = pl.DataFrame({
                'id': range(i, min(i + chunk_size, n_rows)),
                'value': np.random.randn(min(chunk_size, n_rows - i)),
                'category': np.random.choice(['A', 'B', 'C'], min(chunk_size, n_rows - i))
            })
            chunk.write_csv(f, has_header=False)

# Stream processing example
streaming_query = (
    pl.scan_csv('large_dataset.csv')
    .filter(pl.col('value') > 0)
    .groupby('category')
    .agg([
        pl.col('value').mean(),
        pl.col('value').count()
    ])
).collect(streaming=True)

Results Visualization for Memory Analysis

This slide demonstrates the performance metrics and memory usage patterns when processing large datasets using Polars compared to traditional approaches.

import matplotlib.pyplot as plt
import psutil
import time

def measure_memory_usage(func):
    process = psutil.Process()
    start_mem = process.memory_info().rss / 1024 / 1024
    start_time = time.time()
    
    result = func()
    
    end_time = time.time()
    end_mem = process.memory_info().rss / 1024 / 1024
    
    return {
        'execution_time': end_time - start_time,
        'memory_delta': end_mem - start_mem,
        'result': result
    }

# Compare memory usage patterns
def process_with_polars():
    return pl.scan_csv('large_dataset.csv').collect()

def process_with_pandas():
    return pd.read_csv('large_dataset.csv')

polars_metrics = measure_memory_usage(process_with_polars)
pandas_metrics = measure_memory_usage(process_with_pandas)

print(f"Polars Memory Usage: {polars_metrics['memory_delta']:.2f} MB")
print(f"Pandas Memory Usage: {pandas_metrics['memory_delta']:.2f} MB")
print(f"Polars Execution Time: {polars_metrics['execution_time']:.2f} s")
print(f"Pandas Execution Time: {pandas_metrics['execution_time']:.2f} s")

Real-world Example - IoT Sensor Data Processing

This example showcases Polars' efficiency in processing time-series sensor data with high-frequency measurements and complex aggregations.

import polars as pl
from datetime import datetime, timedelta

# Generate IoT sensor data
n_sensors = 100
n_measurements = 1_000_000

sensor_data = pl.DataFrame({
    'timestamp': pl.date_range(
        datetime(2024, 1, 1), 
        datetime(2024, 1, 31), 
        n_measurements
    ),
    'sensor_id': np.random.randint(1, n_sensors + 1, n_measurements),
    'temperature': np.random.normal(25, 5, n_measurements),
    'humidity': np.random.normal(60, 10, n_measurements),
    'pressure': np.random.normal(1013, 10, n_measurements)
})

# Complex sensor data analysis
analysis_result = (
    sensor_data.lazy()
    .with_columns([
        pl.col('timestamp').dt.hour().alias('hour'),
        pl.col('timestamp').dt.date().alias('date')
    ])
    .groupby(['sensor_id', 'date'])
    .agg([
        pl.all().mean().suffix('_avg'),
        pl.all().std().suffix('_std'),
        pl.col(['temperature', 'humidity', 'pressure'])
          .quantile(0.95)
          .suffix('_p95')
    ])
    .sort(['sensor_id', 'date'])
).collect()

Predicate Pushdown Optimization

Polars implements advanced predicate pushdown optimization, pushing filter conditions as close as possible to the data source. This optimization significantly reduces the amount of data that needs to be loaded and processed in memory.

import polars as pl
import numpy as np

# Create sample parquet file with partitioned data
df = pl.DataFrame({
    'date': pl.date_range(
        datetime(2023, 1, 1),
        datetime(2023, 12, 31),
        interval='1h'
    ),
    'region': np.random.choice(['NA', 'EU', 'ASIA'], 8760),
    'sales': np.random.normal(1000, 100, 8760)
})

# Example of predicate pushdown
optimized_query = (
    pl.scan_parquet('sales_data.parquet')
    .filter(
        (pl.col('region') == 'NA') & 
        (pl.col('date').dt.year() == 2023)
    )
    .groupby(pl.col('date').dt.month())
    .agg([
        pl.col('sales').sum().alias('monthly_sales'),
        pl.col('sales').mean().alias('avg_daily_sales')
    ])
).collect()

Advanced Joins and Aggregations

Polars provides highly optimized implementations of joins and aggregations that leverage parallel processing and efficient memory management to handle large-scale data operations with superior performance.

import polars as pl
import numpy as np

# Create sample datasets
customers = pl.DataFrame({
    'customer_id': range(1000000),
    'region': np.random.choice(['NA', 'EU', 'ASIA'], 1000000),
    'segment': np.random.choice(['A', 'B', 'C'], 1000000)
})

transactions = pl.DataFrame({
    'transaction_id': range(5000000),
    'customer_id': np.random.randint(0, 1000000, 5000000),
    'amount': np.random.normal(100, 25, 5000000),
    'date': np.random.choice(pl.date_range(
        datetime(2023, 1, 1),
        datetime(2023, 12, 31),
        interval='1d'
    ), 5000000)
})

# Complex join and aggregation
result = (
    transactions.lazy()
    .join(
        customers.lazy(),
        on='customer_id',
        how='left'
    )
    .groupby(['region', 'segment'])
    .agg([
        pl.col('amount').sum().alias('total_amount'),
        pl.col('amount').mean().alias('avg_amount'),
        pl.col('customer_id').n_unique().alias('unique_customers'),
        pl.col('transaction_id').count().alias('transaction_count')
    ])
    .sort(['region', 'total_amount'], descending=True)
).collect()

Additional Resources

More in Data science

  1. Principal Curves for Nonlinear Data Analysis15 min

    Principal curves provide a nonlinear generalization of principal components analysis, offering a smooth, self-consistent curve that passes through the middle...

  2. Pandas vs. Polars Comparing Python Data Processing Libraries11 min

    Data processing is a crucial task in many fields, and Python offers powerful libraries to handle large datasets efficiently. This presentation compares two...

  3. Python Data Science Cheatsheet5 min

    Data Science with Python involves using powerful libraries and tools to analyze, visualize, and interpret data. This cheatsheet covers essential concepts and...

  4. Pandas to Polars Python Data Transformation Guide8 min

    Converting Pandas DataFrame creation patterns to Polars involves understanding the fundamental differences in syntax while maintaining similar functionality....

  5. Python for Data Science and Basics10 min

    Python is a versatile, high-level programming language known for its simplicity and readability. It supports multiple programming paradigms, including...

  6. Pandas Powerful Python Data Wrangling8 min

    Pandas provides powerful DataFrame structures for handling tabular data efficiently. DataFrames can be created from various data sources including...