xbe.at

Data science

Pandas AutoProfiler Automated DataFrame Analysis

7 min read, 15 sections

Contents
  1. Introduction to Pandas AutoProfiler
  2. Basic DataFrame Profiling
  3. Customizing Profile Settings
  4. Real-time Data Quality Monitoring
  5. Distribution Analysis and Visualization
  6. Statistical Summary Generation
  7. Missing Value Analysis and Handling
  8. Correlation Analysis and Feature Relationships
  9. Automated Feature Engineering
  10. Real-world Example - Customer Transaction Analysis
  11. Results Analysis for Transaction Profiling
  12. Real-world Example - Sensor Data Analysis
  13. Results Analysis for Sensor Data
  14. Advanced Configuration and Optimization
  15. Additional Resources

Introduction to Pandas AutoProfiler

AutoProfiler is an advanced DataFrame profiling tool that automatically generates comprehensive statistical analysis and visual representations of Pandas DataFrames. It provides instant insights into data quality, distributions, correlations, and missing values without writing explicit profiling code.

# Install the autoprofiler package
!pip install pandas-autoprofiler

# Import required libraries
import pandas as pd
from pandas_autoprofiler import AutoProfiler

# Initialize the AutoProfiler
profiler = AutoProfiler()

# Enable automatic profiling for all DataFrames
profiler.enable()

Basic DataFrame Profiling

The automatic profiling begins as soon as a DataFrame is created or modified. The profiler generates detailed statistics including data types, missing values, unique values, and basic statistical measures for each column.

# Create a sample DataFrame
import numpy as np
df = pd.DataFrame({
    'numeric': np.random.normal(0, 1, 1000),
    'categorical': np.random.choice(['A', 'B', 'C'], 1000),
    'datetime': pd.date_range('2023-01-01', periods=1000),
    'missing': np.random.choice([np.nan, 1, 2], 1000)
})

# Profile is automatically generated
# Access the profile
profile = profiler.get_profile(df)
print(profile.summary())

Customizing Profile Settings

AutoProfiler allows customization of profiling parameters including sampling rate, correlation methods, and visualization preferences. These settings can be adjusted to optimize performance for large datasets while maintaining statistical accuracy.

# Configure AutoProfiler with custom settings
profiler = AutoProfiler(
    sample_size=10000,          # Maximum number of rows to analyze
    correlation_threshold=0.7,   # Minimum correlation coefficient to report
    include_plots=True,         # Generate distribution plots
    categorical_max_unique=50,   # Max unique values for categorical analysis
    datetime_analysis=True      # Enable datetime feature analysis
)

# Apply configuration
profiler.enable(config_override=True)

Real-time Data Quality Monitoring

The real-time monitoring capability of AutoProfiler tracks changes in data quality metrics as DataFrames are modified. This feature is essential for detecting data drift and maintaining data quality in production environments.

# Enable real-time monitoring
profiler.enable_monitoring()

# Create initial DataFrame
df = pd.DataFrame({'values': np.random.normal(0, 1, 1000)})

# Monitor changes
for i in range(3):
    # Modify DataFrame
    df['values'] = df['values'] + np.random.normal(0, 0.1, 1000)
    
    # Get quality metrics
    quality_metrics = profiler.get_quality_metrics(df)
    print(f"Iteration {i} metrics:", quality_metrics)

Distribution Analysis and Visualization

The distribution analysis module automatically identifies the best-fitting statistical distributions for numerical columns and generates visualizations for both continuous and discrete variables.

# Configure distribution analysis
profiler.set_distribution_analysis(
    bins=50,                    # Number of bins for histograms
    fit_distributions=True,     # Attempt to fit statistical distributions
    kde_bandwidth='scott',      # Kernel density estimation bandwidth
    test_normality=True        # Perform normality tests
)

# Generate distribution report
df = pd.DataFrame({
    'normal': np.random.normal(0, 1, 1000),
    'exponential': np.random.exponential(2, 1000)
})

distribution_report = profiler.analyze_distributions(df)

Statistical Summary Generation

AutoProfiler generates comprehensive statistical summaries including measures of central tendency, dispersion, and shape. The analysis automatically adapts to the data type of each column.

# Generate statistical summary
def generate_summary(df):
    summary = profiler.get_statistics(df)
    
    # Numerical statistics
    numeric_stats = summary['numeric_summary']
    print("Numeric Column Statistics:")
    print(numeric_stats)
    
    # Categorical statistics
    categorical_stats = summary['categorical_summary']
    print("\nCategorical Column Statistics:")
    print(categorical_stats)
    
    return summary

# Example usage
df = pd.DataFrame({
    'numeric': np.random.normal(0, 1, 1000),
    'categorical': np.random.choice(['A', 'B', 'C'], 1000)
})

stats = generate_summary(df)

Missing Value Analysis and Handling

AutoProfiler provides sophisticated missing value analysis, including pattern detection, correlation between missing values, and impact assessment. The tool automatically identifies potential reasons for missingness and suggests appropriate handling strategies.

# Configure missing value analysis
profiler.set_missing_analysis(
    pattern_analysis=True,      # Analyze missing patterns
    correlation_analysis=True,  # Check correlations between missing values
    impact_analysis=True       # Assess impact on other variables
)

# Create DataFrame with missing values
df = pd.DataFrame({
    'A': [1, np.nan, 3, np.nan, 5],
    'B': [np.nan, 2, 3, 4, 5],
    'C': [1, 2, np.nan, 4, 5]
})

# Get missing value report
missing_report = profiler.analyze_missing(df)
print(missing_report.summary())

Correlation Analysis and Feature Relationships

The correlation analysis module automatically detects and visualizes relationships between variables using multiple correlation methods. It adapts the analysis method based on data types and distributions.

# Configure correlation analysis
profiler.set_correlation_analysis(
    methods=['pearson', 'spearman', 'kendall'],
    plot_matrix=True,
    threshold=0.5
)

# Generate synthetic correlated data
n_samples = 1000
x = np.random.normal(0, 1, n_samples)
df = pd.DataFrame({
    'x': x,
    'y': 2*x + np.random.normal(0, 0.5, n_samples),
    'z': -0.5*x + np.random.normal(0, 0.8, n_samples)
})

# Get correlation report
correlation_report = profiler.analyze_correlations(df)
print("Correlation Analysis Results:")
print(correlation_report.get_strong_correlations())

Automated Feature Engineering

AutoProfiler includes automated feature engineering capabilities that detect and create meaningful derived features based on existing columns, particularly useful for datetime and categorical variables.

# Configure feature engineering
profiler.set_feature_engineering(
    datetime_features=True,     # Extract datetime components
    categorical_encoding=True,  # Automatic encoding of categories
    interaction_terms=True      # Generate interaction features
)

# Create sample DataFrame
df = pd.DataFrame({
    'date': pd.date_range('2023-01-01', periods=100),
    'category': np.random.choice(['A', 'B', 'C'], 100),
    'value': np.random.normal(0, 1, 100)
})

# Generate engineered features
engineered_df = profiler.engineer_features(df)
print("Original shape:", df.shape)
print("Engineered shape:", engineered_df.shape)

Real-world Example - Customer Transaction Analysis

Implementation of AutoProfiler for analyzing customer transaction data, demonstrating its capability to handle complex real-world datasets with multiple data types and quality issues.

# Create realistic transaction dataset
np.random.seed(42)
n_transactions = 5000

transactions = pd.DataFrame({
    'transaction_id': range(n_transactions),
    'date': pd.date_range('2023-01-01', periods=n_transactions, freq='H'),
    'amount': np.random.exponential(100, n_transactions),
    'customer_id': np.random.randint(1, 1000, n_transactions),
    'merchant_category': np.random.choice(['Food', 'Retail', 'Travel', 'Online'], n_transactions),
    'status': np.random.choice(['approved', 'declined', 'pending'], n_transactions,
                              p=[0.85, 0.10, 0.05])
})

# Initialize and configure profiler for transactions
profiler = AutoProfiler(
    sample_size=None,  # Analyze all records
    datetime_analysis=True,
    categorical_max_unique=100
)

# Generate comprehensive profile
transaction_profile = profiler.generate_profile(transactions)
print(transaction_profile.get_insights())

Results Analysis for Transaction Profiling

The detailed results from the transaction data analysis showcase the AutoProfiler's ability to generate meaningful insights and identify patterns in complex datasets.

# Display key findings from transaction analysis
def display_transaction_insights(profile):
    # Transaction patterns
    print("Daily Transaction Patterns:")
    print(profile.time_series_analysis())
    
    # Amount distribution
    print("\nTransaction Amount Statistics:")
    print(profile.get_numeric_stats('amount'))
    
    # Customer segmentation
    print("\nCustomer Activity Patterns:")
    print(profile.get_categorical_stats('customer_id'))
    
    # Merchant category analysis
    print("\nMerchant Category Distribution:")
    print(profile.get_categorical_stats('merchant_category'))
    
    # Transaction status analysis
    print("\nTransaction Status Summary:")
    print(profile.get_categorical_stats('status'))

# Generate and display insights
display_transaction_insights(transaction_profile)

Real-world Example - Sensor Data Analysis

AutoProfiler application for IoT sensor data analysis, demonstrating its capabilities in handling time-series data with multiple sensors and environmental conditions.

# Generate realistic sensor data
n_readings = 10000
sensors_df = pd.DataFrame({
    'timestamp': pd.date_range('2023-01-01', periods=n_readings, freq='5min'),
    'temperature': np.random.normal(25, 3, n_readings),
    'humidity': np.random.normal(60, 10, n_readings),
    'pressure': np.random.normal(1013, 5, n_readings),
    'sensor_id': np.random.choice(['S1', 'S2', 'S3', 'S4'], n_readings),
    'location': np.random.choice(['Room1', 'Room2', 'Room3'], n_readings)
})

# Add some realistic patterns and anomalies
sensors_df['temperature'] += np.sin(np.arange(n_readings) * 2 * np.pi / 288) * 2  # Daily cycle
sensors_df.loc[np.random.choice(n_readings, 50), 'temperature'] = np.nan  # Missing values

# Configure profiler for sensor data
sensor_profiler = AutoProfiler(
    time_series_analysis=True,
    anomaly_detection=True,
    correlation_threshold=0.3
)

# Generate sensor data profile
sensor_profile = sensor_profiler.generate_profile(sensors_df)

Results Analysis for Sensor Data

Comprehensive analysis of the sensor data profile, showing time-series patterns, anomaly detection, and cross-sensor correlations.

# Analyze sensor data results
def analyze_sensor_results(profile):
    # Time series decomposition
    print("Temperature Time Series Analysis:")
    temperature_analysis = profile.analyze_time_series('temperature')
    print(temperature_analysis)
    
    # Cross-correlation between sensors
    print("\nSensor Correlations:")
    correlation_matrix = profile.get_correlation_matrix(['temperature', 'humidity', 'pressure'])
    print(correlation_matrix)
    
    # Anomaly detection results
    print("\nDetected Anomalies:")
    anomalies = profile.get_anomalies('temperature')
    print(f"Number of anomalies detected: {len(anomalies)}")
    
    # Location-based statistics
    print("\nLocation-wise Statistics:")
    location_stats = profile.get_grouped_stats('location')
    print(location_stats)

# Execute analysis
analyze_sensor_results(sensor_profile)

Advanced Configuration and Optimization

Detailed exploration of AutoProfiler's advanced configuration options for optimizing performance and customizing analysis based on specific data requirements and computational constraints.

# Advanced configuration setup
advanced_profiler = AutoProfiler(
    # Performance optimization
    chunk_size=5000,            # Process data in chunks
    parallel_processing=True,   # Enable multiprocessing
    n_jobs=-1,                 # Use all available cores
    
    # Analysis customization
    custom_metrics={
        'skewness': lambda x: x.skew(),
        'kurtosis': lambda x: x.kurtosis()
    },
    
    # Visualization settings
    plot_settings={
        'style': 'seaborn',
        'palette': 'viridis',
        'figure_size': (12, 8)
    },
    
    # Memory management
    low_memory_mode=True,      # Optimize for memory usage
    cache_results=True         # Cache intermediate results
)

# Example usage with optimization
def optimize_profiling(df, profiler):
    # Configure memory-efficient processing
    with profiler.optimization_context():
        # Generate profile with progress tracking
        profile = profiler.generate_profile(
            df,
            progress_callback=lambda x: print(f"Processing: {x}% complete")
        )
    return profile

# Test optimization
test_df = pd.DataFrame(np.random.randn(100000, 5))
optimized_profile = optimize_profiling(test_df, advanced_profiler)

Additional Resources

More in Data science

  1. Pandas DataFrame Cheat Sheet6 min

    A DataFrame is a 2-dimensional labeled data structure in Pandas, similar to a spreadsheet or SQL table. It's the most commonly used Pandas object, capable of...

  2. Optimizing Data Type Conversions in Pandas with astype()10 min

    Data type conversion is a critical operation in data preprocessing that impacts memory usage and computational efficiency. The astype() method in Pandas...

  3. Pandas DataFrame Memory Layout and Efficient Iteration8 min

    The fundamental structure of Pandas DataFrame follows a column-major order where data is stored contiguously in memory by columns rather than rows. This...

  4. NumPy Shortcuts for Efficient Data Analysis6 min

    NumPy is a fundamental library for scientific computing in Python. This guide will walk you through essential NumPy commands and operations, helping you...

  5. Pandas for Data Wrangling and Analysis6 min

    Pandas is a powerful Python library for data manipulation and analysis. It provides data structures like DataFrame and Series, which allow for efficient...

  6. NumPy Fundamentals for TikTok3 min

    Slide 1 (Introduction): NumPy Fundamentals Discover the power of NumPy, the essential library for scientific computing in Python.