xbe.at

Miscellany / DataFrames and pandas

Skimpy Comprehensive Data Summarization Tool

6 min read, 14 sections

Contents
  1. Understanding Skimpy's Core Features
  2. Setting Up Custom Summary Parameters
  3. Advanced Data Type Detection
  4. Handling Missing Values Analysis
  5. Statistical Distribution Analysis
  6. Real-time Data Streaming Analysis
  7. Custom Visualization Integration
  8. Correlation Analysis Enhancement
  9. Time Series Feature Analysis
  10. Large Dataset Optimization
  11. Advanced Memory Management
  12. Custom Statistical Metrics Integration
  13. Results Comparison and Export
  14. Additional Resources

Understanding Skimpy's Core Features

Skimpy revolutionizes data analysis by offering comprehensive statistical summaries and visualizations. It extends beyond basic descriptive statistics to provide nuanced insights into data distributions, missing values patterns, and correlations, making it invaluable for exploratory data analysis.

# Installation and basic usage of Skimpy
!pip install skimpy

import pandas as pd
from skimpy import skim

# Load sample dataset
df = pd.read_csv('dataset.csv')

# Generate comprehensive summary
summary = skim(df)
print(summary)

Setting Up Custom Summary Parameters

The flexibility of Skimpy allows analysts to customize the summary output by configuring specific parameters. This enables focused analysis on particular aspects of the dataset while maintaining the structured reporting format that makes Skimpy powerful.

import pandas as pd
from skimpy import skim

# Create sample dataset
data = {
    'numeric': range(100),
    'categorical': ['A', 'B', 'C'] * 33 + ['A'],
    'dates': pd.date_range('2023-01-01', periods=100)
}
df = pd.DataFrame(data)

# Configure custom summary parameters
summary = skim(df, 
    numeric_stats=['mean', 'sd', 'p0', 'p50', 'p100'],
    categorical_stats=['n_unique', 'top_counts'])

Advanced Data Type Detection

Skimpy implements sophisticated algorithms for automatic data type detection, going beyond Pandas' basic dtypes. It recognizes patterns in data to properly categorize fields as numeric, categorical, datetime, or text, ensuring appropriate statistical treatment.

import pandas as pd
import numpy as np
from skimpy import skim

# Create dataset with mixed types
df = pd.DataFrame({
    'mixed_numeric': [1, 2, '3', '4.5', np.nan],
    'mixed_dates': ['2023-01-01', '2023/02/01', None],
    'mixed_categorical': ['A', 1, 2.5, 'B', None]
})

# Skimpy automatically handles mixed types
summary = skim(df, detect_types=True)

Handling Missing Values Analysis

Skimpy provides detailed insights into missing data patterns, computing not just the count but also the distribution of nulls across the dataset. This helps identify potential systematic issues in data collection or processing.

import pandas as pd
import numpy as np
from skimpy import skim

# Create dataset with strategic missing values
df = pd.DataFrame({
    'col1': [1, 2, np.nan, 4, 5],
    'col2': [np.nan, 2, 3, np.nan, 5],
    'col3': [1, np.nan, np.nan, 4, np.nan]
})

# Generate missing values report
summary = skim(df, missing_stats=True)
print(summary.missing_patterns)

Statistical Distribution Analysis

Skimpy generates comprehensive distribution statistics including skewness, kurtosis, and quantile information. This provides deeper insights into data characteristics beyond simple mean and standard deviation measures.

import pandas as pd
import numpy as np
from skimpy import skim

# Generate non-normal distribution
data = {
    'normal': np.random.normal(0, 1, 1000),
    'skewed': np.random.exponential(2, 1000),
    'bimodal': np.concatenate([
        np.random.normal(-2, 0.5, 500),
        np.random.normal(2, 0.5, 500)
    ])
}
df = pd.DataFrame(data)

# Detailed distribution analysis
summary = skim(df, distribution_stats=['skew', 'kurtosis', 'quantiles'])

Real-time Data Streaming Analysis

Skimpy excels at analyzing streaming data by providing incremental statistics updates. This feature enables monitoring of data quality and distribution changes in real-time, essential for production systems handling continuous data flows.

import pandas as pd
from skimpy import skim
import time

class StreamAnalyzer:
    def __init__(self):
        self.buffer = pd.DataFrame()
        
    def analyze_stream(self, new_data):
        self.buffer = pd.concat([self.buffer, new_data]).tail(1000)
        return skim(self.buffer)

# Simulate streaming data
analyzer = StreamAnalyzer()
while True:
    new_data = pd.DataFrame({
        'value': np.random.normal(0, 1, 100),
        'timestamp': pd.Timestamp.now()
    })
    summary = analyzer.analyze_stream(new_data)
    time.sleep(1)  # Analysis every second

Custom Visualization Integration

Skimpy allows seamless integration with custom visualization functions, extending its capabilities beyond standard plots. This enables tailored visual analysis while maintaining the structured summary format.

import seaborn as sns
from skimpy import skim
import matplotlib.pyplot as plt

def custom_violin(data, column):
    plt.figure(figsize=(10, 6))
    sns.violinplot(data=data, y=column)
    return plt.gcf()

# Create dataset
df = pd.DataFrame({
    'values': np.concatenate([
        np.random.normal(0, 1, 1000),
        np.random.normal(3, 0.5, 1000)
    ])
})

# Generate summary with custom plot
summary = skim(df, plots={'values': custom_violin})

Correlation Analysis Enhancement

Skimpy extends traditional correlation analysis by incorporating advanced statistical measures and visualization techniques. It automatically handles different data types and provides insights into complex relationships between variables.

import pandas as pd
import numpy as np
from skimpy import skim

# Generate correlated data
n = 1000
x = np.random.normal(0, 1, n)
y = 0.7 * x + np.random.normal(0, 0.5, n)
z = np.sin(x) + np.random.normal(0, 0.3, n)

df = pd.DataFrame({
    'x': x,
    'y': y,
    'z': z
})

# Enhanced correlation analysis
summary = skim(df, correlation_method=['pearson', 'spearman'],
              correlation_threshold=0.3)

Time Series Feature Analysis

Skimpy implements specialized analytics for time series data, automatically detecting and analyzing temporal patterns, seasonality, and trends. This provides crucial insights for time-dependent data analysis.

import pandas as pd
import numpy as np
from skimpy import skim

# Generate time series data
dates = pd.date_range('2023-01-01', periods=365)
seasonal = np.sin(np.linspace(0, 4*np.pi, 365))
trend = np.linspace(0, 2, 365)
noise = np.random.normal(0, 0.2, 365)

df = pd.DataFrame({
    'date': dates,
    'value': seasonal + trend + noise
})

# Time series specific analysis
summary = skim(df, time_series_stats=True,
              decompose_seasonal=True)

Large Dataset Optimization

Skimpy implements efficient algorithms for handling large datasets, using sampling and parallel processing techniques. This enables quick analysis of massive datasets while maintaining statistical accuracy.

import pandas as pd
import numpy as np
from skimpy import skim

# Generate large dataset
large_df = pd.DataFrame({
    'numeric': np.random.normal(0, 1, 1_000_000),
    'categorical': np.random.choice(['A', 'B', 'C'], 1_000_000),
    'datetime': pd.date_range('2020-01-01', periods=1_000_000, freq='T')
})

# Optimized analysis for large datasets
summary = skim(large_df, 
              sample_size=100_000,  # Use sampling
              n_jobs=-1)           # Use all CPU cores

Advanced Memory Management

Skimpy implements sophisticated memory optimization techniques for handling large datasets efficiently. It uses chunked processing and memory-mapped files to analyze datasets that exceed available RAM while maintaining computational speed.

import pandas as pd
import numpy as np
from skimpy import skim

class ChunkedAnalyzer:
    def __init__(self, chunk_size=10000):
        self.chunk_size = chunk_size
        
    def analyze_large_file(self, filename):
        chunks = pd.read_csv(filename, chunksize=self.chunk_size)
        results = []
        
        for chunk in chunks:
            chunk_summary = skim(chunk)
            results.append(chunk_summary)
            
        return self.merge_summaries(results)
    
    def merge_summaries(self, summaries):
        # Combine chunk summaries
        return pd.concat(summaries).agg(['mean', 'std'])

# Usage example
analyzer = ChunkedAnalyzer()
final_summary = analyzer.analyze_large_file('large_dataset.csv')

Custom Statistical Metrics Integration

Skimpy provides a flexible framework for integrating custom statistical metrics into the analysis pipeline. This allows domain-specific insights while maintaining the structured reporting format that makes Skimpy powerful.

import pandas as pd
import numpy as np
from skimpy import skim
from scipy import stats

def custom_metrics(series):
    return {
        'geometric_mean': stats.gmean(series[series > 0]),
        'coefficient_variation': stats.variation(series),
        'mode_skew': stats.mode(series)[0][0] - np.mean(series)
    }

# Create dataset
df = pd.DataFrame({
    'values': np.random.lognormal(0, 1, 1000)
})

# Add custom metrics to summary
summary = skim(df, custom_statistics=custom_metrics)

Results Comparison and Export

The exported analysis from Skimpy can be used to compare multiple datasets or track changes over time. This feature enables automated quality control and drift detection in production environments.

import pandas as pd
from skimpy import skim
import json

class DatasetComparator:
    def __init__(self):
        self.baseline = None
        
    def set_baseline(self, df):
        self.baseline = skim(df)
        
    def compare_with_baseline(self, new_df):
        current = skim(new_df)
        comparison = {
            'numeric_drift': self._compare_numeric(
                self.baseline, current),
            'distribution_changes': self._compare_distributions(
                self.baseline, current)
        }
        return comparison
        
    def export_comparison(self, comparison, filename):
        with open(filename, 'w') as f:
            json.dump(comparison, f)

# Usage example
comparator = DatasetComparator()
baseline_df = pd.read_csv('baseline_data.csv')
new_df = pd.read_csv('new_data.csv')

comparator.set_baseline(baseline_df)
results = comparator.compare_with_baseline(new_df)
comparator.export_comparison(results, 'comparison_report.json')

Additional Resources

More in DataFrames and pandas

  1. Mastering QLoRA9 min

    QLoRA (Quantized Low-Rank Adaptation) is an efficient fine-tuning technique for large language models. It combines quantization and low-rank adaptation to...

  2. Mastering NER for Documents with Multiple Entities3 min

    Natural Entity Recognition (NER) becomes more reliable when combined with data enrichment techniques. This approach helps overcome common challenges in...

  3. Mastering DataFrames with .describe()6 min

    The describe() method provides a comprehensive statistical summary of numerical columns in a DataFrame, including count, mean, standard deviation, minimum...

  4. Implementing Multiple Regression from Scratch10 min

    This presentation explores the implementation of Multiple Regression without using Sklearn. We'll dive into the mathematics behind the Ordinary Least Squares...

  5. Fundamentals of Binary Classification9 min

    Binary classification forms the foundation of many machine learning applications, where the goal is to categorize input data into one of two classes. This...

  6. Filling Blank Values in a DataFrame5 min

    To fill all blank (NaN) values in a DataFrame with zero using Python, we can use the fillna() method. This method is part of the pandas library, which...