xbe.at

Miscellany / Statistics and analysis

Overcoming Challenges of Zero-Inflated Regression Modeling

7 min read, 14 sections

Contents
  1. Understanding Zero-Inflated Data
  2. Simulating Zero-Inflated Regression Problem
  3. Traditional Linear Regression on Zero-Inflated Data
  4. Two-Part Model Architecture
  5. Implementation of Two-Part Model Prediction
  6. Model Comparison Framework
  7. Implementing Advanced Zero-Inflated Regression
  8. Feature Engineering for Zero-Inflated Data
  9. Cross-Validation Strategy for Zero-Inflated Models
  10. Model Evaluation Metrics for Zero-Inflated Data
  11. Real-World Application: Insurance Claims Analysis
  12. Handling Temporal Zero-Inflation
  13. Bayesian Treatment of Zero-Inflation
  14. Additional Resources

Understanding Zero-Inflated Data

Zero-inflated datasets occur when the dependent variable contains more zeros than expected under standard probability distributions. This phenomenon is common in count data across various domains like healthcare claims, ecological surveys, and financial transactions.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

# Generate zero-inflated data
np.random.seed(42)
n_samples = 1000
zeros = np.zeros(600)
non_zeros = np.random.gamma(2, 2, 400)
y = np.concatenate([zeros, non_zeros])
X = np.random.normal(0, 1, n_samples)

plt.hist(y, bins=50)
plt.title('Zero-Inflated Distribution')
plt.xlabel('Values')
plt.ylabel('Frequency')

Simulating Zero-Inflated Regression Problem

A practical demonstration of how traditional regression models struggle with zero-inflated data through a synthetic dataset that mimics real-world scenarios where excessive zeros naturally occur in the target variable.

def generate_zero_inflated_data(n_samples=1000, zero_prob=0.6):
    # Generate features
    X = np.random.normal(0, 1, (n_samples, 2))
    
    # Generate underlying true relationship
    true_values = 2 * X[:, 0] + 1.5 * X[:, 1] + np.random.normal(0, 0.5, n_samples)
    
    # Add zero inflation
    zero_mask = np.random.random(n_samples) < zero_prob
    y = np.where(zero_mask, 0, np.maximum(0, true_values))
    
    return X, y

Traditional Linear Regression on Zero-Inflated Data

Traditional linear regression performs poorly on zero-inflated data as it assumes a continuous distribution of the target variable. This implementation demonstrates the limitation using scikit-learn's LinearRegression model.

from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score

# Generate data
X, y = generate_zero_inflated_data()

# Fit linear regression
model = LinearRegression()
model.fit(X, y)

# Predictions
y_pred = model.predict(X)

print(f"R² Score: {r2_score(y, y_pred):.3f}")
print(f"MSE: {mean_squared_error(y, y_pred):.3f}")

# Visualize predictions vs actual
plt.scatter(y, y_pred, alpha=0.5)
plt.plot([0, max(y)], [0, max(y)], 'r--')
plt.xlabel('Actual Values')
plt.ylabel('Predicted Values')

Two-Part Model Architecture

The two-part model combines a binary classifier to predict zero vs non-zero outcomes with a regression model for predicting non-zero values. This architecture effectively handles the inherent structure of zero-inflated data.

class TwoPartModel:
    def __init__(self, classifier, regressor):
        self.classifier = classifier
        self.regressor = regressor
        
    def fit(self, X, y):
        # Fit classifier
        self.classifier.fit(X, y > 0)
        
        # Fit regressor on non-zero data
        mask = y > 0
        self.regressor.fit(X[mask], y[mask])
        
        return self

Implementation of Two-Part Model Prediction

The prediction phase combines both models' outputs, where the classifier determines if a prediction should be zero, and the regressor provides the actual value for non-zero predictions.

def predict(self, X):
    # Get binary predictions
    zero_pred = self.classifier.predict(X)
    
    # Initialize predictions array
    predictions = np.zeros(len(X))
    
    # Get regression predictions for non-zero cases
    reg_indices = np.where(zero_pred == 1)[0]
    if len(reg_indices) > 0:
        predictions[reg_indices] = self.regressor.predict(X[reg_indices])
    
    return predictions

Model Comparison Framework

This framework enables systematic comparison between traditional regression and the two-part model approach, implementing cross-validation and multiple performance metrics to ensure robust evaluation of both methods.

from sklearn.model_selection import KFold
from sklearn.metrics import mean_absolute_error, mean_squared_error
import numpy as np

def compare_models(X, y, n_splits=5):
    kf = KFold(n_splits=n_splits, shuffle=True, random_state=42)
    metrics = {
        'traditional': {'mse': [], 'mae': [], 'zero_accuracy': []},
        'two_part': {'mse': [], 'mae': [], 'zero_accuracy': []}
    }
    
    for train_idx, test_idx in kf.split(X):
        X_train, X_test = X[train_idx], X[test_idx]
        y_train, y_test = y[train_idx], y[test_idx]
        
        # Train and evaluate both models
        results = evaluate_both_models(X_train, X_test, y_train, y_test)
        
        # Store metrics
        for model_type in metrics:
            for metric in metrics[model_type]:
                metrics[model_type][metric].append(results[model_type][metric])
    
    return metrics

Implementing Advanced Zero-Inflated Regression

The advanced implementation incorporates both logistic regression for zero prediction and a sophisticated regression model using gradient boosting, providing better handling of non-linear relationships in non-zero values.

from sklearn.ensemble import GradientBoostingRegressor
from sklearn.linear_model import LogisticRegression

class AdvancedZeroInflatedRegressor:
    def __init__(self):
        self.zero_classifier = LogisticRegression(random_state=42)
        self.regressor = GradientBoostingRegressor(
            n_estimators=100,
            learning_rate=0.1,
            max_depth=3,
            random_state=42
        )
        
    def fit(self, X, y):
        # Train zero classifier
        self.zero_classifier.fit(X, y == 0)
        
        # Train regressor on non-zero data
        mask = y > 0
        if mask.any():
            self.regressor.fit(X[mask], y[mask])
        return self

Feature Engineering for Zero-Inflated Data

Feature engineering specifically designed for zero-inflated data involves creating interaction terms and derived features that help capture the underlying patterns leading to zero values in the target variable.

def engineer_features_for_zero_inflation(X, polynomial_degree=2):
    """
    Create specialized features for zero-inflated data modeling
    """
    # Original features
    features = X.copy()
    
    # Add polynomial features
    for i in range(X.shape[1]):
        for degree in range(2, polynomial_degree + 1):
            features = np.column_stack((
                features, 
                X[:, i] ** degree
            ))
    
    # Add interaction terms
    for i in range(X.shape[1]):
        for j in range(i+1, X.shape[1]):
            features = np.column_stack((
                features,
                X[:, i] * X[:, j]
            ))
            
    return features

Cross-Validation Strategy for Zero-Inflated Models

A specialized cross-validation strategy that maintains the zero-inflation ratio across folds, ensuring that model evaluation reflects real-world performance accurately.

class StratifiedZeroInflatedKFold:
    def __init__(self, n_splits=5, random_state=None):
        self.n_splits = n_splits
        self.random_state = random_state

    def split(self, X, y):
        np.random.seed(self.random_state)
        zero_indices = np.where(y == 0)[0]
        nonzero_indices = np.where(y > 0)[0]
        
        # Shuffle indices
        np.random.shuffle(zero_indices)
        np.random.shuffle(nonzero_indices)
        
        # Calculate fold sizes
        zero_fold_size = len(zero_indices) // self.n_splits
        nonzero_fold_size = len(nonzero_indices) // self.n_splits
        
        for i in range(self.n_splits):
            test_indices = np.concatenate([
                zero_indices[i*zero_fold_size:(i+1)*zero_fold_size],
                nonzero_indices[i*nonzero_fold_size:(i+1)*nonzero_fold_size]
            ])
            train_indices = np.concatenate([
                zero_indices[np.r_[0:i*zero_fold_size, (i+1)*zero_fold_size:len(zero_indices)]],
                nonzero_indices[np.r_[0:i*nonzero_fold_size, (i+1)*nonzero_fold_size:len(nonzero_indices)]]
            ])
            yield train_indices, test_indices

Model Evaluation Metrics for Zero-Inflated Data

Custom evaluation metrics designed specifically for zero-inflated data, incorporating both classification accuracy for zero predictions and regression performance for non-zero values in a single comprehensive framework.

def evaluate_zero_inflated_model(y_true, y_pred):
    # Zero prediction accuracy
    zero_accuracy = np.mean((y_true == 0) == (y_pred == 0))
    
    # Metrics for non-zero values
    mask_true = y_true > 0
    mask_pred = y_pred > 0
    
    # Calculate specialized metrics
    metrics = {
        'zero_accuracy': zero_accuracy,
        'rmse_nonzero': np.sqrt(mean_squared_error(
            y_true[mask_true & mask_pred],
            y_pred[mask_true & mask_pred]
        )),
        'false_zero_rate': np.mean(mask_true & ~mask_pred),
        'false_nonzero_rate': np.mean(~mask_true & mask_pred)
    }
    
    return metrics

Real-World Application: Insurance Claims Analysis

Analysis of insurance claims data where many policyholders have zero claims, demonstrating the practical application of zero-inflated regression in the insurance industry.

def analyze_insurance_claims(claims_data):
    # Prepare features and target
    features = ['age', 'bmi', 'smoker', 'children', 'region']
    X = claims_data[features].values
    y = claims_data['claim_amount'].values
    
    # Initialize models
    traditional_model = LinearRegression()
    zero_inflated_model = AdvancedZeroInflatedRegressor()
    
    # Split data
    X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size=0.2, random_state=42
    )
    
    # Train and evaluate both models
    results = compare_models(X_train, X_test, y_train, y_test)
    
    return results

Handling Temporal Zero-Inflation

Implementation of a time-aware zero-inflated model that accounts for seasonal patterns and temporal dependencies in zero occurrences, crucial for time series data analysis.

class TemporalZeroInflatedModel:
    def __init__(self, lookback_window=3):
        self.lookback_window = lookback_window
        self.classifier = LogisticRegression()
        self.regressor = GradientBoostingRegressor()
        
    def create_temporal_features(self, X, dates):
        # Extract temporal features
        temporal_features = np.column_stack([
            np.sin(2 * np.pi * dates.dayofyear / 365),
            np.cos(2 * np.pi * dates.dayofyear / 365),
            dates.dayofweek,
            dates.month
        ])
        
        # Combine with original features
        return np.hstack([X, temporal_features])
    
    def fit(self, X, y, dates):
        X_temporal = self.create_temporal_features(X, dates)
        self.classifier.fit(X_temporal, y == 0)
        mask = y > 0
        if mask.any():
            self.regressor.fit(X_temporal[mask], y[mask])
        return self

Bayesian Treatment of Zero-Inflation

A Bayesian approach to zero-inflated regression using PyMC3, providing uncertainty estimates and posterior distributions for both the zero-generating process and the continuous component.

import pymc3 as pm
import theano.tensor as tt

def bayesian_zero_inflated_regression(X, y, samples=2000):
    with pm.Model() as model:
        # Priors
        alpha = pm.Normal('alpha', mu=0, sd=10)
        beta = pm.Normal('beta', mu=0, sd=10, shape=X.shape[1])
        psi = pm.Beta('psi', alpha=1, beta=1)  # Zero-inflation parameter
        
        # Expected value
        mu = alpha + pm.math.dot(X, beta)
        
        # Zero-inflated likelihood
        likelihood = pm.ZeroInflatedPoisson(
            'likelihood',
            psi=psi,
            mu=tt.exp(mu),
            observed=y
        )
        
        # Sample from posterior
        trace = pm.sample(samples, tune=1000, return_inferencedata=False)
        
    return trace

Additional Resources

More in Statistics and analysis

  1. Oversampling Techniques for Imbalanced Learning13 min

    Data imbalance occurs when class distributions in a dataset are significantly skewed. This fundamental challenge in machine learning can severely impact...

  2. Multi-Class Classification in Business Problems14 min

    Multi-class classification is a common challenge in business problems, often overlooked in favor of binary classification. This type of problem involves...

  3. Performance Metrics for Regression Models10 min

    Performance metrics in regression are essential tools for evaluating the accuracy and effectiveness of our predictive models. These metrics help us...

  4. Modular ML Pipeline Architecture in 3 Phases14 min

    Feature engineering serves as the foundation of any robust machine learning system. This phase handles data preparation, cleaning, and transformation while...

  5. Probabilistic Multiclass Classification Models10 min

    Accuracy, while intuitive, can be misleading in multiclass settings. It fails to capture the nuances of model performance, especially when dealing with...

  6. Modeling Decision-Making with Linear Ballistic Accumulator9 min

    The Linear Ballistic Accumulator (LBA) represents a mathematical framework for modeling decision-making processes where evidence accumulates linearly and...