xbe.at

Machine learning / Dimensionality reduction and anomaly

Understanding and Mitigating Data Drift in ML Production

6 min read, 17 sections

Contents
  1. Data Drift in ML Production
  2. Source Code for Data Drift in ML Production
  3. Results for Data Drift in ML Production
  4. Importance of Data Drift Monitoring
  5. Case 1: Delayed or Rare Target Information
  6. Source Code for Case 1: Delayed or Rare Target Information
  7. Case 2: Debugging Poor Model Performance
  8. Source Code for Case 2: Debugging Poor Model Performance
  9. Results for Case 2: Debugging Poor Model Performance
  10. Case 3: Identifying Errors in Target Data
  11. Source Code for Case 3: Identifying Errors in Target Data
  12. Real-Life Example: Weather Prediction
  13. Real-Life Example: E-commerce Recommendation System
  14. Implementing Data Drift Detection
  15. Source Code for Implementing Data Drift Detection
  16. Results for Implementing Data Drift Detection
  17. Additional Resources

Data Drift in ML Production

Data drift refers to the change in statistical properties of input features over time. It's a critical concept in machine learning, especially for deployed models. Continuous monitoring of data drift in production environments is essential for maintaining model performance and reliability.

Source Code for Data Drift in ML Production

import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import ks_2samp

def detect_data_drift(reference_data, current_data, threshold=0.05):
    _, p_value = ks_2samp(reference_data, current_data)
    return p_value < threshold

# Generate sample data
np.random.seed(42)
reference_data = np.random.normal(0, 1, 1000)
current_data_no_drift = np.random.normal(0, 1, 1000)
current_data_with_drift = np.random.normal(0.5, 1.2, 1000)

# Detect drift
drift_detected_no_drift = detect_data_drift(reference_data, current_data_no_drift)
drift_detected_with_drift = detect_data_drift(reference_data, current_data_with_drift)

print(f"Drift detected (no drift): {drift_detected_no_drift}")
print(f"Drift detected (with drift): {drift_detected_with_drift}")

# Visualize distributions
plt.figure(figsize=(10, 5))
plt.hist(reference_data, bins=30, alpha=0.5, label='Reference')
plt.hist(current_data_with_drift, bins=30, alpha=0.5, label='Current (with drift)')
plt.legend()
plt.title('Data Distribution Comparison')
plt.show()

Results for Data Drift in ML Production

Drift detected (no drift): False
Drift detected (with drift): True

Importance of Data Drift Monitoring

Continuous data drift monitoring in production is crucial for maintaining model performance. It helps identify changes in input data distribution that may affect model accuracy. Early detection of drift allows for timely model updates and prevents degradation of predictive power.

Case 1: Delayed or Rare Target Information

In scenarios where true target values are infrequently available, undetected data drift can lead to prolonged periods of inaccurate predictions. A data drift detection system serves as a proxy for potential poor model performance, alerting you to issues before they significantly impact your system.

Source Code for Case 1: Delayed or Rare Target Information

import numpy as np
import matplotlib.pyplot as plt

def simulate_delayed_target_scenario(days, drift_start, drift_magnitude):
    np.random.seed(42)
    predictions = np.random.normal(0, 1, days)
    true_values = np.random.normal(0, 1, days)
    
    # Introduce drift
    drift = np.linspace(0, drift_magnitude, days - drift_start)
    predictions[drift_start:] += drift
    
    # Simulate delayed target availability
    available_targets = np.full(days, np.nan)
    available_targets[::5] = true_values[::5]  # Every 5th day
    
    return predictions, true_values, available_targets

days = 100
drift_start = 50
drift_magnitude = 2

predictions, true_values, available_targets = simulate_delayed_target_scenario(days, drift_start, drift_magnitude)

plt.figure(figsize=(12, 6))
plt.plot(predictions, label='Predictions')
plt.plot(true_values, label='True Values')
plt.scatter(range(days), available_targets, color='red', label='Available Targets', alpha=0.5)
plt.axvline(x=drift_start, color='green', linestyle='--', label='Drift Start')
plt.legend()
plt.title('Delayed Target Information Scenario')
plt.xlabel('Days')
plt.ylabel('Values')
plt.show()

Case 2: Debugging Poor Model Performance

When model performance deteriorates, data drift analysis can provide valuable insights. It helps identify the root cause of poor performance and guides decisions on retraining strategies. A data drift monitoring system can expedite this process, saving time and resources in debugging efforts.

Source Code for Case 2: Debugging Poor Model Performance

import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import ks_2samp

def analyze_feature_drift(reference_data, current_data, feature_names, threshold=0.05):
    drifted_features = []
    for i, feature in enumerate(feature_names):
        _, p_value = ks_2samp(reference_data[:, i], current_data[:, i])
        if p_value < threshold:
            drifted_features.append(feature)
    return drifted_features

# Generate sample data
np.random.seed(42)
feature_names = ['Feature A', 'Feature B', 'Feature C', 'Feature D']
reference_data = np.random.normal(0, 1, (1000, 4))
current_data = np.random.normal(0, 1, (1000, 4))

# Introduce drift in Feature B and Feature D
current_data[:, 1] += 0.5
current_data[:, 3] *= 1.2

drifted_features = analyze_feature_drift(reference_data, current_data, feature_names)

print("Drifted features:", drifted_features)

# Visualize drifted features
fig, axs = plt.subplots(2, 2, figsize=(12, 10))
for i, ax in enumerate(axs.flatten()):
    ax.hist(reference_data[:, i], bins=30, alpha=0.5, label='Reference')
    ax.hist(current_data[:, i], bins=30, alpha=0.5, label='Current')
    ax.set_title(feature_names[i])
    ax.legend()

plt.tight_layout()
plt.show()

Results for Case 2: Debugging Poor Model Performance

Drifted features: ['Feature B', 'Feature D']

Case 3: Identifying Errors in Target Data

Data drift monitoring can also help identify issues with newly collected target data. If features haven't drifted but the model shows high error rates, it might indicate problems with the target data. This insight allows you to exclude erroneous data from error computations and model retraining.

Source Code for Case 3: Identifying Errors in Target Data

import numpy as np
import matplotlib.pyplot as plt

def simulate_target_data_error(days, error_start, error_magnitude):
    np.random.seed(42)
    features = np.random.normal(0, 1, (days, 3))
    true_targets = np.sum(features, axis=1) + np.random.normal(0, 0.1, days)
    
    # Introduce error in target data
    erroneous_targets = true_targets.copy()
    erroneous_targets[error_start:] += np.random.normal(error_magnitude, 0.5, days - error_start)
    
    return features, true_targets, erroneous_targets

days = 100
error_start = 70
error_magnitude = 2

features, true_targets, erroneous_targets = simulate_target_data_error(days, error_start, error_magnitude)

# Calculate errors
true_errors = np.abs(np.sum(features, axis=1) - true_targets)
erroneous_errors = np.abs(np.sum(features, axis=1) - erroneous_targets)

plt.figure(figsize=(12, 6))
plt.plot(true_errors, label='True Errors')
plt.plot(erroneous_errors, label='Erroneous Errors')
plt.axvline(x=error_start, color='red', linestyle='--', label='Error Introduction')
plt.legend()
plt.title('Impact of Target Data Errors on Model Performance')
plt.xlabel('Days')
plt.ylabel('Absolute Error')
plt.show()

Real-Life Example: Weather Prediction

Consider a weather prediction model deployed in a coastal city. Over time, climate change affects temperature patterns, leading to data drift. The model, trained on historical data, starts making inaccurate predictions. Continuous monitoring of temperature data distribution helps identify this drift, prompting timely model updates to maintain accurate forecasts.

Real-Life Example: E-commerce Recommendation System

An e-commerce platform uses a recommendation system based on user browsing history. During a global pandemic, user behavior changes dramatically, causing data drift in features like product category preferences and browsing time patterns. Data drift detection alerts the team to these changes, allowing them to adapt the recommendation algorithm to the new normal, maintaining relevant product suggestions.

Implementing Data Drift Detection

To implement data drift detection, you can use statistical tests like the Kolmogorov-Smirnov test or methods based on population stability index. Regular comparisons between the reference dataset (used for training) and current production data can reveal significant distribution changes.

Source Code for Implementing Data Drift Detection

import numpy as np
from scipy.stats import ks_2samp

def detect_feature_drift(reference_data, current_data, feature_names, threshold=0.05):
    drift_results = {}
    for i, feature in enumerate(feature_names):
        statistic, p_value = ks_2samp(reference_data[:, i], current_data[:, i])
        drift_detected = p_value < threshold
        drift_results[feature] = {
            'drift_detected': drift_detected,
            'p_value': p_value,
            'statistic': statistic
        }
    return drift_results

# Example usage
np.random.seed(42)
feature_names = ['user_age', 'session_duration', 'pages_visited', 'cart_value']
reference_data = np.random.normal(0, 1, (1000, 4))
current_data = np.random.normal(0, 1, (1000, 4))

# Introduce drift in 'session_duration' and 'cart_value'
current_data[:, 1] += 0.5  # Shift in session duration
current_data[:, 3] *= 1.2  # Scale change in cart value

drift_results = detect_feature_drift(reference_data, current_data, feature_names)

for feature, result in drift_results.items():
    print(f"{feature}: Drift detected: {result['drift_detected']}, p-value: {result['p_value']:.4f}")

Results for Implementing Data Drift Detection

user_age: Drift detected: False, p-value: 0.8295
session_duration: Drift detected: True, p-value: 0.0000
pages_visited: Drift detected: False, p-value: 0.9809
cart_value: Drift detected: True, p-value: 0.0000

Additional Resources

For more information on data drift in machine learning, consider the following resources:

  1. "Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift" by Rabanser et al. (2019) - ArXiv:1810.11953
  2. "A Survey on Concept Drift Adaptation" by Gama et al. (2014) - ACM Computing Surveys, Vol. 46, No. 4, Article 44
  3. "Learning under Concept Drift: A Review" by Lu et al. (2018) - ArXiv:1810.11944

These papers provide in-depth discussions on data drift detection methods and adaptation strategies in machine learning systems.

More in Dimensionality reduction and anomaly

  1. Unsupervised Dimensionality Reduction in Python7 min

    Unsupervised dimensionality reduction is a technique used to reduce the number of features in a dataset while preserving its essential structure. This...

  2. Time Series Anomaly Detection with Spatial-Temporal Normality Learning12 min

    Time series anomaly detection is a crucial task in various domains, including IoT, cybersecurity, and industrial monitoring. Spatial-Temporal Normality...

  3. Why t-SNE Uses t-Distribution Instead of Gaussian9 min

    t-SNE (t-distributed Stochastic Neighbor Embedding) is a popular dimensionality reduction technique used for visualizing high-dimensional data. It's an...

  4. Spatial-Temporal Normality Learning for Time Series Anomaly Detection11 min

    Spatial-Temporal Normality Learning (STEN) is a powerful technique for detecting anomalies in time series data. It combines temporal and spatial dimensions...

  5. Singular Value Decomposition (SVD) in Machine Learning14 min

    Singular Value Decomposition is a fundamental technique in linear algebra with widespread applications in machine learning and AI. It decomposes a matrix...

  6. Simplifying Complex Data with Principal Component Analysis10 min

    Principal Component Analysis relies on fundamental linear algebra concepts to transform data. The core mathematical foundation involves computing the...