xbe.at

Miscellany / Statistics and analysis

Feature Scaling Techniques for Outlier-Robust Data

6 min read, 14 sections

Contents
  1. Feature Scaling for Outlier-Robust Data Processing
  2. Understanding Outliers
  3. Robust Scaling
  4. Quantile Transformation
  5. Winsorization
  6. Log Transformation
  7. Box-Cox Transformation
  8. Yeo-Johnson Transformation
  9. Modified Z-score
  10. Comparison of Scaling Techniques
  11. Real-life Example: Weather Data
  12. Real-life Example: Sensor Readings
  13. Choosing the Right Scaling Technique
  14. Additional Resources

Feature Scaling for Outlier-Robust Data Processing

Feature scaling is crucial when dealing with datasets containing outliers. This presentation explores various techniques to scale features effectively, focusing on methods that are resistant to extreme values.

import numpy as np
import matplotlib.pyplot as plt

# Generate sample data with outliers
np.random.seed(42)
data = np.random.normal(0, 1, 1000)
outliers = np.random.uniform(10, 20, 50)
data = np.concatenate([data, outliers])

plt.figure(figsize=(10, 5))
plt.boxplot(data)
plt.title("Sample Data with Outliers")
plt.show()

Understanding Outliers

Outliers are data points that significantly differ from other observations. They can have a substantial impact on traditional scaling methods, leading to skewed results and reduced model performance.

# Visualize the impact of outliers on mean and standard deviation
mean = np.mean(data)
std = np.std(data)

plt.figure(figsize=(10, 5))
plt.hist(data, bins=50, density=True, alpha=0.7)
plt.axvline(mean, color='r', linestyle='dashed', linewidth=2, label=f'Mean: {mean:.2f}')
plt.axvline(mean + std, color='g', linestyle='dashed', linewidth=2, label=f'Std Dev: {std:.2f}')
plt.axvline(mean - std, color='g', linestyle='dashed', linewidth=2)
plt.legend()
plt.title("Distribution of Data with Outliers")
plt.show()

Robust Scaling

Robust Scaling uses the median and interquartile range (IQR) instead of mean and standard deviation. This method is less sensitive to outliers.

from sklearn.preprocessing import RobustScaler

robust_scaler = RobustScaler()
data_robust = robust_scaler.fit_transform(data.reshape(-1, 1))

plt.figure(figsize=(10, 5))
plt.boxplot([data, data_robust.flatten()])
plt.xticks([1, 2], ['Original', 'Robust Scaled'])
plt.title("Comparison: Original vs Robust Scaled Data")
plt.show()

Quantile Transformation

Quantile Transformation maps the original distribution to a uniform or normal distribution. It's effective for handling outliers and non-Gaussian distributions.

from sklearn.preprocessing import QuantileTransformer

qt = QuantileTransformer(output_distribution='normal')
data_qt = qt.fit_transform(data.reshape(-1, 1))

plt.figure(figsize=(10, 5))
plt.hist(data_qt, bins=50, density=True, alpha=0.7)
plt.title("Data after Quantile Transformation")
plt.show()

Winsorization

Winsorization caps extreme values at a specified percentile. This technique preserves the data's structure while mitigating the impact of outliers.

def winsorize(data, limits):
    lower = np.percentile(data, limits[0])
    upper = np.percentile(data, limits[1])
    return np.clip(data, lower, upper)

data_winsorized = winsorize(data, (5, 95))

plt.figure(figsize=(10, 5))
plt.boxplot([data, data_winsorized])
plt.xticks([1, 2], ['Original', 'Winsorized'])
plt.title("Comparison: Original vs Winsorized Data")
plt.show()

Log Transformation

Log transformation is useful for handling right-skewed data and reducing the impact of extreme values. It's particularly effective for positive-valued features with a wide range.

# Ensure all values are positive by adding a small constant
data_positive = data - np.min(data) + 1e-5
data_log = np.log(data_positive)

plt.figure(figsize=(10, 5))
plt.hist(data_log, bins=50, density=True, alpha=0.7)
plt.title("Log-transformed Data")
plt.show()

Box-Cox Transformation

Box-Cox transformation is a family of power transformations that includes log transformation as a special case. It's useful for stabilizing variance and making the data more normal-like.

from scipy.stats import boxcox

data_boxcox, lambda_param = boxcox(data_positive)

plt.figure(figsize=(10, 5))
plt.hist(data_boxcox, bins=50, density=True, alpha=0.7)
plt.title(f"Box-Cox Transformed Data (lambda = {lambda_param:.2f})")
plt.show()

Yeo-Johnson Transformation

Yeo-Johnson transformation is similar to Box-Cox but can handle negative values. It's particularly useful when dealing with features that can take both positive and negative values.

from sklearn.preprocessing import PowerTransformer

pt = PowerTransformer(method='yeo-johnson')
data_yj = pt.fit_transform(data.reshape(-1, 1))

plt.figure(figsize=(10, 5))
plt.hist(data_yj, bins=50, density=True, alpha=0.7)
plt.title("Yeo-Johnson Transformed Data")
plt.show()

Modified Z-score

The modified Z-score uses median absolute deviation (MAD) instead of standard deviation, making it more robust to outliers than the standard Z-score.

def modified_zscore(data):
    median = np.median(data)
    mad = np.median(np.abs(data - median))
    modified_z = 0.6745 * (data - median) / mad
    return modified_z

data_mz = modified_zscore(data)

plt.figure(figsize=(10, 5))
plt.boxplot([data, data_mz])
plt.xticks([1, 2], ['Original', 'Modified Z-score'])
plt.title("Comparison: Original vs Modified Z-score")
plt.show()

Comparison of Scaling Techniques

Let's compare the effectiveness of different scaling techniques on our sample data with outliers.

from sklearn.preprocessing import StandardScaler, MinMaxScaler

scalers = {
    'Standard': StandardScaler(),
    'MinMax': MinMaxScaler(),
    'Robust': RobustScaler(),
    'Quantile': QuantileTransformer(output_distribution='normal')
}

plt.figure(figsize=(15, 10))
for i, (name, scaler) in enumerate(scalers.items(), 1):
    scaled_data = scaler.fit_transform(data.reshape(-1, 1)).flatten()
    plt.subplot(2, 2, i)
    plt.boxplot(scaled_data)
    plt.title(f"{name} Scaling")

plt.tight_layout()
plt.show()

Real-life Example: Weather Data

Consider a dataset of daily temperature readings across different cities. Extreme weather events can introduce outliers that skew the data.

# Simulating weather data with outliers
np.random.seed(42)
temperatures = np.random.normal(20, 5, 1000)  # Normal temperatures
extreme_temps = np.random.uniform(-10, 45, 50)  # Extreme temperatures
temperatures = np.concatenate([temperatures, extreme_temps])

# Apply robust scaling
robust_scaler = RobustScaler()
temperatures_scaled = robust_scaler.fit_transform(temperatures.reshape(-1, 1))

plt.figure(figsize=(12, 5))
plt.subplot(1, 2, 1)
plt.hist(temperatures, bins=50, alpha=0.7)
plt.title("Original Temperature Data")
plt.subplot(1, 2, 2)
plt.hist(temperatures_scaled, bins=50, alpha=0.7)
plt.title("Robust Scaled Temperature Data")
plt.tight_layout()
plt.show()

Real-life Example: Sensor Readings

In industrial settings, sensor readings can be affected by various factors, leading to outliers. Let's simulate pH sensor readings from a chemical process.

# Simulating pH sensor readings with outliers
np.random.seed(42)
ph_readings = np.random.normal(7, 0.5, 1000)  # Normal pH readings
faulty_readings = np.random.uniform(0, 14, 50)  # Faulty readings
ph_readings = np.concatenate([ph_readings, faulty_readings])

# Apply Winsorization
ph_winsorized = winsorize(ph_readings, (5, 95))

plt.figure(figsize=(12, 5))
plt.subplot(1, 2, 1)
plt.hist(ph_readings, bins=50, alpha=0.7)
plt.title("Original pH Readings")
plt.subplot(1, 2, 2)
plt.hist(ph_winsorized, bins=50, alpha=0.7)
plt.title("Winsorized pH Readings")
plt.tight_layout()
plt.show()

Choosing the Right Scaling Technique

When selecting a scaling technique for data with outliers, consider:

  1. The nature of your data and outliers
  2. The requirements of your machine learning algorithm
  3. The interpretability of the scaled features
  4. The computational efficiency of the scaling method

Experiment with different techniques and evaluate their impact on your model's performance to find the most suitable approach for your specific use case.

# Function to evaluate different scaling techniques
def evaluate_scaling(data, scalers):
    results = {}
    for name, scaler in scalers.items():
        scaled_data = scaler.fit_transform(data.reshape(-1, 1)).flatten()
        results[name] = {
            'mean': np.mean(scaled_data),
            'std': np.std(scaled_data),
            'min': np.min(scaled_data),
            'max': np.max(scaled_data)
        }
    return results

scalers = {
    'Standard': StandardScaler(),
    'MinMax': MinMaxScaler(),
    'Robust': RobustScaler(),
    'Quantile': QuantileTransformer(output_distribution='normal')
}

results = evaluate_scaling(data, scalers)
for name, stats in results.items():
    print(f"{name} Scaling:")
    for stat, value in stats.items():
        print(f"  {stat}: {value:.2f}")
    print()

Additional Resources

For further exploration of feature scaling techniques and handling outliers, consider the following resources:

  1. Robust Feature Scaling in Machine Learning (arXiv:2107.13626) URL: https://arxiv.org/abs/2107.13626
  2. A Comparative Study of Outlier Detection and Treatment Methods (arXiv:2209.11257) URL: https://arxiv.org/abs/2209.11257
  3. Scikit-learn Documentation on Preprocessing and Normalization https://scikit-learn.org/stable/modules/preprocessing.html

These resources provide in-depth discussions and advanced techniques for dealing with outliers and feature scaling in various machine learning contexts.

More in Statistics and analysis

  1. Limitations of Binary Classifiers for Face Unlock7 min

    A binary classifier is inadequate for a face unlock application due to several inherent limitations. This approach oversimplifies the complex task of facial...

  2. Evaluation Metrics Storytelling for Your Model8 min

    Evaluation metrics are crucial tools in machine learning that help us assess model performance. They provide insights into how well our models are performing...

  3. Limitations of Point Estimates in Regression Modeling13 min

    Point estimates from regression models often fail to capture the full complexity of real-world data distributions. While they provide a single value...

  4. Estimating Causal Impact of Late Deliveries on Customer Satisfaction8 min

    Propensity Score Matching (PSM) estimates treatment effects by creating matched pairs of treated and control subjects with similar characteristics. The...

  5. Limitations of Silhouette Score for Arbitrary-Shaped Clusters6 min

    The Silhouette score is a widely used metric that measures cluster cohesion and separation, ranging from -1 to 1. While effective for evaluating...

  6. Detecting Disruptive Outliers in Data7 min

    Outliers are data points that significantly deviate from the general pattern of a dataset. They can profoundly impact statistical analyses and machine...