xbe.at

Machine learning / Feature engineering and preprocessing

Intuitive Feature Selection with Probe Method

7 min read, 15 sections

Contents
  1. Introduction to the Probe Method
  2. Basic Implementation of Probe Method
  3. Data Preprocessing for Probe Method
  4. Feature Importance Visualization
  5. Convergence Monitoring
  6. Real-world Example - Credit Card Fraud Detection
  7. Results for Credit Card Fraud Detection
  8. Advanced Probe Configuration
  9. Statistical Validation of Selected Features
  10. Real-world Example - Gene Expression Analysis
  11. Cross-validation Strategy for Probe Method
  12. Probe Method with Different Model Backends
  13. Performance Metrics and Evaluation
  14. Handling Imbalanced Datasets
  15. Additional Resources

Introduction to the Probe Method

The Probe Method is a novel feature selection technique that leverages random noise injection to evaluate feature importance. By comparing genuine features against artificially introduced random features, it provides an intuitive way to identify and eliminate irrelevant or redundant predictors from the dataset.

import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.preprocessing import StandardScaler

def add_probe_feature(X):
    """Add random probe feature to dataset"""
    n_samples = X.shape[0]
    probe = np.random.normal(0, 1, size=(n_samples, 1))
    return np.hstack([X, probe])

Basic Implementation of Probe Method

The core implementation involves iteratively adding probe features and measuring their importance against original features. This process continues until no original features rank below the probe, ensuring only truly informative features remain in the final dataset.

def probe_feature_selection(X, y, n_iterations=10):
    selected_features = list(range(X.shape[1]))
    
    for _ in range(n_iterations):
        # Add probe feature
        X_probe = add_probe_feature(X[:, selected_features])
        
        # Train model
        rf = RandomForestClassifier(random_state=42)
        rf.fit(X_probe, y)
        
        # Get feature importance
        importance = rf.feature_importances_
        probe_importance = importance[-1]
        
        # Select features above probe importance
        selected_features = [i for i in range(len(importance)-1) 
                           if importance[i] > probe_importance]
        
        if len(selected_features) == 0:
            break
            
    return selected_features

Data Preprocessing for Probe Method

Before applying the Probe Method, data must be properly preprocessed to ensure fair comparison between features. This includes handling missing values, scaling numerical features, and encoding categorical variables to create a clean, normalized dataset.

def prepare_data_for_probe(X, categorical_cols=None):
    """Prepare dataset for probe method"""
    X_prep = X.copy()
    
    # Handle missing values
    X_prep = pd.DataFrame(X_prep).fillna(X_prep.mean())
    
    # Scale numerical features
    scaler = StandardScaler()
    X_prep = scaler.fit_transform(X_prep)
    
    return X_prep

Feature Importance Visualization

Visualizing feature importance scores alongside the probe feature helps in understanding the selection process and identifying the cutoff threshold for feature elimination. This provides insights into the relative significance of each feature.

import matplotlib.pyplot as plt

def plot_feature_importance(importance_scores, feature_names, probe_idx):
    plt.figure(figsize=(10, 6))
    importance_df = pd.DataFrame({
        'Feature': feature_names,
        'Importance': importance_scores
    })
    importance_df = importance_df.sort_values('Importance', ascending=True)
    
    colors = ['red' if i == probe_idx else 'blue' 
              for i in range(len(feature_names))]
    
    plt.barh(range(len(importance_df)), importance_df['Importance'], 
             color=colors)
    plt.yticks(range(len(importance_df)), importance_df['Feature'])
    plt.xlabel('Feature Importance')
    plt.title('Feature Importance with Probe')
    plt.tight_layout()
    return plt

Convergence Monitoring

A critical aspect of the Probe Method is monitoring convergence to determine when to stop the iterative process. This implementation tracks the number of selected features and their importance scores across iterations.

def monitor_convergence(feature_counts, importance_history):
    plt.figure(figsize=(12, 4))
    
    plt.subplot(1, 2, 1)
    plt.plot(feature_counts)
    plt.title('Number of Selected Features')
    plt.xlabel('Iteration')
    plt.ylabel('Features Count')
    
    plt.subplot(1, 2, 2)
    plt.plot(importance_history)
    plt.title('Average Feature Importance')
    plt.xlabel('Iteration')
    plt.ylabel('Importance Score')
    
    plt.tight_layout()
    return plt

Real-world Example - Credit Card Fraud Detection

Implementing the Probe Method on a credit card fraud detection dataset demonstrates its practical application. This example shows how the method can identify relevant features for detecting fraudulent transactions.

# Load and prepare credit card fraud dataset
from sklearn.datasets import make_classification

# Generate synthetic fraud detection dataset
X, y = make_classification(n_samples=10000, n_features=20, 
                          n_informative=10, n_redundant=5,
                          random_state=42)

# Prepare data
X_prep = prepare_data_for_probe(X)

# Apply probe method
selected_features = probe_feature_selection(X_prep, y)
print(f"Selected features: {selected_features}")

Results for Credit Card Fraud Detection

The probe method significantly reduced the feature set while maintaining model performance. The visualization shows the importance scores of selected features compared to the probe feature, demonstrating the effectiveness of the selection process.

# Train models with original and selected features
def compare_performance(X, y, selected_features):
    # Original model
    rf_original = RandomForestClassifier(random_state=42)
    original_score = cross_val_score(rf_original, X, y, cv=5).mean()
    
    # Selected features model
    rf_selected = RandomForestClassifier(random_state=42)
    selected_score = cross_val_score(
        rf_selected, X[:, selected_features], y, cv=5
    ).mean()
    
    print(f"Original features score: {original_score:.4f}")
    print(f"Selected features score: {selected_score:.4f}")
    print(f"Feature reduction: {100*(1-len(selected_features)/X.shape[1]):.1f}%")

Advanced Probe Configuration

The probe method can be enhanced by implementing different types of probe features and statistical significance tests to improve feature selection reliability and robustness against various data distributions.

def advanced_probe_selection(X, y, probe_type='gaussian', alpha=0.05):
    def generate_probe(n_samples, method):
        if method == 'gaussian':
            return np.random.normal(0, 1, size=(n_samples, 1))
        elif method == 'uniform':
            return np.random.uniform(-1, 1, size=(n_samples, 1))
        elif method == 'bootstrap':
            idx = np.random.choice(n_samples, n_samples)
            return X[idx, np.random.randint(X.shape[1])].reshape(-1, 1)
    
    n_samples = X.shape[0]
    n_probes = 10
    probe_scores = []
    
    # Generate multiple probes
    for _ in range(n_probes):
        probe = generate_probe(n_samples, probe_type)
        X_probe = np.hstack([X, probe])
        
        rf = RandomForestClassifier(random_state=42)
        rf.fit(X_probe, y)
        probe_scores.append(rf.feature_importances_[-1])
    
    # Calculate significance threshold
    threshold = np.percentile(probe_scores, (1-alpha)*100)
    
    return threshold

Statistical Validation of Selected Features

Implementing statistical validation ensures the robustness of selected features through multiple iterations of the probe method, providing confidence intervals for feature importance scores.

def validate_feature_selection(X, y, n_bootstrap=100):
    n_features = X.shape[1]
    feature_counts = np.zeros(n_features)
    
    for i in range(n_bootstrap):
        # Bootstrap sampling
        idx = np.random.choice(len(X), len(X), replace=True)
        X_boot, y_boot = X[idx], y[idx]
        
        # Run probe selection
        selected = probe_feature_selection(X_boot, y_boot)
        feature_counts[selected] += 1
    
    # Calculate selection frequency
    selection_freq = feature_counts / n_bootstrap
    
    # Plot selection frequency
    plt.figure(figsize=(10, 5))
    plt.bar(range(n_features), selection_freq)
    plt.xlabel('Feature Index')
    plt.ylabel('Selection Frequency')
    plt.title('Feature Selection Stability')
    
    return selection_freq

Real-world Example - Gene Expression Analysis

Applying the probe method to high-dimensional gene expression data demonstrates its effectiveness in identifying relevant genes for disease classification while handling the curse of dimensionality.

# Generate synthetic gene expression dataset
X_genes, y_genes = make_classification(
    n_samples=200, n_features=1000,
    n_informative=50, n_redundant=50,
    random_state=42
)

# Scale features
X_genes_scaled = StandardScaler().fit_transform(X_genes)

# Apply probe method with stability selection
selection_frequency = validate_feature_selection(X_genes_scaled, y_genes)

# Select stable features
stable_features = np.where(selection_frequency > 0.8)[0]
print(f"Number of stable features selected: {len(stable_features)}")

Cross-validation Strategy for Probe Method

Implementing a cross-validation strategy ensures that the selected features generalize well across different subsets of the data, preventing overfitting in the feature selection process.

from sklearn.model_selection import StratifiedKFold

def cross_validated_probe_selection(X, y, n_folds=5):
    skf = StratifiedKFold(n_splits=n_folds, shuffle=True, random_state=42)
    feature_stability = np.zeros((n_folds, X.shape[1]))
    
    for fold, (train_idx, val_idx) in enumerate(skf.split(X, y)):
        X_train, y_train = X[train_idx], y[train_idx]
        
        # Run probe selection on training fold
        selected_features = probe_feature_selection(X_train, y_train)
        feature_stability[fold, selected_features] = 1
    
    # Calculate selection consistency
    selection_consistency = feature_stability.mean(axis=0)
    
    return selection_consistency, feature_stability

Probe Method with Different Model Backends

The probe method can utilize various machine learning models as backends for feature importance calculation. This implementation demonstrates how to use different models while maintaining the core probe methodology.

from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from xgboost import XGBClassifier

def probe_with_multiple_models(X, y, models=None):
    if models is None:
        models = {
            'rf': RandomForestClassifier(random_state=42),
            'logistic': LogisticRegression(random_state=42),
            'xgb': XGBClassifier(random_state=42)
        }
    
    results = {}
    for name, model in models.items():
        selected = probe_feature_selection(X, y, model=model)
        results[name] = {
            'selected_features': selected,
            'n_selected': len(selected)
        }
    
    return results

Performance Metrics and Evaluation

Comprehensive evaluation of the probe method requires tracking multiple performance metrics and comparing them against traditional feature selection methods to validate its effectiveness.

from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score

def evaluate_probe_selection(X, y, selected_features):
    # Split data
    X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size=0.2, random_state=42
    )
    
    # Train model with selected features
    rf = RandomForestClassifier(random_state=42)
    rf.fit(X_train[:, selected_features], y_train)
    y_pred = rf.predict(X_test[:, selected_features])
    
    # Calculate metrics
    metrics = {
        'accuracy': accuracy_score(y_test, y_pred),
        'precision': precision_score(y_test, y_pred, average='weighted'),
        'recall': recall_score(y_test, y_pred, average='weighted'),
        'f1': f1_score(y_test, y_pred, average='weighted')
    }
    
    return metrics

Handling Imbalanced Datasets

When applying the probe method to imbalanced datasets, special considerations are needed to ensure fair feature selection across all classes.

from imblearn.over_sampling import SMOTE

def probe_selection_imbalanced(X, y, sampling_strategy='auto'):
    # Apply SMOTE for balanced sampling
    smote = SMOTE(sampling_strategy=sampling_strategy, random_state=42)
    X_balanced, y_balanced = smote.fit_resample(X, y)
    
    # Run probe selection on balanced data
    selected_features = probe_feature_selection(X_balanced, y_balanced)
    
    # Validate on original distribution
    original_metrics = evaluate_probe_selection(X, y, selected_features)
    balanced_metrics = evaluate_probe_selection(
        X_balanced, y_balanced, selected_features
    )
    
    return {
        'selected_features': selected_features,
        'original_metrics': original_metrics,
        'balanced_metrics': balanced_metrics
    }

Additional Resources

More in Feature engineering and preprocessing

  1. Log Transform in Machine Learning with Python8 min

    Introduction to Log Transform in Machine Learning

  2. Importance of Feature Scaling in Machine Learning10 min

    Feature scaling is a crucial preprocessing step in machine learning that transforms the attributes of a dataset to a common scale. This process ensures that...

  3. Master Feature Engineering for Machine Learning21 min

    Missing data handling requires sophisticated approaches beyond simple mean or median imputation. Advanced techniques like Multiple Imputation by Chained...

  4. Importance of Data Quality in Machine Learning22 min

    Data quality assessment is a crucial first step in machine learning projects. We'll create a comprehensive data quality analyzer that checks for missing...

  5. Mastering Data Manipulation with Pandas for Machine Learning12 min

    Pandas provides efficient methods for loading data from various sources and performing basic operations. Understanding these fundamentals is crucial for any...

  6. Imbalanced Stratification Techniques for Machine Learning using Python6 min

    Imbalanced classification is a common problem in machine learning where one class significantly outnumbers the other(s). This can lead to biased models that...