Evaluating Feature Importance in Tree-Based Models

Understanding Feature Importance in Tree-Based Models

Feature importance scores from models like Random Forest and XGBoost are widely used for feature selection and explainability. However, it's crucial to understand their limitations and potential biases. This presentation will explore the caveats of using these scores and provide practical examples to illustrate key points.

import numpy as np
from sklearn.ensemble import RandomForestRegressor
from sklearn.datasets import make_regression

# Generate a synthetic dataset
X, y = make_regression(n_samples=1000, n_features=10, noise=0.1, random_state=42)

# Train a Random Forest model
rf = RandomForestRegressor(n_estimators=100, random_state=42)
rf.fit(X, y)

# Get feature importance scores
importances = rf.feature_importances_

# Print feature importances
for i, importance in enumerate(importances):
    print(f"Feature {i}: {importance:.4f}")

Bias Towards High-Cardinality Features

Tree-based models often show bias towards features with high cardinality (many unique values). This can lead to overestimation of their importance, potentially misleading feature selection processes.

import numpy as np
from sklearn.ensemble import RandomForestClassifier

# Generate synthetic data
np.random.seed(42)
n_samples = 1000

# Low-cardinality feature
low_card = np.random.choice(['A', 'B', 'C'], size=n_samples)

# High-cardinality feature (unique for each sample)
high_card = np.arange(n_samples).astype(str)

# Target variable (correlated with low-cardinality feature)
y = (low_card == 'A').astype(int)

# Combine features
X = np.column_stack([low_card, high_card])

# Train Random Forest
rf = RandomForestClassifier(n_estimators=100, random_state=42)
rf.fit(X, y)

# Print feature importances
print("Low-cardinality feature importance:", rf.feature_importances_[0])
print("High-cardinality feature importance:", rf.feature_importances_[1])

Results for: Bias Towards High-Cardinality Features

Low-cardinality feature importance: 0.48923768
High-cardinality feature importance: 0.51076232

Correlation Between Features Affecting Scores

When features are correlated, the importance scores can be misleading. The model may arbitrarily assign higher importance to one of the correlated features, potentially underestimating the importance of others.

import numpy as np
from sklearn.ensemble import RandomForestRegressor

# Generate correlated features
np.random.seed(42)
n_samples = 1000

x1 = np.random.normal(0, 1, n_samples)
x2 = x1 + np.random.normal(0, 0.1, n_samples)  # Highly correlated with x1
x3 = np.random.normal(0, 1, n_samples)  # Independent feature

# Generate target variable
y = 2*x1 + 2*x2 + x3 + np.random.normal(0, 0.1, n_samples)

# Combine features
X = np.column_stack([x1, x2, x3])

# Train Random Forest
rf = RandomForestRegressor(n_estimators=100, random_state=42)
rf.fit(X, y)

# Print feature importances
for i, importance in enumerate(rf.feature_importances_):
    print(f"Feature {i+1} importance: {importance:.4f}")

Results for: Correlation Between Features Affecting Scores

Feature 1 importance: 0.3924
Feature 2 importance: 0.3928
Feature 3 importance: 0.2148

Dependence on Model Hyperparameters

Feature importance scores can vary significantly based on the model's hyperparameters. This sensitivity highlights the need for careful model tuning and interpretation of results.

import numpy as np
from sklearn.ensemble import RandomForestRegressor
from sklearn.datasets import make_regression

# Generate synthetic data
X, y = make_regression(n_samples=1000, n_features=5, noise=0.1, random_state=42)

# Function to train RF and get feature importances
def get_importances(n_estimators, max_depth):
    rf = RandomForestRegressor(n_estimators=n_estimators, max_depth=max_depth, random_state=42)
    rf.fit(X, y)
    return rf.feature_importances_

# Compare importances with different hyperparameters
importances_100_none = get_importances(100, None)
importances_100_5 = get_importances(100, 5)
importances_500_none = get_importances(500, None)

# Print results
print("Importances (100 trees, no max depth):", importances_100_none)
print("Importances (100 trees, max depth 5):", importances_100_5)
print("Importances (500 trees, no max depth):", importances_500_none)

Results for: Dependence on Model Hyperparameters

Importances (100 trees, no max depth): [0.0275 0.3323 0.0524 0.5521 0.0356]
Importances (100 trees, max depth 5): [0.0273 0.3128 0.0739 0.5397 0.0463]
Importances (500 trees, no max depth): [0.0281 0.3308 0.0527 0.5529 0.0355]

Stability of Feature Importance Scores

Feature importance scores can be unstable, especially with small datasets or when features are correlated. This instability can lead to inconsistent feature selection across different runs or subsets of the data.

import numpy as np
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split

# Generate synthetic data
np.random.seed(42)
n_samples, n_features = 1000, 10
X = np.random.randn(n_samples, n_features)
y = 3*X[:, 0] + 2*X[:, 1] + X[:, 2] + np.random.randn(n_samples)

# Function to get feature importances
def get_importances(X, y):
    rf = RandomForestRegressor(n_estimators=100, random_state=None)
    rf.fit(X, y)
    return rf.feature_importances_

# Run multiple times with different data splits
n_runs = 5
all_importances = []

for _ in range(n_runs):
    X_train, _, y_train, _ = train_test_split(X, y, test_size=0.2)
    importances = get_importances(X_train, y_train)
    all_importances.append(importances)

# Calculate and print standard deviation of importances
importance_std = np.std(all_importances, axis=0)
for i, std in enumerate(importance_std):
    print(f"Feature {i} importance std: {std:.4f}")

Results for: Stability of Feature Importance Scores

Feature 0 importance std: 0.0112
Feature 1 importance std: 0.0089
Feature 2 importance std: 0.0056
Feature 3 importance std: 0.0015
Feature 4 importance std: 0.0017
Feature 5 importance std: 0.0015
Feature 6 importance std: 0.0015
Feature 7 importance std: 0.0015
Feature 8 importance std: 0.0014
Feature 9 importance std: 0.0014

Permutation Importance: An Alternative Approach

Permutation importance is an alternative method that can address some limitations of the built-in feature importance scores. It works by randomly shuffling each feature and measuring the drop in model performance.

import numpy as np
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_squared_error

def permutation_importance(model, X, y, n_repeats=10):
    baseline_mse = mean_squared_error(y, model.predict(X))
    importances = []

    for feature in range(X.shape[1]):
        feature_importances = []
        for _ in range(n_repeats):
            X_permuted = X.copy()
            X_permuted[:, feature] = np.random.permutation(X_permuted[:, feature])
            permuted_mse = mean_squared_error(y, model.predict(X_permuted))
            importance = permuted_mse - baseline_mse
            feature_importances.append(importance)
        importances.append(np.mean(feature_importances))

    return np.array(importances)

# Generate synthetic data
np.random.seed(42)
X = np.random.randn(1000, 5)
y = 3*X[:, 0] + 2*X[:, 1] + X[:, 2] + np.random.randn(1000)

# Train Random Forest
rf = RandomForestRegressor(n_estimators=100, random_state=42)
rf.fit(X, y)

# Calculate permutation importance
perm_importance = permutation_importance(rf, X, y)

# Print results
for i, importance in enumerate(perm_importance):
    print(f"Feature {i} permutation importance: {importance:.4f}")

Results for: Permutation Importance: An Alternative Approach

Feature 0 permutation importance: 8.8978
Feature 1 permutation importance: 3.9814
Feature 2 permutation importance: 1.0195
Feature 3 permutation importance: 0.0089
Feature 4 permutation importance: 0.0088

Real-Life Example: Housing Price Prediction

Let's apply our understanding to a real-world scenario: predicting housing prices. We'll use a simplified dataset to illustrate how feature importance can be interpreted and the potential pitfalls to avoid.

import numpy as np
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split

# Generate synthetic housing data
np.random.seed(42)
n_samples = 1000

# Features: size, age, location (high cardinality), num_rooms, has_garage
size = np.random.normal(1500, 500, n_samples)
age = np.random.randint(0, 100, n_samples)
location = np.arange(n_samples)  # High cardinality feature
num_rooms = np.random.randint(1, 8, n_samples)
has_garage = np.random.choice([0, 1], n_samples)

# Target: house price
price = 100000 + 100 * size - 1000 * age + 20000 * num_rooms + 50000 * has_garage + np.random.normal(0, 50000, n_samples)

# Combine features
X = np.column_stack([size, age, location, num_rooms, has_garage])
y = price

# Split data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Train Random Forest
rf = RandomForestRegressor(n_estimators=100, random_state=42)
rf.fit(X_train, y_train)

# Print feature importances
feature_names = ['Size', 'Age', 'Location', 'Num_Rooms', 'Has_Garage']
for name, importance in zip(feature_names, rf.feature_importances_):
    print(f"{name} importance: {importance:.4f}")

Results for: Real-Life Example: Housing Price Prediction

Size importance: 0.4361
Age importance: 0.1012
Location importance: 0.3486
Num_Rooms importance: 0.0678
Has_Garage importance: 0.0463

Interpreting the Housing Price Prediction Results

The feature importance scores from our housing price prediction model reveal some interesting insights and potential issues:

1. Size is correctly identified as the most important feature, which aligns with our data generation process.
2. Location, our high-cardinality feature, shows high importance despite not being a strong factor in our price calculation. This illustrates the bias towards high-cardinality features.
3. Age and number of rooms show lower importance than expected, possibly due to correlation with size or the influence of the high-cardinality location feature.
4. The garage feature shows the lowest importance, which might be accurate but could also be underestimated due to the presence of stronger or high-cardinality features.

These results highlight the need for careful interpretation of feature importance scores and the potential benefits of using alternative methods like permutation importance for more robust feature evaluation.

# Calculate permutation importance for the housing price model
def permutation_importance(model, X, y, n_repeats=10):
    baseline_mse = mean_squared_error(y, model.predict(X))
    importances = []

    for feature in range(X.shape[1]):
        feature_importances = []
        for _ in range(n_repeats):
            X_permuted = X.copy()
            X_permuted[:, feature] = np.random.permutation(X_permuted[:, feature])
            permuted_mse = mean_squared_error(y, model.predict(X_permuted))
            importance = permuted_mse - baseline_mse
            feature_importances.append(importance)
        importances.append(np.mean(feature_importances))

    return np.array(importances)

# Calculate permutation importance
perm_importance = permutation_importance(rf, X_test, y_test)

# Print permutation importance
for name, importance in zip(feature_names, perm_importance):
    print(f"{name} permutation importance: {importance:.4f}")

Results for: Interpreting the Housing Price Prediction Results

Size permutation importance: 8023056488.4980
Age permutation importance: 570661862.0198
Location permutation importance: 245827.7014
Num_Rooms permutation importance: 318816066.7098
Has_Garage permutation importance: 242554724.1907

Additional Resources

For more in-depth information on feature importance in tree-based models and their limitations, consider the following resources:

1. "Understanding Random Forests: From Theory to Practice" by Gilles Louppe (ArXiv:1407.7502) URL: https://arxiv.org/abs/1407.7502
2. "Feature Importance and Feature Selection With XGBoost in Python" by Jason Brownlee (Machine Learning Mastery blog)
3. "Beware Default Random Forest Importances" by Terence Parr, Kerem Turgutlu, Christopher Csiszar, and Jeremy Howard (Explained.ai article)
4. "Permutation Importance vs Random Forest Feature Importance (MDI)" by Shaked Zychlinski (Towards Data Science article)

These resources provide a deeper understanding of the topic and offer alternative approaches to assessing feature importance in machine learning models.