Unveiling Feature Selection in Machine Learning with Python

The Importance of Feature Selection

Feature selection is a crucial step in machine learning that involves choosing the most relevant variables for your model. It helps improve model performance, reduce overfitting, and increase interpretability. By selecting the right features, we can build more efficient and accurate models.

import numpy as np
import matplotlib.pyplot as plt

# Generate sample data
np.random.seed(42)
X = np.random.randn(100, 2)
y = 3*X[:, 0] + 2*X[:, 1] + np.random.randn(100)

# Plot the data
plt.figure(figsize=(10, 6))
plt.scatter(X[:, 0], y, c='b', label='Feature 1')
plt.scatter(X[:, 1], y, c='r', label='Feature 2')
plt.xlabel('Feature Value')
plt.ylabel('Target Variable')
plt.legend()
plt.title('Relationship between Features and Target Variable')
plt.show()

Types of Feature Selection Methods

There are three main types of feature selection methods: filter methods, wrapper methods, and embedded methods. Filter methods use statistical techniques to evaluate the relevance of features. Wrapper methods use a predictive model to score feature subsets. Embedded methods perform feature selection as part of the model training process.

from sklearn.feature_selection import SelectKBest, f_regression
from sklearn.datasets import make_regression

# Generate a random regression problem
X, y = make_regression(n_samples=100, n_features=20, n_informative=2, noise=0.1, random_state=42)

# Apply filter method (f_regression)
selector = SelectKBest(score_func=f_regression, k=5)
X_selected = selector.fit_transform(X, y)

print(f"Original number of features: {X.shape[1]}")
print(f"Number of features after selection: {X_selected.shape[1]}")

Correlation-based Feature Selection

Correlation-based feature selection is a simple yet effective method. It involves calculating the correlation between features and the target variable, as well as between features themselves. Features with high correlation to the target and low correlation with other features are preferred.

import pandas as pd
import seaborn as sns

# Create a sample dataset
data = pd.DataFrame(np.random.randn(100, 5), columns=['A', 'B', 'C', 'D', 'target'])
data['target'] = 2*data['A'] + 3*data['B'] + np.random.randn(100)

# Calculate correlation matrix
corr_matrix = data.corr()

# Plot heatmap
plt.figure(figsize=(10, 8))
sns.heatmap(corr_matrix, annot=True, cmap='coolwarm', vmin=-1, vmax=1, center=0)
plt.title('Correlation Heatmap')
plt.show()

# Select features based on correlation with target
threshold = 0.5
selected_features = corr_matrix['target'][abs(corr_matrix['target']) > threshold].index.tolist()
selected_features.remove('target')
print(f"Selected features: {selected_features}")

Variance Threshold

Variance threshold is a simple technique to remove features with low variance. This method is particularly useful for removing constant or quasi-constant features that don't contribute much information to the model.

from sklearn.feature_selection import VarianceThreshold

# Create a sample dataset with some low-variance features
X = np.array([[0, 2, 0, 3], [0, 1, 4, 3], [0, 1, 1, 3]])

# Apply VarianceThreshold
selector = VarianceThreshold(threshold=0.1)
X_selected = selector.fit_transform(X)

print("Original features:")
print(X)
print("\nSelected features:")
print(X_selected)
print(f"\nNumber of features removed: {X.shape[1] - X_selected.shape[1]}")

Recursive Feature Elimination (RFE)

Recursive Feature Elimination is a wrapper method that works by recursively removing attributes and building a model on those attributes that remain. It uses the model accuracy to identify which attributes (and combination of attributes) contribute the most to predicting the target attribute.

from sklearn.feature_selection import RFE
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification

# Generate a random classification problem
X, y = make_classification(n_samples=1000, n_features=20, n_informative=2, n_redundant=10, n_classes=2, random_state=42)

# Create the RFE object and specify the estimator
estimator = LogisticRegression()
selector = RFE(estimator, n_features_to_select=5, step=1)

# Fit RFE
selector = selector.fit(X, y)

# Print the selected features
selected_features = [i for i, selected in enumerate(selector.support_) if selected]
print(f"Selected features: {selected_features}")

# Print feature ranking
print("\nFeature Ranking:")
for i, rank in enumerate(selector.ranking_):
    print(f"Feature {i}: Rank {rank}")

Lasso Regularization for Feature Selection

Lasso (Least Absolute Shrinkage and Selection Operator) is a regression analysis method that performs both variable selection and regularization. It adds a penalty term to the loss function, which can force some feature coefficients to zero, effectively selecting features.

from sklearn.linear_model import Lasso
from sklearn.preprocessing import StandardScaler

# Generate sample data
np.random.seed(42)
X = np.random.randn(100, 20)
y = 3*X[:, 0] + 2*X[:, 1] - 5*X[:, 2] + np.random.randn(100)

# Standardize features
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# Apply Lasso
lasso = Lasso(alpha=0.1)
lasso.fit(X_scaled, y)

# Print non-zero coefficients
non_zero = [(i, coef) for i, coef in enumerate(lasso.coef_) if coef != 0]
print("Non-zero coefficients (feature index, coefficient):")
for feature, coef in non_zero:
    print(f"Feature {feature}: {coef:.4f}")

# Plot coefficients
plt.figure(figsize=(12, 6))
plt.bar(range(len(lasso.coef_)), lasso.coef_)
plt.xlabel('Feature Index')
plt.ylabel('Coefficient Value')
plt.title('Lasso Coefficients')
plt.show()

Random Forest Feature Importance

Random Forest is an ensemble learning method that can be used for both classification and regression tasks. It provides a feature importance measure based on how much each feature contributes to decreasing the weighted impurity in a tree.

from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification

# Generate a random classification dataset
X, y = make_classification(n_samples=1000, n_features=20, n_informative=5, n_redundant=5, n_classes=2, random_state=42)

# Create and train the random forest classifier
rf = RandomForestClassifier(n_estimators=100, random_state=42)
rf.fit(X, y)

# Get feature importances
importances = rf.feature_importances_

# Sort features by importance
feature_importances = sorted(zip(range(X.shape[1]), importances), key=lambda x: x[1], reverse=True)

# Plot feature importances
plt.figure(figsize=(12, 6))
plt.bar(range(len(importances)), [imp for (feat, imp) in feature_importances])
plt.xlabel('Feature Index')
plt.ylabel('Importance')
plt.title('Random Forest Feature Importances')
plt.show()

# Print top 5 important features
print("Top 5 important features (index, importance):")
for feat, imp in feature_importances[:5]:
    print(f"Feature {feat}: {imp:.4f}")

Principal Component Analysis (PCA)

PCA is a dimensionality reduction technique that can be used for feature selection. It transforms the original features into a new set of uncorrelated features called principal components. By selecting the top k principal components, we can reduce the dimensionality of our dataset while retaining most of the variance.

from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

# Generate sample data
np.random.seed(42)
X = np.random.randn(100, 10)

# Standardize the features
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# Apply PCA
pca = PCA()
X_pca = pca.fit_transform(X_scaled)

# Calculate cumulative explained variance ratio
cumulative_variance_ratio = np.cumsum(pca.explained_variance_ratio_)

# Plot cumulative explained variance ratio
plt.figure(figsize=(10, 6))
plt.plot(range(1, len(cumulative_variance_ratio) + 1), cumulative_variance_ratio, 'bo-')
plt.xlabel('Number of Components')
plt.ylabel('Cumulative Explained Variance Ratio')
plt.title('PCA: Cumulative Explained Variance Ratio')
plt.grid(True)
plt.show()

# Select number of components that explain 95% of the variance
n_components = np.argmax(cumulative_variance_ratio >= 0.95) + 1
print(f"Number of components explaining 95% of variance: {n_components}")

Mutual Information

Mutual Information is a measure of the mutual dependence between two variables. In feature selection, it can be used to measure the relationship between features and the target variable. Features with high mutual information are considered more relevant.

from sklearn.feature_selection import mutual_info_classif
from sklearn.datasets import make_classification

# Generate a random classification dataset
X, y = make_classification(n_samples=1000, n_features=20, n_informative=5, n_redundant=5, n_classes=2, random_state=42)

# Calculate mutual information
mi_scores = mutual_info_classif(X, y)

# Sort features by mutual information score
sorted_mi = sorted(zip(range(X.shape[1]), mi_scores), key=lambda x: x[1], reverse=True)

# Plot mutual information scores
plt.figure(figsize=(12, 6))
plt.bar(range(len(mi_scores)), [mi for (feat, mi) in sorted_mi])
plt.xlabel('Feature Index')
plt.ylabel('Mutual Information')
plt.title('Mutual Information Scores')
plt.show()

# Print top 5 features with highest mutual information
print("Top 5 features (index, mutual information):")
for feat, mi in sorted_mi[:5]:
    print(f"Feature {feat}: {mi:.4f}")

Real-life Example: Iris Dataset

Let's apply feature selection techniques to the famous Iris dataset, which contains measurements of iris flowers. We'll use correlation analysis and mutual information to select the most relevant features for species classification.

from sklearn.datasets import load_iris
from sklearn.feature_selection import mutual_info_classif
import pandas as pd
import seaborn as sns

# Load the Iris dataset
iris = load_iris()
X, y = iris.data, iris.target
feature_names = iris.feature_names

# Create a DataFrame
df = pd.DataFrame(X, columns=feature_names)
df['species'] = y

# Correlation analysis
corr_matrix = df.corr()
plt.figure(figsize=(10, 8))
sns.heatmap(corr_matrix, annot=True, cmap='coolwarm', vmin=-1, vmax=1, center=0)
plt.title('Correlation Heatmap - Iris Dataset')
plt.show()

# Mutual Information
mi_scores = mutual_info_classif(X, y)
sorted_mi = sorted(zip(feature_names, mi_scores), key=lambda x: x[1], reverse=True)

plt.figure(figsize=(10, 6))
plt.bar(range(len(mi_scores)), [mi for (feat, mi) in sorted_mi])
plt.xticks(range(len(mi_scores)), [feat for (feat, mi) in sorted_mi], rotation=45)
plt.xlabel('Feature')
plt.ylabel('Mutual Information')
plt.title('Mutual Information Scores - Iris Dataset')
plt.tight_layout()
plt.show()

print("Feature ranking by Mutual Information:")
for feat, mi in sorted_mi:
    print(f"{feat}: {mi:.4f}")

Real-life Example: Wine Quality Dataset

In this example, we'll use the Wine Quality dataset to demonstrate how feature selection can improve model performance. We'll compare a model using all features to one using only selected features based on correlation and Lasso regularization.

from sklearn.datasets import load_wine
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import Lasso
from sklearn.metrics import mean_squared_error
import pandas as pd

# Load the Wine Quality dataset
wine = load_wine()
X, y = wine.data, wine.target
feature_names = wine.feature_names

# Split the data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Standardize features
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Train a model using all features
from sklearn.ensemble import RandomForestRegressor
rf_all = RandomForestRegressor(n_estimators=100, random_state=42)
rf_all.fit(X_train_scaled, y_train)
y_pred_all = rf_all.predict(X_test_scaled)
mse_all = mean_squared_error(y_test, y_pred_all)

# Feature selection using Lasso
lasso = Lasso(alpha=0.1)
lasso.fit(X_train_scaled, y_train)

# Select features with non-zero coefficients
selected_features = [feature for feature, coef in zip(feature_names, lasso.coef_) if coef != 0]

# Train a model using selected features
X_train_selected = X_train_scaled[:, lasso.coef_ != 0]
X_test_selected = X_test_scaled[:, lasso.coef_ != 0]

rf_selected = RandomForestRegressor(n_estimators=100, random_state=42)
rf_selected.fit(X_train_selected, y_train)
y_pred_selected = rf_selected.predict(X_test_selected)
mse_selected = mean_squared_error(y_test, y_pred_selected)

print(f"MSE (all features): {mse_all:.4f}")
print(f"MSE (selected features): {mse_selected:.4f}")
print(f"Selected features: {selected_features}")

Cross-validation in Feature Selection

Cross-validation is crucial in feature selection to avoid overfitting and ensure that the selected features generalize well to unseen data. We'll demonstrate how to use cross-validation with Recursive Feature Elimination (RFE) to select features robustly.

from sklearn.model_selection import cross_val_score
from sklearn.feature_selection import RFECV
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification

# Generate a random classification dataset
X, y = make_classification(n_samples=1000, n_features=20, n_informative=5, 
                           n_redundant=5, n_classes=2, random_state=42)

# Create the RFE object with cross-validation
rf = RandomForestClassifier(n_estimators=100, random_state=42)
rfecv = RFECV(estimator=rf, step=1, cv=5, scoring='accuracy')

# Fit RFECV
rfecv.fit(X, y)

# Plot number of features VS. cross-validation scores
plt.figure(figsize=(10, 6))
plt.plot(range(1, len(rfecv.grid_scores_) + 1), rfecv.grid_scores_)
plt.xlabel("Number of features selected")
plt.ylabel("Cross-validation score (accuracy)")
plt.title("Feature Selection using RFE with Cross-validation")
plt.show()

print(f"Optimal number of features: {rfecv.n_features_}")
print(f"Best cross-validation score: {rfecv.grid_scores_[rfecv.n_features_ - 1]:.4f}")

Feature Selection Pipeline

Creating a pipeline that combines feature selection with model training can streamline the process and ensure that feature selection is properly cross-validated. We'll demonstrate how to build such a pipeline using scikit-learn.

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV
from sklearn.datasets import load_breast_cancer

# Load the breast cancer dataset
data = load_breast_cancer()
X, y = data.data, data.target

# Create a pipeline
pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('select', SelectKBest(f_classif)),
    ('classifier', SVC())
])

# Define parameter grid
param_grid = {
    'select__k': [5, 10, 15, 20],
    'classifier__C': [0.1, 1, 10],
    'classifier__kernel': ['linear', 'rbf']
}

# Perform grid search
grid_search = GridSearchCV(pipeline, param_grid, cv=5, scoring='accuracy')
grid_search.fit(X, y)

# Print best parameters and score
print("Best parameters:", grid_search.best_params_)
print("Best cross-validation score:", grid_search.best_score_)

# Get selected features
best_k = grid_search.best_params_['select__k']
selected_features = [data.feature_names[i] for i in 
                     grid_search.best_estimator_.named_steps['select'].get_support(indices=True)]
print(f"Selected {best_k} features:", selected_features)

Handling Multicollinearity

Multicollinearity occurs when features are highly correlated with each other. It can lead to unstable and unreliable model estimates. We'll explore how to detect and handle multicollinearity using Variance Inflation Factor (VIF).

import pandas as pd
import numpy as np
from statsmodels.stats.outliers_influence import variance_inflation_factor

# Generate sample data with multicollinearity
np.random.seed(42)
X = np.random.randn(100, 3)
X[:, 2] = X[:, 0] + X[:, 1] + np.random.randn(100) * 0.1
df = pd.DataFrame(X, columns=['A', 'B', 'C'])

# Calculate VIF for each feature
vif_data = pd.DataFrame()
vif_data["feature"] = df.columns
vif_data["VIF"] = [variance_inflation_factor(df.values, i) for i in range(len(df.columns))]

print("Variance Inflation Factors:")
print(vif_data)

# Remove features with high VIF
vif_threshold = 5
high_vif_features = vif_data[vif_data["VIF"] > vif_threshold]["feature"]
df_reduced = df.drop(columns=high_vif_features)

print("\nFeatures removed due to high VIF:", list(high_vif_features))
print("Remaining features:", list(df_reduced.columns))

Additional Resources

For further exploration of feature selection techniques in machine learning, consider the following resources:

1. "Feature Selection for Machine Learning" by Jundong Li et al. (2018) ArXiv: https://arxiv.org/abs/1601.07996
2. "An Introduction to Variable and Feature Selection" by Isabelle Guyon and André Elisseeff (2003) Journal of Machine Learning Research
3. "Feature Selection: A Data Perspective" by Jundong Li et al. (2017) ArXiv: https://arxiv.org/abs/1704.08103

These resources provide in-depth discussions on various feature selection methods, their theoretical foundations, and practical applications in machine learning.