Step-by-Step Guide to Building a Fraud Detection Model with Random Forest and XGBoost in Python

Introduction to Fraud Detection Models

Fraud detection is a critical application of machine learning in various industries. This presentation will guide you through building a fraud detection model using two powerful algorithms: Random Forest and XGBoost. We'll use Python to implement these models, focusing on practical, actionable steps for beginners and intermediate practitioners.

import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from xgboost import XGBClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, confusion_matrix

# Load a sample dataset (replace with your own data)
data = pd.read_csv('fraud_data.csv')
print(data.head())

Data Preparation

Before building our models, we need to prepare our data. This involves loading the dataset, handling missing values, encoding categorical variables, and splitting the data into training and testing sets.

# Handle missing values
data = data.fillna(data.mean())

# Encode categorical variables
data = pd.get_dummies(data, columns=['category', 'payment_method'])

# Split features and target
X = data.drop('is_fraud', axis=1)
y = data['is_fraud']

# Split into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

print("Training set shape:", X_train.shape)
print("Testing set shape:", X_test.shape)

Random Forest Model

Random Forest is an ensemble learning method that constructs multiple decision trees and merges them to get a more accurate and stable prediction. Let's implement a Random Forest classifier for our fraud detection model.

# Initialize and train the Random Forest model
rf_model = RandomForestClassifier(n_estimators=100, random_state=42)
rf_model.fit(X_train, y_train)

# Make predictions
rf_predictions = rf_model.predict(X_test)

# Evaluate the model
rf_accuracy = accuracy_score(y_test, rf_predictions)
rf_cm = confusion_matrix(y_test, rf_predictions)

print("Random Forest Accuracy:", rf_accuracy)
print("Random Forest Confusion Matrix:\n", rf_cm)

XGBoost Model

XGBoost (eXtreme Gradient Boosting) is another powerful ensemble learning method that uses gradient boosting to create a strong predictive model. Let's implement an XGBoost classifier for our fraud detection task.

# Initialize and train the XGBoost model
xgb_model = XGBClassifier(n_estimators=100, random_state=42)
xgb_model.fit(X_train, y_train)

# Make predictions
xgb_predictions = xgb_model.predict(X_test)

# Evaluate the model
xgb_accuracy = accuracy_score(y_test, xgb_predictions)
xgb_cm = confusion_matrix(y_test, xgb_predictions)

print("XGBoost Accuracy:", xgb_accuracy)
print("XGBoost Confusion Matrix:\n", xgb_cm)

Feature Importance

Understanding which features contribute most to our models' decisions is crucial. Both Random Forest and XGBoost provide methods to calculate feature importance. Let's visualize this for both models.

import matplotlib.pyplot as plt

# Get feature importance for Random Forest
rf_importance = pd.DataFrame({'feature': X.columns, 'importance': rf_model.feature_importances_})
rf_importance = rf_importance.sort_values('importance', ascending=False).head(10)

# Get feature importance for XGBoost
xgb_importance = pd.DataFrame({'feature': X.columns, 'importance': xgb_model.feature_importances_})
xgb_importance = xgb_importance.sort_values('importance', ascending=False).head(10)

# Plot feature importance
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15, 5))

rf_importance.plot(x='feature', y='importance', kind='bar', ax=ax1, title='Random Forest Feature Importance')
xgb_importance.plot(x='feature', y='importance', kind='bar', ax=ax2, title='XGBoost Feature Importance')

plt.tight_layout()
plt.show()

Model Comparison

Now that we have implemented both Random Forest and XGBoost models, let's compare their performance using various metrics such as accuracy, precision, recall, and F1-score.

from sklearn.metrics import classification_report

# Generate classification reports
rf_report = classification_report(y_test, rf_predictions)
xgb_report = classification_report(y_test, xgb_predictions)

print("Random Forest Classification Report:")
print(rf_report)
print("\nXGBoost Classification Report:")
print(xgb_report)

# Compare ROC curves
from sklearn.metrics import roc_curve, auc
import matplotlib.pyplot as plt

rf_fpr, rf_tpr, _ = roc_curve(y_test, rf_model.predict_proba(X_test)[:, 1])
xgb_fpr, xgb_tpr, _ = roc_curve(y_test, xgb_model.predict_proba(X_test)[:, 1])

plt.figure(figsize=(10, 6))
plt.plot(rf_fpr, rf_tpr, label=f'Random Forest (AUC = {auc(rf_fpr, rf_tpr):.2f})')
plt.plot(xgb_fpr, xgb_tpr, label=f'XGBoost (AUC = {auc(xgb_fpr, xgb_tpr):.2f})')
plt.plot([0, 1], [0, 1], linestyle='--')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver Operating Characteristic (ROC) Curve')
plt.legend()
plt.show()

Hyperparameter Tuning

To improve our models' performance, we can tune their hyperparameters. We'll use GridSearchCV to find the best combination of hyperparameters for both Random Forest and XGBoost.

from sklearn.model_selection import GridSearchCV

# Random Forest hyperparameter tuning
rf_param_grid = {
    'n_estimators': [50, 100, 200],
    'max_depth': [5, 10, None],
    'min_samples_split': [2, 5, 10]
}

rf_grid_search = GridSearchCV(RandomForestClassifier(random_state=42), rf_param_grid, cv=3, n_jobs=-1)
rf_grid_search.fit(X_train, y_train)

print("Best Random Forest parameters:", rf_grid_search.best_params_)
print("Best Random Forest score:", rf_grid_search.best_score_)

# XGBoost hyperparameter tuning
xgb_param_grid = {
    'n_estimators': [50, 100, 200],
    'max_depth': [3, 5, 7],
    'learning_rate': [0.01, 0.1, 0.3]
}

xgb_grid_search = GridSearchCV(XGBClassifier(random_state=42), xgb_param_grid, cv=3, n_jobs=-1)
xgb_grid_search.fit(X_train, y_train)

print("Best XGBoost parameters:", xgb_grid_search.best_params_)
print("Best XGBoost score:", xgb_grid_search.best_score_)

Model Interpretation with SHAP

SHAP (SHapley Additive exPlanations) values help us understand how each feature contributes to the model's predictions for individual instances. Let's use SHAP to interpret our XGBoost model.

import shap

# Create a SHAP explainer for the XGBoost model
explainer = shap.TreeExplainer(xgb_model)

# Calculate SHAP values for the test set
shap_values = explainer.shap_values(X_test)

# Visualize SHAP values for a single prediction
shap.initjs()
shap.force_plot(explainer.expected_value, shap_values[0], X_test.iloc[0])

# Plot summary of SHAP values
shap.summary_plot(shap_values, X_test)

Handling Imbalanced Data

Fraud detection datasets are often imbalanced, with many more non-fraud cases than fraud cases. Let's explore techniques to handle this imbalance, such as using class weights and oversampling.

from sklearn.utils.class_weight import compute_class_weight
from imblearn.over_sampling import SMOTE

# Compute class weights
class_weights = compute_class_weight('balanced', classes=np.unique(y_train), y=y_train)
class_weight_dict = dict(zip(np.unique(y_train), class_weights))

# Train Random Forest with class weights
rf_weighted = RandomForestClassifier(n_estimators=100, class_weight=class_weight_dict, random_state=42)
rf_weighted.fit(X_train, y_train)

# Apply SMOTE oversampling
smote = SMOTE(random_state=42)
X_train_resampled, y_train_resampled = smote.fit_resample(X_train, y_train)

# Train XGBoost on resampled data
xgb_resampled = XGBClassifier(n_estimators=100, random_state=42)
xgb_resampled.fit(X_train_resampled, y_train_resampled)

# Evaluate and compare the models
print("Weighted Random Forest Accuracy:", accuracy_score(y_test, rf_weighted.predict(X_test)))
print("XGBoost with SMOTE Accuracy:", accuracy_score(y_test, xgb_resampled.predict(X_test)))

Cross-Validation

To ensure our models generalize well, we'll use k-fold cross-validation to evaluate their performance across different subsets of the data.

from sklearn.model_selection import cross_val_score

# Perform 5-fold cross-validation for Random Forest
rf_cv_scores = cross_val_score(rf_model, X, y, cv=5)

# Perform 5-fold cross-validation for XGBoost
xgb_cv_scores = cross_val_score(xgb_model, X, y, cv=5)

print("Random Forest CV Scores:", rf_cv_scores)
print("Random Forest Mean CV Score:", rf_cv_scores.mean())
print("\nXGBoost CV Scores:", xgb_cv_scores)
print("XGBoost Mean CV Score:", xgb_cv_scores.mean())

# Visualize cross-validation results
plt.figure(figsize=(10, 6))
plt.boxplot([rf_cv_scores, xgb_cv_scores], labels=['Random Forest', 'XGBoost'])
plt.title('Cross-Validation Scores')
plt.ylabel('Accuracy')
plt.show()

Real-Life Example: E-commerce Order Fraud Detection

Let's apply our fraud detection models to an e-commerce scenario. We'll use features such as order value, customer history, shipping address, and payment method to predict fraudulent orders.

# Sample e-commerce order data
order_data = pd.DataFrame({
    'order_value': [100, 500, 50, 1000, 200],
    'customer_age_days': [30, 365, 10, 180, 90],
    'shipping_billing_match': [1, 1, 0, 1, 1],
    'payment_method': ['credit_card', 'paypal', 'gift_card', 'credit_card', 'debit_card'],
    'is_fraud': [0, 0, 1, 0, 0]
})

# Prepare the data
order_data_encoded = pd.get_dummies(order_data, columns=['payment_method'])
X_order = order_data_encoded.drop('is_fraud', axis=1)
y_order = order_data_encoded['is_fraud']

# Train and evaluate the model
xgb_order = XGBClassifier(random_state=42)
xgb_order.fit(X_order, y_order)

# Make predictions on new data
new_order = pd.DataFrame({
    'order_value': [750],
    'customer_age_days': [5],
    'shipping_billing_match': [0],
    'payment_method_credit_card': [1],
    'payment_method_debit_card': [0],
    'payment_method_gift_card': [0],
    'payment_method_paypal': [0]
})

prediction = xgb_order.predict(new_order)
print("Fraud prediction for new order:", "Fraudulent" if prediction[0] == 1 else "Legitimate")

Real-Life Example: Spam Email Detection

Another common application of fraud detection is identifying spam emails. We'll use features such as email content, sender information, and message metadata to classify emails as spam or not spam.

from sklearn.feature_extraction.text import TfidfVectorizer

# Sample email data
emails = [
    "Get rich quick! Limited time offer!",
    "Meeting agenda for tomorrow's conference",
    "Congratulations! You've won a free iPhone!",
    "Quarterly report attached for your review",
    "Urgent: Your account has been suspended"
]
labels = [1, 0, 1, 0, 1]  # 1 for spam, 0 for not spam

# Convert text to numerical features using TF-IDF
vectorizer = TfidfVectorizer()
X_emails = vectorizer.fit_transform(emails)

# Train and evaluate the model
rf_spam = RandomForestClassifier(n_estimators=100, random_state=42)
rf_spam.fit(X_emails, labels)

# Make predictions on new emails
new_emails = [
    "Free trial membership for premium services!",
    "Project status update and next steps"
]
new_email_features = vectorizer.transform(new_emails)
predictions = rf_spam.predict(new_email_features)

for email, pred in zip(new_emails, predictions):
    print(f"Email: {email}")
    print(f"Prediction: {'Spam' if pred == 1 else 'Not Spam'}\n")

Model Deployment and Monitoring

Once we have a well-performing fraud detection model, it's crucial to deploy it effectively and monitor its performance over time. Here's an example of how to save and load our model, and set up basic monitoring.

import joblib
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score
import datetime
import pandas as pd

# Save the model
joblib.dump(xgb_model, 'fraud_detection_model.joblib')

# Load the model (simulating deployment)
loaded_model = joblib.load('fraud_detection_model.joblib')

# Function to make predictions and log results
def predict_and_log(model, X, y_true):
    y_pred = model.predict(X)
    metrics = {
        'accuracy': accuracy_score(y_true, y_pred),
        'precision': precision_score(y_true, y_pred),
        'recall': recall_score(y_true, y_pred),
        'f1_score': f1_score(y_true, y_pred)
    }
    
    # Log the results
    log_entry = pd.DataFrame([metrics], index=[datetime.datetime.now()])
    log_entry.to_csv('model_performance_log.csv', mode='a', header=False)
    
    return y_pred, metrics

# Simulate periodic monitoring
for i in range(3):  # Simulating 3 time periods
    print(f"Time period {i+1}")
    # In practice, you would use new data for each time period
    predictions, metrics = predict_and_log(loaded_model, X_test, y_test)
    print(f"Metrics: {metrics}")
    print()

# Read and display the log
log_df = pd.read_csv('model_performance_log.csv', 
                     names=['timestamp', 'accuracy', 'precision', 'recall', 'f1_score'],
                     parse_dates=['timestamp'],
                     index_col='timestamp')
print("Model Performance Log:")
print(log_df)

# Plot performance metrics over time
log_df.plot(figsize=(10, 6))
plt.title('Model Performance Metrics Over Time')
plt.ylabel('Score')
plt.xlabel('Timestamp')
plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))
plt.tight_layout()
plt.show()

Handling Concept Drift

Concept drift occurs when the statistical properties of the target variable change over time. In fraud detection, this can happen as fraudsters adapt their techniques. Let's implement a simple drift detection method.

import numpy as np
from scipy import stats

def detect_drift(baseline_predictions, new_predictions, threshold=0.05):
    # Perform Kolmogorov-Smirnov test
    ks_statistic, p_value = stats.ks_2samp(baseline_predictions, new_predictions)
    
    if p_value < threshold:
        print(f"Concept drift detected! p-value: {p_value}")
        return True
    else:
        print(f"No significant drift detected. p-value: {p_value}")
        return False

# Generate baseline predictions
baseline_predictions = xgb_model.predict_proba(X_test)[:, 1]

# Simulate new data (with potential drift)
np.random.seed(42)
drift_factor = np.random.normal(1, 0.2, size=X_test.shape[0])
X_test_drift = X_test * drift_factor

# Generate new predictions
new_predictions = xgb_model.predict_proba(X_test_drift)[:, 1]

# Detect drift
drift_detected = detect_drift(baseline_predictions, new_predictions)

if drift_detected:
    print("Consider retraining the model or investigating the cause of the drift.")
else:
    print("The model appears to be stable.")

Additional Resources

For further exploration of fraud detection techniques and machine learning algorithms, consider the following resources:

1. "XGBoost: A Scalable Tree Boosting System" by Chen and Guestrin (2016) ArXiv: https://arxiv.org/abs/1603.02754
2. "Random Forests" by Breiman (2001) ArXiv: https://www.stat.berkeley.edu/~breiman/randomforest2001.pdf
3. "A Survey of Credit Card Fraud Detection Techniques" by Zojaji et al. (2016) ArXiv: https://arxiv.org/abs/1611.06439
4. "Learning from Imbalanced Data" by He and Garcia (2009) IEEE: https://ieeexplore.ieee.org/document/5128907

These resources provide in-depth information on the algorithms we've used and additional techniques for fraud detection and handling imbalanced datasets.