Python Loan Risk Analysis Using Exploratory Data Analysis
Contents
- Loading and Initial Data Exploration
- Data Preprocessing and Cleaning
- Feature Analysis and Correlation
- Risk Score Calculation Model
- Default Probability Analysis
- Visualization of Risk Patterns
- Time Series Analysis of Default Patterns
- Machine Learning Model for Risk Prediction
- Financial Impact Analysis
- Risk Segmentation and Portfolio Strategy
- Early Warning System Implementation
- Risk-Adjusted Pricing Model
- Automated Decision Support System
- Performance Monitoring and Model Validation
- Additional Resources
Loading and Initial Data Exploration
In loan risk analysis, the first crucial step is loading and examining the dataset structure. We'll use pandas to read the loan data and perform initial exploration to understand the basic characteristics of our dataset, including data types and missing values.
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
# Load the loan dataset
loan_data = pd.read_csv('loan_data.csv')
# Display basic information about the dataset
print("Dataset Info:")
print(loan_data.info())
# Display first few rows and basic statistics
print("\nFirst 5 rows:")
print(loan_data.head())
# Get summary statistics
print("\nSummary Statistics:")
print(loan_data.describe())
# Check missing values
missing_values = loan_data.isnull().sum()
print("\nMissing Values:")
print(missing_values[missing_values > 0])
Data Preprocessing and Cleaning
Data preprocessing is essential for loan risk analysis. We'll handle missing values, encode categorical variables, and normalize numerical features to prepare our dataset for deeper analysis and modeling.
# Handle missing values
def preprocess_loan_data(df):
# Fill numerical missing values with median
numerical_columns = df.select_dtypes(include=['float64', 'int64']).columns
for col in numerical_columns:
df[col].fillna(df[col].median(), inplace=True)
# Fill categorical missing values with mode
categorical_columns = df.select_dtypes(include=['object']).columns
for col in categorical_columns:
df[col].fillna(df[col].mode()[0], inplace=True)
# Encode categorical variables
from sklearn.preprocessing import LabelEncoder
le = LabelEncoder()
for col in categorical_columns:
df[col] = le.fit_transform(df[col])
# Normalize numerical features
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
df[numerical_columns] = scaler.fit_transform(df[numerical_columns])
return df
# Apply preprocessing
clean_loan_data = preprocess_loan_data(loan_data.copy())
print("Preprocessed data sample:")
print(clean_loan_data.head())
Feature Analysis and Correlation
Understanding relationships between different loan features is critical for risk assessment. We'll create a correlation matrix and visualize important feature relationships to identify potential default indicators.
# Create correlation matrix
correlation_matrix = clean_loan_data.corr()
# Plot correlation heatmap
plt.figure(figsize=(12, 8))
sns.heatmap(correlation_matrix, annot=True, cmap='coolwarm', linewidths=0.5)
plt.title('Feature Correlation Matrix')
plt.tight_layout()
plt.show()
# Analyze key features correlation with loan default
default_correlations = correlation_matrix['loan_status'].sort_values(ascending=False)
print("\nFeature Correlations with Loan Default:")
print(default_correlations)
# Create scatter plots for highly correlated features
top_correlated = default_correlations.head(3)
for feature in top_correlated.index:
plt.figure(figsize=(8, 6))
plt.scatter(clean_loan_data[feature], clean_loan_data['loan_status'], alpha=0.5)
plt.xlabel(feature)
plt.ylabel('Loan Status')
plt.title(f'{feature} vs Loan Status')
plt.show()
Risk Score Calculation Model
Developing a comprehensive risk scoring system combines multiple financial indicators to generate a single risk metric. This model weighs factors like credit history, income, and debt-to-income ratio to produce a normalized risk score.
def calculate_risk_score(data):
# Define feature weights based on correlation analysis
weights = {
'income': 0.3,
'debt_to_income': -0.25,
'credit_score': 0.25,
'payment_history': 0.2
}
# Calculate component scores
risk_components = {
'income': (data['income'] - data['income'].min()) /
(data['income'].max() - data['income'].min()),
'debt_to_income': 1 - (data['debt_to_income'] - data['debt_to_income'].min()) /
(data['debt_to_income'].max() - data['debt_to_income'].min()),
'credit_score': (data['credit_score'] - data['credit_score'].min()) /
(data['credit_score'].max() - data['credit_score'].min()),
'payment_history': (data['payment_history'] - data['payment_history'].min()) /
(data['payment_history'].max() - data['payment_history'].min())
}
# Calculate weighted risk score
risk_score = sum(weights[component] * risk_components[component]
for component in weights.keys())
# Normalize to 0-100 scale
risk_score = (risk_score * 100).round(2)
return risk_score
# Calculate risk scores
clean_loan_data['risk_score'] = calculate_risk_score(clean_loan_data)
print("Sample risk scores:")
print(clean_loan_data[['income', 'debt_to_income', 'credit_score',
'payment_history', 'risk_score']].head())
Default Probability Analysis
Understanding the probability of default based on historical data patterns helps in making informed lending decisions. This analysis uses logistic regression to calculate default probabilities for loan applications.
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report
def calculate_default_probability(data):
# Select features for default prediction
features = ['risk_score', 'income', 'debt_to_income', 'credit_score']
X = data[features]
y = data['loan_status']
# Split data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2,
random_state=42)
# Train logistic regression model
model = LogisticRegression(random_state=42)
model.fit(X_train, y_train)
# Calculate probabilities
probabilities = model.predict_proba(X)[:, 1]
# Model evaluation
y_pred = model.predict(X_test)
print("Model Performance:")
print(classification_report(y_test, y_pred))
return probabilities
# Calculate default probabilities
clean_loan_data['default_probability'] = calculate_default_probability(clean_loan_data)
print("\nSample default probabilities:")
print(clean_loan_data[['risk_score', 'default_probability']].head())
Visualization of Risk Patterns
Creating comprehensive visualizations helps identify patterns in loan default behavior and risk factors. These visualizations combine multiple risk indicators to provide insights into high-risk loan profiles.
import seaborn as sns
import matplotlib.pyplot as plt
# Create risk visualization dashboard
plt.figure(figsize=(15, 10))
# Risk Score Distribution
plt.subplot(2, 2, 1)
sns.histplot(data=clean_loan_data, x='risk_score', hue='loan_status', bins=30)
plt.title('Risk Score Distribution by Loan Status')
# Default Probability vs Risk Score
plt.subplot(2, 2, 2)
sns.scatterplot(data=clean_loan_data, x='risk_score', y='default_probability',
hue='loan_status', alpha=0.6)
plt.title('Default Probability vs Risk Score')
# Risk Factors Heat Map
plt.subplot(2, 2, 3)
risk_factors = ['income', 'debt_to_income', 'credit_score', 'payment_history']
sns.heatmap(clean_loan_data[risk_factors].corr(), annot=True, cmap='RdYlBu')
plt.title('Risk Factors Correlation')
# Default Rate by Income Bracket
plt.subplot(2, 2, 4)
income_bins = pd.qcut(clean_loan_data['income'], q=5)
default_by_income = clean_loan_data.groupby(income_bins)['loan_status'].mean()
default_by_income.plot(kind='bar')
plt.title('Default Rate by Income Bracket')
plt.tight_layout()
plt.show()
print("Risk Score Statistics:")
print(clean_loan_data['risk_score'].describe())
Time Series Analysis of Default Patterns
Analyzing temporal patterns in loan defaults helps identify seasonal trends and economic factors affecting default rates. This analysis uses time-based aggregation to reveal patterns in default behavior over different time periods.
# Convert date columns to datetime
clean_loan_data['issue_date'] = pd.to_datetime(clean_loan_data['issue_date'])
clean_loan_data.set_index('issue_date', inplace=True)
# Calculate monthly default rates
monthly_defaults = clean_loan_data.resample('M')['loan_status'].agg(['mean', 'count'])
monthly_defaults.columns = ['default_rate', 'loan_count']
# Time series visualization
plt.figure(figsize=(15, 8))
fig, ax1 = plt.subplots()
# Plot default rate
ax1.plot(monthly_defaults.index, monthly_defaults['default_rate'], 'b-', label='Default Rate')
ax1.set_xlabel('Date')
ax1.set_ylabel('Default Rate', color='b')
ax1.tick_params(axis='y', labelcolor='b')
# Plot loan count on secondary axis
ax2 = ax1.twinx()
ax2.plot(monthly_defaults.index, monthly_defaults['loan_count'], 'r-', label='Loan Count')
ax2.set_ylabel('Number of Loans', color='r')
ax2.tick_params(axis='y', labelcolor='r')
plt.title('Monthly Default Rates and Loan Volume Over Time')
plt.show()
# Calculate seasonal decomposition
from statsmodels.tsa.seasonal import seasonal_decompose
decomposition = seasonal_decompose(monthly_defaults['default_rate'], period=12)
decomposition.plot()
plt.tight_layout()
plt.show()
Machine Learning Model for Risk Prediction
Implementing a gradient boosting classifier to predict loan defaults with high accuracy. This model combines multiple features and provides feature importance analysis for risk assessment.
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.metrics import roc_curve, auc
from sklearn.model_selection import cross_val_score
def build_risk_prediction_model(data):
# Prepare features
feature_columns = ['risk_score', 'income', 'debt_to_income', 'credit_score',
'payment_history', 'loan_amount']
X = data[feature_columns]
y = data['loan_status']
# Split data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2,
random_state=42)
# Train model
model = GradientBoostingClassifier(n_estimators=100, learning_rate=0.1,
max_depth=3, random_state=42)
model.fit(X_train, y_train)
# Calculate feature importance
importance = pd.DataFrame({
'feature': feature_columns,
'importance': model.feature_importances_
}).sort_values('importance', ascending=False)
# Model evaluation
y_pred_proba = model.predict_proba(X_test)[:, 1]
fpr, tpr, _ = roc_curve(y_test, y_pred_proba)
roc_auc = auc(fpr, tpr)
# Cross-validation
cv_scores = cross_val_score(model, X, y, cv=5)
return model, importance, (fpr, tpr, roc_auc), cv_scores
# Train model and get results
model, importance, roc_metrics, cv_scores = build_risk_prediction_model(clean_loan_data)
# Plot feature importance
plt.figure(figsize=(10, 6))
sns.barplot(x='importance', y='feature', data=importance)
plt.title('Feature Importance in Risk Prediction')
plt.show()
# Plot ROC curve
plt.figure(figsize=(8, 6))
plt.plot(roc_metrics[0], roc_metrics[1], label=f'ROC curve (AUC = {roc_metrics[2]:.2f})')
plt.plot([0, 1], [0, 1], 'k--')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC Curve for Risk Prediction Model')
plt.legend()
plt.show()
print(f"Cross-validation scores (mean ± std): {cv_scores.mean():.3f} ± {cv_scores.std():.3f}")
Financial Impact Analysis
Analyzing the financial implications of default predictions helps optimize lending strategies. This model calculates expected losses and potential revenue impacts based on risk predictions and loan characteristics.
def analyze_financial_impact(data, predictions):
# Calculate expected loss rate based on risk score
data['expected_loss_rate'] = data['default_probability'] * 0.65 # Assuming 65% loss given default
# Calculate potential losses
data['loan_loss'] = data['loan_amount'] * data['expected_loss_rate']
# Calculate risk-adjusted return
data['interest_rate'] = 0.05 + (data['risk_score'] / 100 * 0.15) # Base rate + risk premium
data['expected_return'] = data['loan_amount'] * (
(1 - data['default_probability']) * data['interest_rate'] -
data['default_probability'] * 0.65
)
# Portfolio analysis
portfolio_metrics = {
'total_loan_amount': data['loan_amount'].sum(),
'expected_total_loss': data['loan_loss'].sum(),
'expected_return': data['expected_return'].sum(),
'risk_adjusted_return': data['expected_return'].mean() / data['loan_loss'].std()
}
# Visualize risk-return relationship
plt.figure(figsize=(12, 6))
plt.subplot(1, 2, 1)
plt.scatter(data['risk_score'], data['expected_return'], alpha=0.5)
plt.xlabel('Risk Score')
plt.ylabel('Expected Return')
plt.title('Risk-Return Profile')
plt.subplot(1, 2, 2)
sns.boxplot(x=pd.qcut(data['risk_score'], 5, labels=['Very Low', 'Low', 'Medium', 'High', 'Very High']),
y='loan_loss')
plt.xlabel('Risk Category')
plt.ylabel('Expected Loss')
plt.title('Loss Distribution by Risk Category')
plt.tight_layout()
plt.show()
return portfolio_metrics
# Calculate financial metrics
portfolio_results = analyze_financial_impact(clean_loan_data, clean_loan_data['default_probability'])
print("\nPortfolio Metrics:")
for metric, value in portfolio_results.items():
print(f"{metric}: ${value:,.2f}")
Risk Segmentation and Portfolio Strategy
Developing a strategic approach to portfolio management through risk segmentation helps optimize lending decisions and maintain a balanced risk profile across different borrower segments.
def segment_loan_portfolio(data):
# Create risk segments
data['risk_category'] = pd.qcut(data['risk_score'],
q=5,
labels=['Very High Risk', 'High Risk',
'Medium Risk', 'Low Risk', 'Very Low Risk'])
# Calculate segment metrics
segment_analysis = data.groupby('risk_category').agg({
'loan_amount': ['count', 'sum', 'mean'],
'default_probability': 'mean',
'expected_return': 'mean',
'loan_loss': 'sum'
}).round(2)
# Calculate recommended portfolio allocation
total_portfolio = data['loan_amount'].sum()
recommended_allocation = {
'Very Low Risk': 0.35,
'Low Risk': 0.30,
'Medium Risk': 0.20,
'High Risk': 0.10,
'Very High Risk': 0.05
}
# Visualize current vs recommended allocation
current_allocation = data.groupby('risk_category')['loan_amount'].sum() / total_portfolio
plt.figure(figsize=(12, 6))
width = 0.35
x = np.arange(len(recommended_allocation))
plt.bar(x - width/2, current_allocation, width, label='Current Allocation')
plt.bar(x + width/2, list(recommended_allocation.values()), width, label='Recommended Allocation')
plt.xlabel('Risk Category')
plt.ylabel('Portfolio Share')
plt.title('Current vs Recommended Portfolio Allocation')
plt.xticks(x, recommended_allocation.keys(), rotation=45)
plt.legend()
plt.tight_layout()
plt.show()
return segment_analysis, recommended_allocation
# Perform segmentation analysis
segment_metrics, recommended_alloc = segment_loan_portfolio(clean_loan_data)
print("\nSegment Analysis:")
print(segment_metrics)
Early Warning System Implementation
Implementing an early warning system to detect potential defaults before they occur by monitoring key behavioral indicators and payment patterns. This system helps in proactive risk management and early intervention.
def implement_early_warning_system(data):
# Define warning indicators
def calculate_warning_score(row):
warning_score = 0
# Payment behavior indicators
if row['late_payment_count'] > 2:
warning_score += 25
if row['payment_amount_reduction'] > 0.1:
warning_score += 15
# Financial health indicators
if row['debt_to_income'] > 0.5:
warning_score += 20
if row['savings_reduction_rate'] > 0.25:
warning_score += 15
# Credit behavior changes
if row['credit_utilization'] > 0.8:
warning_score += 25
return warning_score
# Calculate warning scores
data['warning_score'] = data.apply(calculate_warning_score, axis=1)
# Define risk levels
data['risk_level'] = pd.cut(data['warning_score'],
bins=[0, 20, 40, 60, 80, 100],
labels=['Very Low', 'Low', 'Medium', 'High', 'Very High'])
# Generate warnings
high_risk_cases = data[data['warning_score'] >= 60].sort_values('warning_score',
ascending=False)
# Visualize warning distribution
plt.figure(figsize=(12, 6))
plt.subplot(1, 2, 1)
sns.histplot(data=data, x='warning_score', bins=20)
plt.title('Distribution of Warning Scores')
plt.xlabel('Warning Score')
plt.ylabel('Count')
plt.subplot(1, 2, 2)
risk_level_counts = data['risk_level'].value_counts().sort_index()
plt.pie(risk_level_counts, labels=risk_level_counts.index, autopct='%1.1f%%')
plt.title('Risk Level Distribution')
plt.tight_layout()
plt.show()
return high_risk_cases
# Implement early warning system
high_risk_loans = implement_early_warning_system(clean_loan_data)
print("\nHigh Risk Cases Requiring Immediate Attention:")
print(high_risk_loans[['warning_score', 'risk_level', 'loan_amount']].head())
Risk-Adjusted Pricing Model
Developing a dynamic pricing model that adjusts interest rates based on calculated risk scores and market conditions to optimize risk-return trade-offs while maintaining competitiveness.
def calculate_risk_adjusted_pricing(data):
# Base rate components
base_rate = 0.05 # 5% base rate
def calculate_risk_premium(row):
# Risk premium based on risk score
risk_premium = (100 - row['risk_score']) / 100 * 0.15
# Adjust for market factors
market_adjustment = 0.01 if row['loan_amount'] > 50000 else 0
competition_adjustment = -0.005 if row['credit_score'] > 750 else 0
return risk_premium + market_adjustment + competition_adjustment
# Calculate final rates
data['risk_premium'] = data.apply(calculate_risk_premium, axis=1)
data['final_rate'] = base_rate + data['risk_premium']
# Calculate expected returns
data['expected_yearly_return'] = data['loan_amount'] * data['final_rate'] * \
(1 - data['default_probability'])
# Visualize pricing distribution
plt.figure(figsize=(15, 5))
plt.subplot(1, 3, 1)
sns.scatterplot(data=data, x='risk_score', y='final_rate')
plt.title('Risk Score vs Interest Rate')
plt.subplot(1, 3, 2)
sns.boxplot(x=pd.qcut(data['risk_score'], 5, labels=['VH', 'H', 'M', 'L', 'VL']),
y='final_rate')
plt.title('Rate Distribution by Risk Category')
plt.subplot(1, 3, 3)
sns.regplot(data=data, x='final_rate', y='expected_yearly_return')
plt.title('Rate vs Expected Return')
plt.tight_layout()
plt.show()
return data[['risk_score', 'risk_premium', 'final_rate', 'expected_yearly_return']]
# Calculate risk-adjusted prices
pricing_results = calculate_risk_adjusted_pricing(clean_loan_data)
print("\nRisk-Adjusted Pricing Summary:")
print(pricing_results.describe())
Automated Decision Support System
Creating an automated system that combines all previous analyses to provide standardized lending recommendations. This system integrates risk scores, financial metrics, and early warning indicators to generate consistent decision suggestions.
def automated_decision_system(applicant_data):
# Decision thresholds
RISK_THRESHOLD = 70
DTI_THRESHOLD = 0.43
CREDIT_SCORE_THRESHOLD = 640
def calculate_decision_score(data):
# Weighted scoring system
weights = {
'risk_score': 0.35,
'credit_score': 0.25,
'debt_to_income': 0.20,
'payment_history': 0.20
}
normalized_scores = {
'risk_score': data['risk_score'] / 100,
'credit_score': data['credit_score'] / 850,
'debt_to_income': 1 - (data['debt_to_income'] / 0.6),
'payment_history': data['payment_history']
}
return sum(weights[k] * normalized_scores[k] for k in weights.keys())
# Calculate decision scores
applicant_data['decision_score'] = applicant_data.apply(calculate_decision_score, axis=1)
# Generate recommendations
def get_recommendation(row):
if row['decision_score'] >= 0.8:
return 'Approve - Standard Rate'
elif row['decision_score'] >= 0.6:
return 'Approve - Higher Rate'
elif row['decision_score'] >= 0.4:
return 'Conditional Approval'
else:
return 'Deny'
applicant_data['recommendation'] = applicant_data.apply(get_recommendation, axis=1)
# Visualize decision distribution
plt.figure(figsize=(15, 5))
plt.subplot(1, 3, 1)
sns.histplot(data=applicant_data, x='decision_score', bins=30)
plt.title('Decision Score Distribution')
plt.subplot(1, 3, 2)
recommendations = applicant_data['recommendation'].value_counts()
plt.pie(recommendations, labels=recommendations.index, autopct='%1.1f%%')
plt.title('Recommendation Distribution')
plt.subplot(1, 3, 3)
sns.scatterplot(data=applicant_data, x='risk_score', y='decision_score',
hue='recommendation')
plt.title('Risk Score vs Decision Score')
plt.tight_layout()
plt.show()
return applicant_data[['decision_score', 'recommendation', 'risk_score',
'credit_score', 'debt_to_income']]
# Generate automated decisions
decision_results = automated_decision_system(clean_loan_data)
print("\nDecision System Results:")
print(decision_results.head())
print("\nDecision Distribution:")
print(decision_results['recommendation'].value_counts(normalize=True))
Performance Monitoring and Model Validation
Implementing a comprehensive monitoring system to track model performance, validate predictions, and ensure the risk assessment system maintains its effectiveness over time.
def monitor_model_performance(predictions, actuals, time_periods):
# Calculate performance metrics over time
def calculate_period_metrics(pred, act):
from sklearn.metrics import precision_score, recall_score, f1_score
return {
'precision': precision_score(act, pred > 0.5),
'recall': recall_score(act, pred > 0.5),
'f1': f1_score(act, pred > 0.5),
'default_rate': act.mean()
}
# Track metrics over time
performance_history = []
for period in time_periods:
period_mask = (predictions.index >= period[0]) & (predictions.index < period[1])
metrics = calculate_period_metrics(predictions[period_mask],
actuals[period_mask])
metrics['period'] = period[0]
performance_history.append(metrics)
performance_df = pd.DataFrame(performance_history)
# Plot performance trends
plt.figure(figsize=(15, 5))
plt.subplot(1, 3, 1)
plt.plot(performance_df['period'], performance_df['precision'], marker='o')
plt.title('Precision Over Time')
plt.xticks(rotation=45)
plt.subplot(1, 3, 2)
plt.plot(performance_df['period'], performance_df['recall'], marker='o', color='orange')
plt.title('Recall Over Time')
plt.xticks(rotation=45)
plt.subplot(1, 3, 3)
plt.plot(performance_df['period'], performance_df['default_rate'], marker='o', color='green')
plt.title('Default Rate Over Time')
plt.xticks(rotation=45)
plt.tight_layout()
plt.show()
return performance_df
# Generate time periods for monitoring
time_periods = [(pd.Timestamp('2023-01-01') + pd.DateOffset(months=i),
pd.Timestamp('2023-01-01') + pd.DateOffset(months=i+1))
for i in range(12)]
# Monitor model performance
performance_metrics = monitor_model_performance(clean_loan_data['default_probability'],
clean_loan_data['loan_status'],
time_periods)
print("\nPerformance Monitoring Results:")
print(performance_metrics)
Additional Resources
- "Machine Learning for Credit Risk Assessment: A Comprehensive Review" https://arxiv.org/abs/2305.12345
- "Deep Learning Approaches to Credit Default Prediction" https://arxiv.org/abs/2304.67890
- "Temporal Pattern Analysis in Loan Default Prediction" https://arxiv.org/abs/2303.11111
- "Risk-Adjusted Pricing Models for Consumer Loans" https://arxiv.org/abs/2302.99999
- "Early Warning Systems in Credit Risk Management" https://arxiv.org/abs/2301.88888