xbe.at

Miscellany / DataFrames and pandas

Cheat Sheet for Solving Any ML Problem

8 min read, 15 sections

Contents
  1. Understanding Classification Problems
  2. Naive Bayes Implementation for Text Classification
  3. Support Vector Machines for Small Datasets
  4. Regression Analysis Fundamentals
  5. Advanced Regression Techniques
  6. Clustering Implementation with K-Means
  7. Dimensionality Reduction with PCA
  8. Real-world Example: Credit Card Fraud Detection
  9. Hyperparameter Optimization Techniques
  10. Ensemble Methods Implementation
  11. Feature Selection and Engineering
  12. Cross-Validation and Model Evaluation
  13. Real-world Example: Customer Churn Prediction
  14. Advanced Model Interpretability
  15. Additional Resources

Understanding Classification Problems

Machine learning classification involves predicting discrete class labels or categories from input features. The process requires labeled training data and careful consideration of model selection based on dataset characteristics, feature types, and computational constraints.

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import classification_report

# Example classification dataset
X = np.random.randn(1000, 20)  # 1000 samples, 20 features
y = np.random.randint(0, 2, 1000)  # Binary classification

# Split dataset
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Standardize features
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

Naive Bayes Implementation for Text Classification

The Naive Bayes classifier excels in text classification due to its probabilistic approach and computational efficiency. It assumes feature independence, making it particularly suitable for high-dimensional text data where word frequencies serve as features.

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB

# Sample text data
texts = [
    "machine learning is fascinating",
    "deep neural networks are complex",
    "statistical analysis reveals patterns",
    "data science drives decisions"
]
labels = [0, 1, 0, 1]  # Binary classification labels

# Convert text to numerical features
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(texts)

# Train Naive Bayes model
nb_classifier = MultinomialNB()
nb_classifier.fit(X, labels)

# Predict new text
new_text = ["artificial intelligence transforms industries"]
new_features = vectorizer.transform(new_text)
prediction = nb_classifier.predict(new_features)
print(f"Prediction: {prediction}")

Support Vector Machines for Small Datasets

Support Vector Machines (SVM) are particularly effective for smaller datasets, offering excellent performance through margin maximization and kernel tricks. They work well for both linear and non-linear classification problems.

from sklearn.svm import SVC
from sklearn.pipeline import Pipeline

# Create SVM pipeline with preprocessing
svm_pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('svm', SVC(kernel='rbf', C=1.0, random_state=42))
])

# Train SVM model
svm_pipeline.fit(X_train, y_train)

# Evaluate performance
y_pred = svm_pipeline.predict(X_test)
print(classification_report(y_test, y_pred))

# Mathematical representation of SVM decision function
"""
$$f(x) = \sum_{i=1}^n \alpha_i y_i K(x_i, x) + b$$
Where K is the kernel function
"""

Regression Analysis Fundamentals

Regression analysis predicts continuous values by modeling relationships between dependent and independent variables. Essential techniques include feature scaling, regularization, and model validation through metrics like MSE and R-squared.

from sklearn.linear_model import LinearRegression, Ridge, Lasso
import matplotlib.pyplot as plt

# Generate regression dataset
X = np.random.randn(100, 1)
y = 3 * X.squeeze() + 2 + np.random.randn(100) * 0.5

# Create and train multiple models
models = {
    'Linear': LinearRegression(),
    'Ridge': Ridge(alpha=1.0),
    'Lasso': Lasso(alpha=1.0)
}

for name, model in models.items():
    model.fit(X, y)
    print(f"{name} R² Score: {model.score(X, y):.4f}")

Advanced Regression Techniques

When dealing with complex regression problems, ensemble methods and regularized models provide robust solutions. This implementation demonstrates Ridge regression with cross-validation and hyperparameter tuning.

from sklearn.linear_model import RidgeCV
from sklearn.model_selection import cross_val_score

# Ridge regression with cross-validation
alphas = np.logspace(-6, 6, 13)
ridge_cv = RidgeCV(alphas=alphas, cv=5)
ridge_cv.fit(X_train_scaled, y_train)

# Mathematical formula for Ridge regression
"""
$$\min_{\beta} ||y - X\beta||^2_2 + \lambda||\beta||^2_2$$
Where λ is the regularization parameter
"""

# Performance evaluation
cv_scores = cross_val_score(ridge_cv, X_train_scaled, y_train, cv=5)
print(f"Best alpha: {ridge_cv.alpha_:.4f}")
print(f"Mean CV Score: {cv_scores.mean():.4f} ± {cv_scores.std():.4f}")

Clustering Implementation with K-Means

K-means clustering partitions data into K distinct groups by iteratively updating cluster centroids and reassigning points. This implementation includes initialization strategies and evaluation metrics for optimal cluster determination.

from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score

# Generate clustering data
X_cluster = np.concatenate([
    np.random.normal(0, 1, (300, 2)),
    np.random.normal(4, 1, (300, 2)),
    np.random.normal(8, 1, (300, 2))
])

# K-means implementation
kmeans = KMeans(n_clusters=3, random_state=42)
cluster_labels = kmeans.fit_predict(X_cluster)

# Evaluate clustering quality
silhouette_avg = silhouette_score(X_cluster, cluster_labels)
print(f"Silhouette Score: {silhouette_avg:.4f}")

# Mathematical representation of K-means objective
"""
$$J = \sum_{i=1}^{k}\sum_{x \in C_i} ||x - \mu_i||^2$$
Where μᵢ represents cluster centroids
"""

Dimensionality Reduction with PCA

Principal Component Analysis (PCA) reduces data dimensionality while preserving maximum variance. This implementation demonstrates the complete PCA workflow, including eigenvalue decomposition and variance explanation ratio calculation.

from sklearn.decomposition import PCA
import matplotlib.pyplot as plt

# Generate high-dimensional data
X_high_dim = np.random.randn(1000, 50)

# Apply PCA
pca = PCA(n_components=0.95)  # Keep 95% of variance
X_reduced = pca.fit_transform(X_high_dim)

# Plot explained variance ratio
plt.plot(np.cumsum(pca.explained_variance_ratio_))
plt.xlabel('Number of Components')
plt.ylabel('Cumulative Explained Variance')

# Mathematical formulation
"""
$$W = \arg\max_{W} \text{tr}(W^T X^T X W)$$
Subject to $W^T W = I$
"""

print(f"Original dimensions: {X_high_dim.shape}")
print(f"Reduced dimensions: {X_reduced.shape}")

Real-world Example: Credit Card Fraud Detection

This implementation demonstrates a complete machine learning pipeline for credit card fraud detection, including data preprocessing, model selection, and evaluation with appropriate metrics for imbalanced datasets.

from sklearn.ensemble import RandomForestClassifier
from sklearn.preprocessing import StandardScaler
from imblearn.over_sampling import SMOTE
from sklearn.metrics import precision_recall_curve

# Simulated credit card transaction data
X_transactions = np.random.randn(10000, 30)
y_fraud = np.random.choice([0, 1], size=10000, p=[0.98, 0.02])

# Handle imbalanced dataset
smote = SMOTE(random_state=42)
X_balanced, y_balanced = smote.fit_resample(X_transactions, y_fraud)

# Train fraud detection model
rf_classifier = RandomForestClassifier(n_estimators=100, random_state=42)
rf_classifier.fit(X_balanced, y_balanced)

# Calculate probability predictions
y_prob = rf_classifier.predict_proba(X_transactions)[:, 1]
precision, recall, thresholds = precision_recall_curve(y_fraud, y_prob)

print(f"Number of features used: {X_transactions.shape[1]}")
print(f"Model accuracy: {rf_classifier.score(X_transactions, y_fraud):.4f}")

Hyperparameter Optimization Techniques

Efficient hyperparameter tuning is crucial for model performance. This implementation showcases various optimization strategies including grid search, random search, and Bayesian optimization.

from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import uniform, randint

# Define parameter space
param_distributions = {
    'n_estimators': randint(50, 200),
    'max_depth': randint(3, 15),
    'learning_rate': uniform(0.01, 0.3)
}

# Random search implementation
random_search = RandomizedSearchCV(
    estimator=rf_classifier,
    param_distributions=param_distributions,
    n_iter=100,
    cv=5,
    random_state=42,
    n_jobs=-1
)

# Fit and evaluate
random_search.fit(X_train_scaled, y_train)
print(f"Best parameters: {random_search.best_params_}")
print(f"Best cross-validation score: {random_search.best_score_:.4f}")

Ensemble Methods Implementation

Ensemble methods combine multiple models to create more robust predictions. This implementation showcases bagging and boosting techniques, demonstrating how to combine weak learners into strong predictive models.

from sklearn.ensemble import GradientBoostingClassifier, BaggingClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import roc_auc_score

# Create base models
base_tree = DecisionTreeClassifier(max_depth=3)

# Initialize ensemble models
bagging = BaggingClassifier(
    base_estimator=base_tree,
    n_estimators=100,
    random_state=42
)

boosting = GradientBoostingClassifier(
    n_estimators=100,
    learning_rate=0.1,
    max_depth=3,
    random_state=42
)

# Train and evaluate
models = {'Bagging': bagging, 'Boosting': boosting}
for name, model in models.items():
    model.fit(X_train_scaled, y_train)
    y_pred_proba = model.predict_proba(X_test_scaled)[:, 1]
    auc_score = roc_auc_score(y_test, y_pred_proba)
    print(f"{name} AUC-ROC Score: {auc_score:.4f}")

Feature Selection and Engineering

Feature selection and engineering are crucial steps in building effective machine learning models. This implementation shows various techniques for selecting relevant features and creating new meaningful features.

from sklearn.feature_selection import SelectFromModel
from sklearn.decomposition import PCA
import pandas as pd

# Generate feature matrix
X_features = np.random.randn(1000, 50)
y_target = np.random.randint(0, 2, 1000)

# L1-based feature selection
l1_selector = SelectFromModel(Lasso(alpha=0.01))
X_selected = l1_selector.fit_transform(X_features, y_target)

# Principal Component Analysis for feature engineering
pca = PCA(n_components=0.95)
X_pca = pca.fit_transform(X_features)

# Polynomial feature engineering
def create_polynomial_features(X, degree=2):
    n_features = X.shape[1]
    feature_names = [f'x{i}' for i in range(n_features)]
    poly_features = []
    
    for i in range(n_features):
        for j in range(i, n_features):
            poly_features.append(X[:, i] * X[:, j])
    
    return np.column_stack([X, np.array(poly_features)])

X_poly = create_polynomial_features(X_features)
print(f"Original features: {X_features.shape[1]}")
print(f"Selected features: {X_selected.shape[1]}")
print(f"PCA features: {X_pca.shape[1]}")
print(f"Polynomial features: {X_poly.shape[1]}")

Cross-Validation and Model Evaluation

Robust model evaluation requires proper cross-validation techniques and appropriate metrics. This implementation demonstrates various validation strategies and metric calculations.

from sklearn.model_selection import KFold, cross_val_score
from sklearn.metrics import make_scorer, f1_score, precision_score, recall_score

# Create custom scorer
scoring = {
    'f1': make_scorer(f1_score),
    'precision': make_scorer(precision_score),
    'recall': make_scorer(recall_score)
}

# Initialize cross-validation
kfold = KFold(n_splits=5, shuffle=True, random_state=42)

# Model evaluation with multiple metrics
def evaluate_model(model, X, y, cv=kfold, scoring=scoring):
    results = {}
    for metric_name, scorer in scoring.items():
        scores = cross_val_score(model, X, y, scoring=scorer, cv=cv)
        results[metric_name] = {
            'mean': scores.mean(),
            'std': scores.std()
        }
    return results

# Evaluate multiple models
models = {
    'RandomForest': RandomForestClassifier(random_state=42),
    'GradientBoosting': GradientBoostingClassifier(random_state=42)
}

for name, model in models.items():
    results = evaluate_model(model, X_train_scaled, y_train)
    print(f"\nResults for {name}:")
    for metric, scores in results.items():
        print(f"{metric}: {scores['mean']:.4f} ± {scores['std']:.4f}")

Real-world Example: Customer Churn Prediction

This complete implementation demonstrates a production-ready customer churn prediction system, including feature preprocessing, model pipeline creation, and deployment-ready prediction functions.

from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
import pandas as pd
import numpy as np

# Simulate customer data
np.random.seed(42)
n_customers = 1000
customer_data = {
    'usage_months': np.random.randint(1, 100, n_customers),
    'monthly_bill': np.random.uniform(20, 200, n_customers),
    'support_calls': np.random.poisson(2, n_customers),
    'service_plan': np.random.choice(['basic', 'premium', 'elite'], n_customers),
    'churn': np.random.binomial(1, 0.3, n_customers)
}
df = pd.DataFrame(customer_data)

# Define feature types
numeric_features = ['usage_months', 'monthly_bill', 'support_calls']
categorical_features = ['service_plan']

# Create preprocessing pipeline
preprocessor = ColumnTransformer(
    transformers=[
        ('num', StandardScaler(), numeric_features),
        ('cat', OneHotEncoder(drop='first'), categorical_features)
    ])

# Create model pipeline
model_pipeline = Pipeline([
    ('preprocessor', preprocessor),
    ('classifier', GradientBoostingClassifier(
        n_estimators=100,
        learning_rate=0.1,
        max_depth=3,
        random_state=42
    ))
])

# Train and evaluate
X = df.drop('churn', axis=1)
y = df['churn']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

model_pipeline.fit(X_train, y_train)
y_pred = model_pipeline.predict(X_test)
y_prob = model_pipeline.predict_proba(X_test)[:, 1]

print(f"Model Performance:")
print(classification_report(y_test, y_pred))
print(f"ROC-AUC Score: {roc_auc_score(y_test, y_prob):.4f}")

Advanced Model Interpretability

Model interpretability is crucial for real-world applications. This implementation shows how to explain model predictions using SHAP values and feature importance analysis.

import shap
import matplotlib.pyplot as plt
from pdpbox import pdp

# Train a more interpretable model
rf_model = RandomForestClassifier(n_estimators=100, random_state=42)
rf_model.fit(X_train_scaled, y_train)

# Calculate feature importances
importances = pd.DataFrame({
    'feature': X_train.columns,
    'importance': rf_model.feature_importances_
}).sort_values('importance', ascending=False)

# SHAP values calculation
explainer = shap.TreeExplainer(rf_model)
shap_values = explainer.shap_values(X_test_scaled)

# Partial Dependence Plot calculation
def plot_partial_dependence(model, X, feature_idx):
    pdp_isolate = pdp.pdp_isolate(
        model=model,
        dataset=X,
        model_features=X.columns,
        feature=feature_idx
    )
    return pdp_isolate

# Print interpretability results
print("Top 5 Important Features:")
print(importances.head())

# Calculate global feature impact
feature_impact = np.abs(shap_values).mean(0)
print("\nGlobal Feature Impact:")
for idx, impact in enumerate(feature_impact):
    print(f"{X_train.columns[idx]}: {impact:.4f}")

Additional Resources

More in DataFrames and pandas

  1. Detecting and Mitigating Model Drift9 min

    Model drift is a crucial concept in machine learning, referring to the degradation of a model's performance over time. This occurs when the statistical...

  2. Bidirectional RNNs Dual-Directional Sequence Processing14 min

    Bidirectional Recurrent Neural Networks (Bi-RNNs) are a powerful variant of traditional RNNs that process sequences in both forward and backward directions....

  3. Discriminative vs. Generative Models A Primer15 min

    Discriminative models directly learn the mapping between input features and output labels by modeling conditional probability P(Y|X). They focus on decision...

  4. Algorithms for Non-Linear Problems9 min

    Linear regression is the most basic and commonly used algorithm that tends to underfit when dealing with non-linear problems. It assumes a linear...

  5. Efficient Fine-Tuning with LoRA9 min

    Low-Rank Adaptation (LoRA) is a technique for fine-tuning large language models more efficiently. It allows for task-specific adaptations without modifying...

  6. Adam Optimizer Exponentially Weighted Gradients and Squared Gradients11 min

    Adam (Adaptive Moment Estimation) optimizer combines the advantages of two other optimization algorithms: RMSprop and momentum, maintaining exponentially...