xbe.at

Deep learning / Fundamentals and architectures

Enhancing Predictive Performance with Fibonacci Neural Networks and XGBoost

10 min read, 15 sections

Contents
  1. Introduction to Fibonacci Neural Networks and XGBoost Stacking
  2. Fibonacci Neural Networks: Concept and Structure
  3. Implementing Fibonacci Neural Networks in Python
  4. XGBoost: An Overview
  5. XGBoost Hyperparameter Tuning
  6. Stacking: Concept and Implementation
  7. Combining Fibonacci Neural Networks and XGBoost Stacking
  8. Evaluating the Combined Model
  9. Real-Life Example: Weather Prediction
  10. Real-Life Example: Image Classification
  11. Combining FibonacciCNN and XGBoost for MNIST
  12. Visualizing MNIST Predictions
  13. Performance Comparison
  14. Conclusion and Future Directions
  15. Additional Resources

Introduction to Fibonacci Neural Networks and XGBoost Stacking

Fibonacci Neural Networks and XGBoost Stacking are advanced techniques in machine learning that can significantly enhance predictive performance. This presentation will explore these concepts, their implementation in Python, and how they can be combined to create powerful predictive models.

import numpy as np
import matplotlib.pyplot as plt

def fibonacci(n):
    if n <= 1:
        return n
    else:
        return fibonacci(n-1) + fibonacci(n-2)

fib_sequence = [fibonacci(i) for i in range(10)]
plt.plot(fib_sequence)
plt.title("Fibonacci Sequence")
plt.xlabel("Index")
plt.ylabel("Value")
plt.show()

Fibonacci Neural Networks: Concept and Structure

Fibonacci Neural Networks are a novel architecture inspired by the Fibonacci sequence. They leverage the golden ratio inherent in the sequence to create a unique network structure that can potentially capture complex patterns in data more effectively than traditional neural networks.

import torch
import torch.nn as nn

class FibonacciLayer(nn.Module):
    def __init__(self, input_size, output_size):
        super(FibonacciLayer, self).__init__()
        self.linear = nn.Linear(input_size, output_size)
        self.activation = nn.ReLU()

    def forward(self, x):
        return self.activation(self.linear(x))

class FibonacciNN(nn.Module):
    def __init__(self, input_size, hidden_sizes, output_size):
        super(FibonacciNN, self).__init__()
        self.layers = nn.ModuleList()
        
        # Create Fibonacci sequence of layer sizes
        sizes = [input_size] + hidden_sizes + [output_size]
        for i in range(len(sizes) - 1):
            self.layers.append(FibonacciLayer(sizes[i], sizes[i+1]))

    def forward(self, x):
        for layer in self.layers:
            x = layer(x)
        return x

# Example usage
model = FibonacciNN(10, [13, 21, 34], 1)
print(model)

Implementing Fibonacci Neural Networks in Python

Let's implement a basic Fibonacci Neural Network using PyTorch. We'll create a custom layer that follows the Fibonacci sequence in its structure.

import torch
import torch.nn as nn
import torch.optim as optim

# Assuming FibonacciNN class is defined as in the previous slide

# Generate some dummy data
X = torch.randn(100, 10)
y = torch.randn(100, 1)

# Create and train the model
model = FibonacciNN(10, [13, 21, 34], 1)
criterion = nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr=0.01)

for epoch in range(100):
    optimizer.zero_grad()
    outputs = model(X)
    loss = criterion(outputs, y)
    loss.backward()
    optimizer.step()
    
    if epoch % 10 == 0:
        print(f'Epoch [{epoch}/100], Loss: {loss.item():.4f}')

# Make predictions
with torch.no_grad():
    predictions = model(X)
    print("Sample predictions:", predictions[:5].numpy())

XGBoost: An Overview

XGBoost (eXtreme Gradient Boosting) is a powerful machine learning algorithm known for its speed and performance. It's an implementation of gradient boosted decision trees designed for speed and performance.

import xgboost as xgb
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split

# Generate sample data
X, y = make_regression(n_samples=1000, n_features=20, noise=0.1)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

# Create DMatrix for XGBoost
dtrain = xgb.DMatrix(X_train, label=y_train)
dtest = xgb.DMatrix(X_test, label=y_test)

# Set parameters
params = {
    'max_depth': 3,
    'eta': 0.1,
    'objective': 'reg:squarederror',
    'eval_metric': 'rmse'
}

# Train the model
num_rounds = 100
model = xgb.train(params, dtrain, num_rounds)

# Make predictions
preds = model.predict(dtest)
print("Sample predictions:", preds[:5])

XGBoost Hyperparameter Tuning

Tuning XGBoost hyperparameters is crucial for optimal performance. Let's use GridSearchCV to find the best parameters for our XGBoost model.

from sklearn.model_selection import GridSearchCV
from xgboost import XGBRegressor

# Define the parameter grid
param_grid = {
    'max_depth': [3, 5, 7],
    'learning_rate': [0.01, 0.1, 0.3],
    'n_estimators': [100, 200, 300],
    'subsample': [0.8, 1.0]
}

# Create the XGBoost model
xgb_model = XGBRegressor(objective='reg:squarederror')

# Perform grid search
grid_search = GridSearchCV(estimator=xgb_model, param_grid=param_grid, 
                           cv=3, scoring='neg_mean_squared_error')
grid_search.fit(X_train, y_train)

# Print the best parameters
print("Best parameters:", grid_search.best_params_)
print("Best score:", -grid_search.best_score_)

# Use the best model for predictions
best_model = grid_search.best_estimator_
preds = best_model.predict(X_test)
print("Sample predictions with best model:", preds[:5])

Stacking: Concept and Implementation

Stacking is an ensemble learning technique that combines multiple models to improve predictive performance. Let's implement a basic stacking model using scikit-learn.

from sklearn.ensemble import RandomForestRegressor
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import cross_val_predict

# Assuming X_train, y_train, X_test are defined

# Base models
rf_model = RandomForestRegressor(n_estimators=100, random_state=42)
xgb_model = XGBRegressor(n_estimators=100, random_state=42)

# Make predictions using cross-validation
rf_preds = cross_val_predict(rf_model, X_train, y_train, cv=5)
xgb_preds = cross_val_predict(xgb_model, X_train, y_train, cv=5)

# Train the base models on the full training data
rf_model.fit(X_train, y_train)
xgb_model.fit(X_train, y_train)

# Create a new feature matrix with base model predictions
X_train_stack = np.column_stack((rf_preds, xgb_preds))
X_test_stack = np.column_stack((rf_model.predict(X_test), xgb_model.predict(X_test)))

# Train a meta-model
meta_model = LinearRegression()
meta_model.fit(X_train_stack, y_train)

# Make final predictions
final_preds = meta_model.predict(X_test_stack)
print("Sample stacked predictions:", final_preds[:5])

Combining Fibonacci Neural Networks and XGBoost Stacking

Now, let's combine our Fibonacci Neural Network with XGBoost in a stacking ensemble to create a powerful predictive model.

import torch
import torch.nn as nn
import xgboost as xgb
from sklearn.model_selection import cross_val_predict
from sklearn.linear_model import LinearRegression

# Assuming FibonacciNN, X_train, y_train, X_test are defined

# Create and train Fibonacci NN
fib_nn = FibonacciNN(X_train.shape[1], [13, 21, 34], 1)
criterion = nn.MSELoss()
optimizer = optim.Adam(fib_nn.parameters(), lr=0.01)

for epoch in range(100):
    optimizer.zero_grad()
    outputs = fib_nn(torch.FloatTensor(X_train))
    loss = criterion(outputs, torch.FloatTensor(y_train.reshape(-1, 1)))
    loss.backward()
    optimizer.step()

# XGBoost model
xgb_model = xgb.XGBRegressor(n_estimators=100, random_state=42)

# Make predictions using cross-validation
fib_preds = cross_val_predict(lambda X, y: fib_nn(torch.FloatTensor(X)).detach().numpy(), X_train, y_train, cv=5)
xgb_preds = cross_val_predict(xgb_model, X_train, y_train, cv=5)

# Train base models on full data
fib_nn.fit(torch.FloatTensor(X_train), torch.FloatTensor(y_train.reshape(-1, 1)))
xgb_model.fit(X_train, y_train)

# Create stacked feature matrix
X_train_stack = np.column_stack((fib_preds, xgb_preds))
X_test_stack = np.column_stack((fib_nn(torch.FloatTensor(X_test)).detach().numpy(), xgb_model.predict(X_test)))

# Train meta-model
meta_model = LinearRegression()
meta_model.fit(X_train_stack, y_train)

# Make final predictions
final_preds = meta_model.predict(X_test_stack)
print("Sample combined predictions:", final_preds[:5])

Evaluating the Combined Model

Let's evaluate our combined Fibonacci Neural Network and XGBoost stacking model and compare it with individual models.

from sklearn.metrics import mean_squared_error, r2_score

# Individual model predictions
fib_preds = fib_nn(torch.FloatTensor(X_test)).detach().numpy()
xgb_preds = xgb_model.predict(X_test)

# Calculate MSE and R2 for each model
models = {
    "Fibonacci NN": fib_preds,
    "XGBoost": xgb_preds,
    "Combined Model": final_preds
}

for name, preds in models.items():
    mse = mean_squared_error(y_test, preds)
    r2 = r2_score(y_test, preds)
    print(f"{name} - MSE: {mse:.4f}, R2: {r2:.4f}")

# Visualize predictions
plt.figure(figsize=(10, 6))
plt.scatter(y_test, final_preds, alpha=0.5)
plt.plot([y_test.min(), y_test.max()], [y_test.min(), y_test.max()], 'r--', lw=2)
plt.xlabel("Actual Values")
plt.ylabel("Predicted Values")
plt.title("Combined Model: Actual vs Predicted")
plt.show()

Real-Life Example: Weather Prediction

Let's apply our combined model to predict daily maximum temperatures using historical weather data.

import pandas as pd
from sklearn.preprocessing import StandardScaler

# Load sample weather data (you would need to replace this with real data)
data = pd.DataFrame({
    'date': pd.date_range(start='1/1/2020', end='12/31/2020'),
    'temp_max': np.random.normal(25, 5, 366),
    'humidity': np.random.normal(60, 10, 366),
    'wind_speed': np.random.normal(10, 3, 366),
    'pressure': np.random.normal(1013, 5, 366)
})

# Prepare features and target
X = data[['humidity', 'wind_speed', 'pressure']]
y = data['temp_max']

# Scale features
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# Split data
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.2, random_state=42)

# Train combined model (assuming previous model definitions)
# ... (training code here)

# Make predictions
predictions = meta_model.predict(X_test_stack)

# Visualize results
plt.figure(figsize=(10, 6))
plt.plot(y_test.values, label='Actual')
plt.plot(predictions, label='Predicted')
plt.xlabel('Sample')
plt.ylabel('Maximum Temperature (°C)')
plt.title('Weather Prediction: Actual vs Predicted Max Temperature')
plt.legend()
plt.show()

print(f"Model MSE: {mean_squared_error(y_test, predictions):.2f}")
print(f"Model R2 Score: {r2_score(y_test, predictions):.2f}")

Real-Life Example: Image Classification

Let's apply our combined model to an image classification task, specifically identifying handwritten digits from the MNIST dataset.

import torch
import torch.nn as nn
import torch.optim as optim
import torchvision
from torchvision import datasets, transforms
from torch.utils.data import DataLoader

# Load MNIST dataset
transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))])
mnist_train = datasets.MNIST(root='./data', train=True, download=True, transform=transform)
mnist_test = datasets.MNIST(root='./data', train=False, download=True, transform=transform)

train_loader = DataLoader(mnist_train, batch_size=64, shuffle=True)
test_loader = DataLoader(mnist_test, batch_size=1000, shuffle=False)

# Define FibonacciCNN
class FibonacciCNN(nn.Module):
    def __init__(self):
        super(FibonacciCNN, self).__init__()
        self.conv1 = nn.Conv2d(1, 13, kernel_size=3)
        self.conv2 = nn.Conv2d(13, 21, kernel_size=3)
        self.fc1 = nn.Linear(21 * 5 * 5, 34)
        self.fc2 = nn.Linear(34, 10)

    def forward(self, x):
        x = torch.relu(torch.max_pool2d(self.conv1(x), 2))
        x = torch.relu(torch.max_pool2d(self.conv2(x), 2))
        x = x.view(-1, 21 * 5 * 5)
        x = torch.relu(self.fc1(x))
        x = self.fc2(x)
        return torch.log_softmax(x, dim=1)

# Train FibonacciCNN
fib_cnn = FibonacciCNN()
optimizer = optim.Adam(fib_cnn.parameters())
criterion = nn.CrossEntropyLoss()

for epoch in range(5):
    for batch_idx, (data, target) in enumerate(train_loader):
        optimizer.zero_grad()
        output = fib_cnn(data)
        loss = criterion(output, target)
        loss.backward()
        optimizer.step()

# Evaluate FibonacciCNN
fib_cnn.eval()
correct = 0
with torch.no_grad():
    for data, target in test_loader:
        output = fib_cnn(data)
        pred = output.argmax(dim=1, keepdim=True)
        correct += pred.eq(target.view_as(pred)).sum().item()

print(f'FibonacciCNN Accuracy: {correct / len(test_loader.dataset):.2f}')

Combining FibonacciCNN and XGBoost for MNIST

Now, let's combine our FibonacciCNN with XGBoost for the MNIST classification task.

import numpy as np
from xgboost import XGBClassifier
from sklearn.metrics import accuracy_score

# Extract features from FibonacciCNN
def extract_features(model, loader):
    features = []
    labels = []
    model.eval()
    with torch.no_grad():
        for data, target in loader:
            output = model.fc1(model.conv2(model.conv1(data)).view(-1, 21 * 5 * 5))
            features.append(output.numpy())
            labels.append(target.numpy())
    return np.concatenate(features), np.concatenate(labels)

# Extract features
X_train, y_train = extract_features(fib_cnn, train_loader)
X_test, y_test = extract_features(fib_cnn, test_loader)

# Train XGBoost
xgb_model = XGBClassifier(n_estimators=100, learning_rate=0.1)
xgb_model.fit(X_train, y_train)

# Combine predictions
fib_preds = fib_cnn(next(iter(test_loader))[0]).argmax(dim=1).numpy()
xgb_preds = xgb_model.predict(X_test)

combined_preds = (fib_preds + xgb_preds) // 2

# Evaluate combined model
combined_accuracy = accuracy_score(y_test, combined_preds)
print(f'Combined Model Accuracy: {combined_accuracy:.2f}')

Visualizing MNIST Predictions

Let's visualize some predictions from our combined model on the MNIST dataset.

import matplotlib.pyplot as plt

# Get a batch of test images
test_images, test_labels = next(iter(test_loader))

# Make predictions
fib_preds = fib_cnn(test_images).argmax(dim=1).numpy()
xgb_preds = xgb_model.predict(test_images.view(test_images.shape[0], -1).numpy())
combined_preds = (fib_preds + xgb_preds) // 2

# Plot images and predictions
fig, axes = plt.subplots(3, 3, figsize=(12, 12))
for i, ax in enumerate(axes.flat):
    ax.imshow(test_images[i][0], cmap='gray')
    ax.set_title(f"True: {test_labels[i]}, Pred: {combined_preds[i]}")
    ax.axis('off')

plt.tight_layout()
plt.show()

Performance Comparison

Let's compare the performance of our individual models (FibonacciCNN and XGBoost) with the combined model on the MNIST dataset.

from sklearn.metrics import accuracy_score, confusion_matrix
import seaborn as sns

# Evaluate individual models
fib_accuracy = accuracy_score(y_test, fib_preds)
xgb_accuracy = accuracy_score(y_test, xgb_preds)
combined_accuracy = accuracy_score(y_test, combined_preds)

print(f"FibonacciCNN Accuracy: {fib_accuracy:.4f}")
print(f"XGBoost Accuracy: {xgb_accuracy:.4f}")
print(f"Combined Model Accuracy: {combined_accuracy:.4f}")

# Plot confusion matrix for combined model
cm = confusion_matrix(y_test, combined_preds)
plt.figure(figsize=(10, 8))
sns.heatmap(cm, annot=True, fmt='d', cmap='Blues')
plt.title('Confusion Matrix - Combined Model')
plt.xlabel('Predicted Label')
plt.ylabel('True Label')
plt.show()

Conclusion and Future Directions

In this presentation, we explored the combination of Fibonacci Neural Networks and XGBoost Stacking to enhance predictive performance. We implemented these techniques in Python and applied them to real-world problems like weather prediction and image classification.

Key takeaways:

  1. Fibonacci Neural Networks offer a unique architecture inspired by the Fibonacci sequence.
  2. XGBoost is a powerful gradient boosting algorithm known for its performance.
  3. Stacking these models can lead to improved predictive accuracy.
  4. The combined approach shows promise in various domains, from regression to classification tasks.

Future directions:

  • Explore other variations of Fibonacci-inspired neural architectures.
  • Investigate the impact of different meta-learners in the stacking process.
  • Apply the combined model to more complex datasets and problem domains.
  • Optimize the model for specific use cases and performance requirements.

Additional Resources

For those interested in diving deeper into these topics, here are some valuable resources:

  1. "XGBoost: A Scalable Tree Boosting System" by Chen and Guestrin (2016) ArXiv: https://arxiv.org/abs/1603.02754
  2. "Fibonacci Networks for Machine Learning" by Li et al. (2021) ArXiv: https://arxiv.org/abs/2111.09599
  3. "A Survey of Deep Learning Techniques for Neural Machine Translation" by Stahlberg (2020) ArXiv: https://arxiv.org/abs/1905.13091

These papers provide in-depth discussions on the techniques we've explored and their applications in various machine learning tasks.

More in Fundamentals and architectures

  1. Recommender Systems Evaluation Metrics and Loss Functions in Python9 min

    Recommender systems are algorithms designed to suggest relevant items to users based on their preferences and behavior. These systems are widely used in...

  2. The Importance of Data in Deep Learning18 min

    Data quality assessment is fundamental in deep learning pipelines. This implementation demonstrates how to evaluate dataset characteristics including missing...