Neural Hierarchical Interpolation for Time Series Forecasting Using Python

Title: Introduction to Neural Hierarchical Interpolation for Time Series

Neural Hierarchical Interpolation is a powerful technique for modeling and forecasting complex time series data. It combines the strengths of neural networks with hierarchical structures to capture both short-term and long-term patterns in temporal data. This approach is particularly useful for handling irregularly sampled or missing data points in time series.

import numpy as np
import tensorflow as tf
from tensorflow import keras

# Example of a simple time series
time_series = np.array([1, 2, 4, 7, 11, 16, 22, 29, 37, 46])

# Reshape for neural network input
X = time_series[:-1].reshape(-1, 1)
y = time_series[1:].reshape(-1, 1)

# Create a basic neural network model
model = keras.Sequential([
    keras.layers.Dense(64, activation='relu', input_shape=(1,)),
    keras.layers.Dense(32, activation='relu'),
    keras.layers.Dense(1)
])

model.compile(optimizer='adam', loss='mse')
model.fit(X, y, epochs=100, verbose=0)

Title: Hierarchical Structure in Time Series

Hierarchical structures in time series data represent different levels of temporal granularity. For example, in a sales dataset, we might have daily, weekly, monthly, and yearly patterns. Neural Hierarchical Interpolation leverages these structures to improve prediction accuracy and capture complex relationships across different time scales.

import pandas as pd

# Create a sample hierarchical time series
dates = pd.date_range(start='2023-01-01', end='2023-12-31', freq='D')
daily_sales = np.random.randint(50, 200, size=len(dates))
df = pd.DataFrame({'date': dates, 'sales': daily_sales})

# Add hierarchical features
df['day_of_week'] = df['date'].dt.dayofweek
df['week_of_year'] = df['date'].dt.isocalendar().week
df['month'] = df['date'].dt.month

print(df.head())

Title: Data Preprocessing for Neural Hierarchical Interpolation

Before applying Neural Hierarchical Interpolation, it's crucial to preprocess the time series data. This involves handling missing values, normalizing the data, and creating appropriate input-output pairs for training the neural network.

from sklearn.preprocessing import MinMaxScaler

# Normalize the sales data
scaler = MinMaxScaler()
df['normalized_sales'] = scaler.fit_transform(df[['sales']])

# Create lagged features
for i in range(1, 8):
    df[f'sales_lag_{i}'] = df['normalized_sales'].shift(i)

# Drop rows with NaN values
df_clean = df.dropna().reset_index(drop=True)

# Prepare input features and target variable
X = df_clean[['sales_lag_1', 'sales_lag_2', 'sales_lag_3', 'sales_lag_4', 'sales_lag_5', 'sales_lag_6', 'sales_lag_7', 'day_of_week', 'week_of_year', 'month']]
y = df_clean['normalized_sales']

print(X.head())
print(y.head())

Title: Building a Neural Network for Hierarchical Interpolation

The core of Neural Hierarchical Interpolation is a neural network architecture designed to capture hierarchical patterns. This typically involves using multiple layers with different activation functions to model complex relationships at various time scales.

from tensorflow.keras.layers import Input, Dense, Concatenate
from tensorflow.keras.models import Model

# Define input layers for different hierarchical levels
daily_input = Input(shape=(7,), name='daily_input')
weekly_input = Input(shape=(1,), name='weekly_input')
monthly_input = Input(shape=(1,), name='monthly_input')

# Process each hierarchical level
daily_dense = Dense(32, activation='relu')(daily_input)
weekly_dense = Dense(16, activation='relu')(weekly_input)
monthly_dense = Dense(8, activation='relu')(monthly_input)

# Combine hierarchical features
combined = Concatenate()([daily_dense, weekly_dense, monthly_dense])

# Output layer
output = Dense(1, activation='linear')(combined)

# Create and compile the model
model = Model(inputs=[daily_input, weekly_input, monthly_input], outputs=output)
model.compile(optimizer='adam', loss='mse')

print(model.summary())

Title: Training the Neural Hierarchical Interpolation Model

Training the model involves feeding it with preprocessed data and adjusting its parameters to minimize the prediction error. It's important to use appropriate loss functions and optimization algorithms for time series forecasting.

# Prepare input data for the model
X_daily = X[['sales_lag_1', 'sales_lag_2', 'sales_lag_3', 'sales_lag_4', 'sales_lag_5', 'sales_lag_6', 'sales_lag_7']].values
X_weekly = X[['week_of_year']].values
X_monthly = X[['month']].values

# Split the data into training and testing sets
train_size = int(0.8 * len(X))
X_daily_train, X_daily_test = X_daily[:train_size], X_daily[train_size:]
X_weekly_train, X_weekly_test = X_weekly[:train_size], X_weekly[train_size:]
X_monthly_train, X_monthly_test = X_monthly[:train_size], X_monthly[train_size:]
y_train, y_test = y[:train_size], y[train_size:]

# Train the model
history = model.fit(
    [X_daily_train, X_weekly_train, X_monthly_train],
    y_train,
    epochs=100,
    batch_size=32,
    validation_split=0.2,
    verbose=0
)

# Plot training history
import matplotlib.pyplot as plt
plt.plot(history.history['loss'], label='Training Loss')
plt.plot(history.history['val_loss'], label='Validation Loss')
plt.title('Model Training History')
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.legend()
plt.show()

Title: Handling Irregular Time Series Data

One of the strengths of Neural Hierarchical Interpolation is its ability to handle irregularly sampled or missing data points. This is achieved by incorporating time information into the model and using appropriate interpolation techniques.

import numpy as np
from scipy.interpolate import interp1d

# Create an irregular time series
timestamps = np.sort(np.random.choice(range(100), size=50, replace=False))
values = np.sin(timestamps * 0.1) + np.random.normal(0, 0.1, size=50)

# Interpolate missing values
regular_timestamps = np.arange(100)
interpolator = interp1d(timestamps, values, kind='linear', fill_value='extrapolate')
interpolated_values = interpolator(regular_timestamps)

# Plot original and interpolated data
plt.figure(figsize=(12, 6))
plt.scatter(timestamps, values, label='Original Data')
plt.plot(regular_timestamps, interpolated_values, label='Interpolated Data')
plt.title('Irregular Time Series Interpolation')
plt.xlabel('Time')
plt.ylabel('Value')
plt.legend()
plt.show()

Title: Attention Mechanisms in Neural Hierarchical Interpolation

Attention mechanisms can significantly enhance the performance of Neural Hierarchical Interpolation models by allowing the network to focus on the most relevant parts of the input sequence when making predictions.

from tensorflow.keras.layers import MultiHeadAttention, LayerNormalization

def attention_block(x, num_heads=4):
    attention_output = MultiHeadAttention(num_heads=num_heads, key_dim=32)(x, x)
    x = LayerNormalization(epsilon=1e-6)(x + attention_output)
    return x

# Modify the previous model to include attention
daily_input = Input(shape=(7,), name='daily_input')
weekly_input = Input(shape=(1,), name='weekly_input')
monthly_input = Input(shape=(1,), name='monthly_input')

daily_dense = Dense(32, activation='relu')(daily_input)
daily_attention = attention_block(daily_dense)

weekly_dense = Dense(16, activation='relu')(weekly_input)
monthly_dense = Dense(8, activation='relu')(monthly_input)

combined = Concatenate()([daily_attention, weekly_dense, monthly_dense])
output = Dense(1, activation='linear')(combined)

attention_model = Model(inputs=[daily_input, weekly_input, monthly_input], outputs=output)
attention_model.compile(optimizer='adam', loss='mse')

print(attention_model.summary())

Title: Evaluating Neural Hierarchical Interpolation Models

Proper evaluation of Neural Hierarchical Interpolation models is crucial to assess their performance and compare them with other forecasting techniques. Common metrics include Mean Squared Error (MSE), Mean Absolute Error (MAE), and Root Mean Squared Error (RMSE).

from sklearn.metrics import mean_squared_error, mean_absolute_error
import numpy as np

# Make predictions using the trained model
y_pred = model.predict([X_daily_test, X_weekly_test, X_monthly_test])

# Calculate evaluation metrics
mse = mean_squared_error(y_test, y_pred)
mae = mean_absolute_error(y_test, y_pred)
rmse = np.sqrt(mse)

print(f"Mean Squared Error: {mse:.4f}")
print(f"Mean Absolute Error: {mae:.4f}")
print(f"Root Mean Squared Error: {rmse:.4f}")

# Plot actual vs predicted values
plt.figure(figsize=(12, 6))
plt.plot(y_test, label='Actual')
plt.plot(y_pred, label='Predicted')
plt.title('Actual vs Predicted Values')
plt.xlabel('Time')
plt.ylabel('Normalized Sales')
plt.legend()
plt.show()

Title: Handling Multiple Seasonalities

Many real-world time series exhibit multiple seasonalities, such as daily, weekly, and yearly patterns. Neural Hierarchical Interpolation can be adapted to capture these complex seasonal structures effectively.

import statsmodels.api as sm

# Generate a time series with multiple seasonalities
dates = pd.date_range(start='2020-01-01', end='2023-12-31', freq='H')
hourly_data = pd.Series(
    np.random.randn(len(dates)) +
    5 * np.sin(2 * np.pi * dates.hour / 24) +  # Daily seasonality
    3 * np.sin(2 * np.pi * dates.dayofweek / 7) +  # Weekly seasonality
    2 * np.sin(2 * np.pi * dates.dayofyear / 365),  # Yearly seasonality
    index=dates
)

# Decompose the time series
decomposition = sm.tsa.seasonal_decompose(hourly_data, model='additive', period=24*7)

# Plot the decomposition
fig, (ax1, ax2, ax3, ax4) = plt.subplots(4, 1, figsize=(12, 16))
decomposition.observed.plot(ax=ax1)
ax1.set_title('Observed')
decomposition.trend.plot(ax=ax2)
ax2.set_title('Trend')
decomposition.seasonal.plot(ax=ax3)
ax3.set_title('Seasonality')
decomposition.resid.plot(ax=ax4)
ax4.set_title('Residuals')
plt.tight_layout()
plt.show()

Title: Incorporating External Factors

Neural Hierarchical Interpolation can be extended to incorporate external factors that may influence the time series. This can improve forecasting accuracy by considering relevant contextual information.

import pandas as pd
import numpy as np
from sklearn.preprocessing import StandardScaler

# Create a sample dataset with external factors
dates = pd.date_range(start='2023-01-01', end='2023-12-31', freq='D')
np.random.seed(42)
df = pd.DataFrame({
    'date': dates,
    'sales': np.random.randint(100, 1000, size=len(dates)),
    'temperature': np.random.uniform(0, 30, size=len(dates)),
    'is_holiday': np.random.choice([0, 1], size=len(dates), p=[0.9, 0.1])
})

# Preprocess the data
scaler = StandardScaler()
df['scaled_sales'] = scaler.fit_transform(df[['sales']])
df['scaled_temperature'] = scaler.fit_transform(df[['temperature']])

# Create lagged features and external factors
for i in range(1, 8):
    df[f'sales_lag_{i}'] = df['scaled_sales'].shift(i)

X = df.dropna().reset_index(drop=True)
y = X['scaled_sales']
X = X[['sales_lag_1', 'sales_lag_2', 'sales_lag_3', 'sales_lag_4', 'sales_lag_5', 'sales_lag_6', 'sales_lag_7', 'scaled_temperature', 'is_holiday']]

# Build a model with external factors
input_sales = Input(shape=(7,), name='sales_input')
input_external = Input(shape=(2,), name='external_input')

sales_dense = Dense(32, activation='relu')(input_sales)
external_dense = Dense(16, activation='relu')(input_external)

combined = Concatenate()([sales_dense, external_dense])
output = Dense(1, activation='linear')(combined)

model_with_external = Model(inputs=[input_sales, input_external], outputs=output)
model_with_external.compile(optimizer='adam', loss='mse')

print(model_with_external.summary())

Title: Handling Long-Term Dependencies

Long-term dependencies in time series can be challenging to capture. Neural Hierarchical Interpolation can be enhanced with techniques like LSTM (Long Short-Term Memory) or GRU (Gated Recurrent Unit) layers to better model these long-range patterns.

from tensorflow.keras.layers import LSTM, GRU

# Create a sample long sequence
sequence_length = 1000
time_steps = 100
features = 1

X = np.random.randn(sequence_length, time_steps, features)
y = np.sum(X[:, -5:, :], axis=1)  # Target is the sum of the last 5 time steps

# Build a model with LSTM for long-term dependencies
model_lstm = keras.Sequential([
    LSTM(64, return_sequences=True, input_shape=(time_steps, features)),
    LSTM(32),
    Dense(1)
])

model_lstm.compile(optimizer='adam', loss='mse')
print(model_lstm.summary())

# Train the model
history = model_lstm.fit(X, y, epochs=50, validation_split=0.2, verbose=0)

# Plot training history
plt.plot(history.history['loss'], label='Training Loss')
plt.plot(history.history['val_loss'], label='Validation Loss')
plt.title('LSTM Model Training History')
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.legend()
plt.show()

Title: Uncertainty Quantification in Neural Hierarchical Interpolation

Quantifying uncertainty in time series forecasts is crucial for making informed decisions. Neural Hierarchical Interpolation can be extended to provide probabilistic forecasts using techniques like Monte Carlo Dropout or Bayesian Neural Networks.

import tensorflow_probability as tfp

# Create a Bayesian Neural Network for time series forecasting
def create_bayesian_model(input_shape):
    model = keras.Sequential([
        tfp.layers.DenseVariational(64, activation='relu', input_shape=input_shape),
        tfp.layers.Dense

Response:

Certainly. I'll rewrite Slide 12 and continue with the remaining slides.

Title: Uncertainty Quantification in Neural Hierarchical Interpolation

Quantifying uncertainty in time series forecasts is crucial for making informed decisions. Neural Hierarchical Interpolation can be extended to provide probabilistic forecasts using techniques like Monte Carlo Dropout or Bayesian Neural Networks.

import tensorflow_probability as tfp

# Create a Bayesian Neural Network for time series forecasting
def create_bayesian_model(input_shape):
    model = keras.Sequential([
        tfp.layers.DenseVariational(64, activation='relu', input_shape=input_shape),
        tfp.layers.DenseVariational(32, activation='relu'),
        tfp.layers.DenseVariational(1)
    ])
    return model

# Prepare sample data
X = np.random.randn(1000, 10)
y = np.sum(X, axis=1) + np.random.randn(1000) * 0.1

# Create and compile the model
bayesian_model = create_bayesian_model((10,))
bayesian_model.compile(optimizer='adam', loss='mse')

# Train the model
history = bayesian_model.fit(X, y, epochs=100, validation_split=0.2, verbose=0)

# Make predictions with uncertainty
num_samples = 100
predictions = [bayesian_model(X) for _ in range(num_samples)]
mean_prediction = np.mean(predictions, axis=0)
std_prediction = np.std(predictions, axis=0)

# Plot results
plt.figure(figsize=(10, 6))
plt.scatter(y, mean_prediction, alpha=0.5)
plt.fill_between(y, mean_prediction - 2*std_prediction, 
                 mean_prediction + 2*std_prediction, alpha=0.2)
plt.xlabel('True Values')
plt.ylabel('Predicted Values')
plt.title('Bayesian Neural Network Predictions with Uncertainty')
plt.show()

Title: Transfer Learning in Neural Hierarchical Interpolation

Transfer learning can be a powerful technique in Neural Hierarchical Interpolation, especially when dealing with multiple related time series or limited data for a specific series. It allows leveraging knowledge from pre-trained models to improve performance on new tasks.

# Create a base model
def create_base_model(input_shape):
    model = keras.Sequential([
        Dense(64, activation='relu', input_shape=input_shape),
        Dense(32, activation='relu'),
        Dense(16, activation='relu'),
        Dense(1)
    ])
    model.compile(optimizer='adam', loss='mse')
    return model

# Generate sample data for two related time series
X1 = np.random.randn(1000, 10)
y1 = np.sum(X1, axis=1) + np.random.randn(1000) * 0.1

X2 = np.random.randn(200, 10)
y2 = np.sum(X2, axis=1) * 1.5 + np.random.randn(200) * 0.2

# Train the base model on the first time series
base_model = create_base_model((10,))
base_model.fit(X1, y1, epochs=100, verbose=0)

# Create a new model for transfer learning
transfer_model = keras.models.clone_model(base_model)
transfer_model.set_weights(base_model.get_weights())

# Freeze the first two layers
for layer in transfer_model.layers[:2]:
    layer.trainable = False

# Recompile and train on the second time series
transfer_model.compile(optimizer='adam', loss='mse')
history = transfer_model.fit(X2, y2, epochs=50, validation_split=0.2, verbose=0)

# Plot training history
plt.plot(history.history['loss'], label='Training Loss')
plt.plot(history.history['val_loss'], label='Validation Loss')
plt.title('Transfer Learning Model Training History')
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.legend()
plt.show()

Title: Explainable AI in Neural Hierarchical Interpolation

Explainable AI techniques can help interpret the decisions made by Neural Hierarchical Interpolation models. This is crucial for building trust in the model's predictions and understanding the key factors influencing the forecasts.

import shap

# Create a simple model for demonstration
model = keras.Sequential([
    Dense(64, activation='relu', input_shape=(10,)),
    Dense(32, activation='relu'),
    Dense(1)
])

model.compile(optimizer='adam', loss='mse')

# Generate sample data
X = np.random.randn(1000, 10)
y = np.sum(X, axis=1) + np.random.randn(1000) * 0.1

# Train the model
model.fit(X, y, epochs=100, verbose=0)

# Create a SHAP explainer
explainer = shap.DeepExplainer(model, X[:100])

# Calculate SHAP values for a set of instances
shap_values = explainer.shap_values(X[:10])

# Plot SHAP summary
shap.summary_plot(shap_values[0], X[:10], plot_type="bar")

Title: Conclusion and Future Directions

Neural Hierarchical Interpolation offers a powerful approach to modeling complex time series data. By combining hierarchical structures with neural networks, it can capture intricate patterns and relationships across different time scales. Future research directions include:

1. Integrating advanced attention mechanisms for better long-term dependency modeling
2. Developing hybrid models that combine Neural Hierarchical Interpolation with traditional statistical methods
3. Exploring the use of graph neural networks for capturing inter-series relationships in multivariate time series
4. Enhancing interpretability and explainability of the models for real-world applications

# Pseudo-code for a future research direction: Graph Neural Network for multivariate time series
class GraphNeuralNetwork(keras.Model):
    def __init__(self, num_series, hidden_dim):
        super().__init__()
        self.gnn_layer = GraphConvolution(hidden_dim)
        self.lstm = LSTM(hidden_dim)
        self.output_layer = Dense(1)

    def call(self, inputs, adj_matrix):
        x = self.gnn_layer(inputs, adj_matrix)
        x = self.lstm(x)
        return self.output_layer(x)

# Usage
num_series = 10
hidden_dim = 64
model = GraphNeuralNetwork(num_series, hidden_dim)

# Train the model
# model.fit(...)

# Make predictions
# predictions = model.predict(...)

Title: Additional Resources

For those interested in diving deeper into Neural Hierarchical Interpolation and related topics, here are some valuable resources:

1. "Temporal Pattern Attention for Multivariate Time Series Forecasting" by Shih et al. (2019) ArXiv: https://arxiv.org/abs/1809.04206
2. "DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks" by Salinas et al. (2020) ArXiv: https://arxiv.org/abs/1704.04110
3. "N-BEATS: Neural basis expansion analysis for interpretable time series forecasting" by Oreshkin et al. (2020) ArXiv: https://arxiv.org/abs/1905.10437
4. "Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting" by Zhou et al. (2021) ArXiv: https://arxiv.org/abs/2012.07436

These papers provide in-depth discussions on various aspects of neural network-based time series forecasting and can serve as excellent starting points for further research in the field of Neural Hierarchical Interpolation.