Instance-wise LoRA (iLoRA) for Sequential Recommendation

iLoRA is a novel fine-tuning framework designed to address the challenges of individual variability in user behaviors within sequential recommendation systems. It integrates the Mixture of Experts (MoE) concept into the basic Low-Rank Adaptation (LoRA) module, allowing for dynamic adjustment to diverse user behaviors.

import torch
from torch import nn

class iLoRA(nn.Module):
    def __init__(self, num_experts, d_model, r):
        super().__init__()
        self.num_experts = num_experts
        self.experts = nn.ModuleList([
            nn.Sequential(
                nn.Linear(d_model, r),
                nn.Linear(r, d_model)
            ) for _ in range(num_experts)
        ])
        self.gate = nn.Linear(d_model, num_experts)

    def forward(self, x, sequence_repr):
        expert_weights = torch.softmax(self.gate(sequence_repr), dim=-1)
        expert_outputs = [expert(x) for expert in self.experts]
        return sum(w * out for w, out in zip(expert_weights, expert_outputs))

Motivation Behind iLoRA

Traditional LoRA methods apply a uniform adaptation across all user sequences, potentially overlooking individual variability. iLoRA addresses this by treating each sequence as a separate task, mitigating negative transfer between disparate sequences.

import matplotlib.pyplot as plt
import numpy as np

def visualize_gradient_similarity(grad_sim_matrix):
    plt.figure(figsize=(10, 8))
    plt.imshow(grad_sim_matrix, cmap='viridis')
    plt.colorbar(label='Gradient Similarity')
    plt.title('Gradient Similarity Heatmap')
    plt.xlabel('Sequence Clusters')
    plt.ylabel('Sequence Clusters')
    plt.show()

# Example gradient similarity matrix
grad_sim = np.random.rand(8, 8)
np.fill_diagonal(grad_sim, 1)  # Perfect similarity on diagonal
visualize_gradient_similarity(grad_sim)

Key Components of iLoRA

Expert Array: Multiple specialized LoRA modules
Gating Network: Determines expert contributions
Sequence Representation: Guides the gating network

class ExpertArray(nn.Module):
    def __init__(self, num_experts, d_model, r):
        super().__init__()
        self.experts = nn.ModuleList([
            nn.Sequential(
                nn.Linear(d_model, r),
                nn.Linear(r, d_model)
            ) for _ in range(num_experts)
        ])

    def forward(self, x):
        return [expert(x) for expert in self.experts]

class GatingNetwork(nn.Module):
    def __init__(self, d_model, num_experts):
        super().__init__()
        self.gate = nn.Linear(d_model, num_experts)

    def forward(self, sequence_repr):
        return torch.softmax(self.gate(sequence_repr), dim=-1)

Sequence Representation

iLoRA uses a conventional recommender model (e.g., SASRec) to generate a holistic sequence representation, capturing user behavior patterns.

import torch.nn.functional as F

class SASRec(nn.Module):
    def __init__(self, num_items, d_model, num_heads, num_layers):
        super().__init__()
        self.item_embeddings = nn.Embedding(num_items, d_model)
        self.position_embeddings = nn.Embedding(1000, d_model)  # Assume max sequence length of 1000
        self.transformer_layers = nn.TransformerEncoderLayer(d_model, num_heads)
        self.transformer = nn.TransformerEncoder(self.transformer_layers, num_layers)

    def forward(self, sequence):
        positions = torch.arange(sequence.size(1), device=sequence.device).unsqueeze(0)
        embeddings = self.item_embeddings(sequence) + self.position_embeddings(positions)
        mask = torch.triu(torch.ones(sequence.size(1), sequence.size(1)), diagonal=1).bool()
        output = self.transformer(embeddings.transpose(0, 1), mask=mask)
        return output[-1]  # Return the last hidden state as the sequence representation

Gating Network

The gating network uses the sequence representation to generate customized attention scores for each expert, determining their participation in the final recommendation.

class GatingNetwork(nn.Module):
    def __init__(self, d_model, num_experts):
        super().__init__()
        self.gate = nn.Linear(d_model, num_experts)

    def forward(self, sequence_repr):
        logits = self.gate(sequence_repr)
        return F.softmax(logits, dim=-1)

# Example usage
d_model, num_experts = 64, 4
gating_network = GatingNetwork(d_model, num_experts)
sequence_repr = torch.randn(1, d_model)
expert_weights = gating_network(sequence_repr)
print("Expert weights:", expert_weights)

Instance-wise LoRA Aggregation

iLoRA combines the outputs of multiple experts based on the attention scores to create a personalized LoRA module for each sequence.

class iLoRA(nn.Module):
    def __init__(self, num_experts, d_model, r):
        super().__init__()
        self.expert_array = ExpertArray(num_experts, d_model, r)
        self.gating_network = GatingNetwork(d_model, num_experts)

    def forward(self, x, sequence_repr):
        expert_outputs = self.expert_array(x)
        expert_weights = self.gating_network(sequence_repr)
        return sum(w * out for w, out in zip(expert_weights, expert_outputs))

# Example usage
num_experts, d_model, r = 4, 64, 16
ilora = iLoRA(num_experts, d_model, r)
x = torch.randn(1, d_model)
sequence_repr = torch.randn(1, d_model)
output = ilora(x, sequence_repr)
print("iLoRA output shape:", output.shape)

Integration with Large Language Models

iLoRA can be integrated with Large Language Models (LLMs) for sequential recommendation tasks, enhancing their ability to capture individual user preferences.

from transformers import AutoModelForCausalLM, AutoTokenizer

class LLMWithiLoRA(nn.Module):
    def __init__(self, llm_name, num_experts, d_model, r):
        super().__init__()
        self.llm = AutoModelForCausalLM.from_pretrained(llm_name)
        self.tokenizer = AutoTokenizer.from_pretrained(llm_name)
        self.ilora = iLoRA(num_experts, d_model, r)

    def forward(self, input_ids, sequence_repr):
        llm_output = self.llm(input_ids).last_hidden_state
        ilora_output = self.ilora(llm_output, sequence_repr)
        return ilora_output

# Example usage
llm_name = "gpt2"
num_experts, d_model, r = 4, 768, 32
model = LLMWithiLoRA(llm_name, num_experts, d_model, r)
input_text = "Recommend a movie based on my history:"
input_ids = model.tokenizer(input_text, return_tensors="pt").input_ids
sequence_repr = torch.randn(1, d_model)
output = model(input_ids, sequence_repr)
print("LLM with iLoRA output shape:", output.shape)

Training Process

The training process for iLoRA involves fine-tuning the LLM with the personalized LoRA modules on a dataset of user sequences and next item predictions.

import torch.optim as optim

def train_ilora(model, train_dataloader, num_epochs, lr):
    optimizer = optim.Adam(model.parameters(), lr=lr)
    criterion = nn.CrossEntropyLoss()

    for epoch in range(num_epochs):
        for batch in train_dataloader:
            input_ids, sequence_repr, labels = batch
            outputs = model(input_ids, sequence_repr)
            loss = criterion(outputs.view(-1, outputs.size(-1)), labels.view(-1))
            
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

        print(f"Epoch {epoch+1}/{num_epochs}, Loss: {loss.item():.4f}")

# Example usage (assuming we have a DataLoader)
num_epochs, lr = 5, 1e-4
train_ilora(model, train_dataloader, num_epochs, lr)

Evaluation Metrics

iLoRA's performance can be evaluated using metrics such as HitRatio@K and NDCG@K, which measure the model's ability to recommend relevant items.

import numpy as np

def calculate_metrics(predictions, ground_truth, k=10):
    hits = 0
    ndcg = 0
    for pred, true in zip(predictions, ground_truth):
        if true in pred[:k]:
            hits += 1
            rank = pred.index(true) + 1
            ndcg += 1 / np.log2(rank + 1)
    
    hit_ratio = hits / len(ground_truth)
    ndcg = ndcg / len(ground_truth)
    return hit_ratio, ndcg

# Example usage
predictions = [[1, 3, 2, 4, 5], [2, 1, 3, 5, 4]]
ground_truth = [3, 1]
hit_ratio, ndcg = calculate_metrics(predictions, ground_truth, k=3)
print(f"HitRatio@3: {hit_ratio:.4f}, NDCG@3: {ndcg:.4f}")

Comparison with Uniform LoRA

iLoRA's performance can be compared to uniform LoRA to demonstrate its effectiveness in capturing individual user preferences.

import matplotlib.pyplot as plt

def compare_performance(uniform_lora_results, ilora_results, metrics):
    fig, ax = plt.subplots(figsize=(10, 6))
    x = range(len(metrics))
    width = 0.35

    ax.bar([i - width/2 for i in x], uniform_lora_results, width, label='Uniform LoRA')
    ax.bar([i + width/2 for i in x], ilora_results, width, label='iLoRA')

    ax.set_ylabel('Performance')
    ax.set_title('Uniform LoRA vs iLoRA Performance Comparison')
    ax.set_xticks(x)
    ax.set_xticklabels(metrics)
    ax.legend()

    plt.tight_layout()
    plt.show()

# Example usage
metrics = ['HitRatio@1', 'HitRatio@5', 'NDCG@5', 'NDCG@10']
uniform_lora_results = [0.3, 0.5, 0.4, 0.45]
ilora_results = [0.35, 0.55, 0.45, 0.5]
compare_performance(uniform_lora_results, ilora_results, metrics)

Real-life Example: Movie Recommendation

iLoRA can be applied to movie recommendation systems, where it can capture individual preferences for genres, actors, or directors.

class MovieRecommender(nn.Module):
    def __init__(self, num_movies, num_experts, d_model, r):
        super().__init__()
        self.movie_embeddings = nn.Embedding(num_movies, d_model)
        self.ilora = iLoRA(num_experts, d_model, r)

    def forward(self, movie_ids, sequence_repr):
        movie_embeds = self.movie_embeddings(movie_ids)
        return self.ilora(movie_embeds, sequence_repr)

# Example usage
num_movies, num_experts, d_model, r = 10000, 4, 64, 16
recommender = MovieRecommender(num_movies, num_experts, d_model, r)

# Simulate a user's movie history
user_history = torch.tensor([1, 4, 7, 2])  # Movie IDs
sequence_repr = torch.randn(1, d_model)  # Derived from SASRec or similar

# Get recommendations
recommendations = recommender(user_history, sequence_repr)
top_k = torch.topk(recommendations, k=5).indices
print("Top 5 movie recommendations:", top_k)

Real-life Example: Music Playlist Generation

iLoRA can enhance music playlist generation by adapting to individual listening patterns and preferences.

class PlaylistGenerator(nn.Module):
    def __init__(self, num_songs, num_experts, d_model, r):
        super().__init__()
        self.song_embeddings = nn.Embedding(num_songs, d_model)
        self.ilora = iLoRA(num_experts, d_model, r)

    def forward(self, playlist_history, sequence_repr):
        song_embeds = self.song_embeddings(playlist_history)
        return self.ilora(song_embeds, sequence_repr)

# Example usage
num_songs, num_experts, d_model, r = 1000000, 8, 128, 32
generator = PlaylistGenerator(num_songs, num_experts, d_model, r)

# Simulate a user's playlist history
playlist_history = torch.tensor([10, 25, 100, 5, 78])  # Song IDs
sequence_repr = torch.randn(1, d_model)  # Derived from listening history

# Generate next song recommendations
next_songs = generator(playlist_history, sequence_repr)
top_k = torch.topk(next_songs, k=3).indices
print("Top 3 song recommendations for the playlist:", top_k)

Challenges and Future Directions

While iLoRA shows promise, there are challenges to address:

Scalability with increasing number of experts
Balancing personalization and generalization
Handling cold-start problems for new users

Challenges and Future Directions

Future research directions include:

Exploring dynamic expert creation and pruning
Incorporating multi-modal data for richer representations
Investigating federated learning approaches for privacy-preserving recommendations

Challenges and Future Directions

class DynamicExpertILoRA(nn.Module):
    def __init__(self, initial_experts, d_model, r, expert_threshold):
        super().__init__()
        self.experts = nn.ModuleList([ExpertModule(d_model, r) for _ in range(initial_experts)])
        self.gating = GatingNetwork(d_model, initial_experts)
        self.expert_threshold = expert_threshold

    def forward(self, x, sequence_repr):
        expert_weights = self.gating(sequence_repr)
        outputs = [expert(x) for expert in self.experts]
        combined_output = sum(w * out for w, out in zip(expert_weights, outputs))

        # Dynamic expert management
        self.manage_experts(expert_weights)

        return combined_output

    def manage_experts(self, expert_weights):
        if expert_weights.max() < self.expert_threshold:
            self.add_expert()
        elif len(self.experts) > 1 and expert_weights.min() < self.expert_threshold / 10:
            self.prune_expert(expert_weights.argmin())

    def add_expert(self):
        new_expert = ExpertModule(self.experts[0].d_model, self.experts[0].r)
        self.experts.append(new_expert)
        self.gating.expand_gate(len(self.experts))

    def prune_expert(self, index):
        del self.experts[index]
        self.gating.contract_gate(len(self.experts))

Multimodal iLoRA for Enhanced Recommendations

Incorporating multiple modalities (e.g., text, images, audio) can provide richer representations for more accurate recommendations.

class MultimodalILoRA(nn.Module):
    def __init__(self, num_experts, d_model, r, modalities):
        super().__init__()
        self.modality_encoders = nn.ModuleDict({
            mod: nn.Linear(input_dim, d_model) for mod, input_dim in modalities.items()
        })
        self.ilora = iLoRA(num_experts, d_model, r)

    def forward(self, modality_inputs, sequence_repr):
        encoded_inputs = [
            encoder(modality_inputs[mod]) 
            for mod, encoder in self.modality_encoders.items()
        ]
        fused_input = torch.cat(encoded_inputs, dim=-1)
        return self.ilora(fused_input, sequence_repr)

# Example usage
modalities = {'text': 300, 'image': 2048, 'audio': 512}
multimodal_ilora = MultimodalILoRA(num_experts=4, d_model=256, r=32, modalities=modalities)

# Simulate multimodal inputs
text_input = torch.randn(1, 300)
image_input = torch.randn(1, 2048)
audio_input = torch.randn(1, 512)
sequence_repr = torch.randn(1, 256)

output = multimodal_ilora(
    {'text': text_input, 'image': image_input, 'audio': audio_input},
    sequence_repr
)
print("Multimodal iLoRA output shape:", output.shape)

Privacy-Preserving iLoRA with Federated Learning

Implementing iLoRA in a federated learning setup can enhance user privacy by keeping sensitive data on user devices.

import 

def federated_ilora_update(global_model, client_models, client_data):
    global_weights = global_model.state_dict()
    
    for client_model, data in zip(client_models, client_data):
        client_model.load_state_dict(.deep(global_weights))
        train_client_model(client_model, data)
    
    # Aggregate client models
    averaged_weights = average_weights([model.state_dict() for model in client_models])
    global_model.load_state_dict(averaged_weights)

def train_client_model(model, data):
    optimizer = optim.Adam(model.parameters())
    for epoch in range(LOCAL_EPOCHS):
        for batch in data:
            inputs, labels = batch
            outputs = model(inputs)
            loss = criterion(outputs, labels)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

def average_weights(weight_list):
    averaged_weights = .deep(weight_list[0])
    for key in averaged_weights.keys():
        averaged_weights[key] = torch.mean(torch.stack([weights[key] for weights in weight_list]), dim=0)
    return averaged_weights

# Example usage (pseudocode)
global_model = iLoRA(num_experts=4, d_model=256, r=32)
client_models = [.deep(global_model) for _ in range(NUM_CLIENTS)]
client_data = [get_client_data(i) for i in range(NUM_CLIENTS)]

for round in range(FEDERATED_ROUNDS):
    federated_ilora_update(global_model, client_models, client_data)

Additional Resources

For more information on the topics covered in this presentation, consider exploring the following resources:

"Customizing Language Models with Instance-wise LoRA for Sequential Recommendation" by Xiaoyu Kong et al. (2023) ArXiv: https://arxiv.org/abs/2408.10159
"LoRA: Low-Rank Adaptation of Large Language Models" by Edward J. Hu et al. (2021) ArXiv: https://arxiv.org/abs/2106.09685
"Mixture of Experts: A Literature Survey" by Sebastian Bischoff et al. (2021) ArXiv: https://arxiv.org/abs/2107.11447
"Federated Learning: Challenges, Methods, and Future Directions" by Tian Li et al. (2020) ArXiv: https://arxiv.org/abs/1908.07873

These resources provide deeper insights into the concepts and techniques discussed in this presentation, offering a solid foundation for further exploration and research in the field of personalized recommendation systems and adaptive language models.