Deep learning / Autoencoders and generative
Poisson Variational Autoencoder Modeling Count Data with Python
Introduction to Poisson Variational Autoencoder
The Poisson Variational Autoencoder (PVAE) is a type of variational autoencoder specifically designed for modeling count data, such as word frequencies or event counts. It addresses the limitations of traditional variational autoencoders in handling non-negative integer data by using the Poisson distribution as the output distribution.
Code:
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.distributions import Poisson
Variational Autoencoder (VAE) Recap
Before diving into the PVAE, let's briefly recap the standard Variational Autoencoder (VAE). VAEs are generative models that learn to encode input data into a latent space and then reconstruct the input from the latent representation. They use variational inference to approximate the intractable posterior distribution over the latent variables.
Code:
class VAE(nn.Module):
def __init__(self, input_dim, latent_dim):
super(VAE, self).__init__()
self.encoder = ... # Define encoder network
self.decoder = ... # Define decoder network
def reparameterize(self, mu, logvar):
std = torch.exp(0.5 * logvar)
eps = torch.randn_like(std)
return mu + eps * std
def forward(self, x):
mu, logvar = self.encoder(x)
z = self.reparameterize(mu, logvar)
recon_x = self.decoder(z)
return recon_x, mu, logvar
Poisson Distribution
The Poisson distribution is a discrete probability distribution that models the number of events occurring in a fixed interval of time or space, given an average rate of occurrence. It is particularly useful for modeling count data, such as word frequencies or event counts.
Code:
from torch.distributions import Poisson
# Example usage
rate = torch.tensor([2.0, 4.0]) # Mean rates
dist = Poisson(rate)
samples = dist.sample() # Sample from the Poisson distribution
log_probs = dist.log_prob(samples) # Compute log-probabilities
Poisson Variational Autoencoder (PVAE) Model
The PVAE model consists of an encoder network that maps the input data to a latent representation, and a decoder network that reconstructs the input from the latent representation. The decoder output is modeled using a Poisson distribution, making it suitable for count data.
Code:
class PVAE(nn.Module):
def __init__(self, input_dim, latent_dim):
super(PVAE, self).__init__()
self.encoder = ... # Define encoder network
self.decoder = ... # Define decoder network
def reparameterize(self, mu, logvar):
std = torch.exp(0.5 * logvar)
eps = torch.randn_like(std)
return mu + eps * std
def forward(self, x):
mu, logvar = self.encoder(x)
z = self.reparameterize(mu, logvar)
recon_rates = self.decoder(z)
return recon_rates, mu, logvar
PVAE Loss Function
The PVAE loss function is composed of two terms: the reconstruction loss, which measures the discrepancy between the input data and the reconstructed output, and the Kullback-Leibler (KL) divergence, which enforces a regularization on the latent space to ensure a smooth latent representation.
Code:
def pvae_loss(x, recon_rates, mu, logvar):
bce_loss = F.poisson_nll_loss(recon_rates, x, log_input=False)
kl_div = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())
return bce_loss + kl_div
PVAE Training
Training the PVAE involves minimizing the loss function over the training data. This can be achieved using stochastic gradient descent or any other suitable optimization algorithm.
Code:
model = PVAE(input_dim, latent_dim)
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
for epoch in range(num_epochs):
for x in data_loader:
optimizer.zero_grad()
recon_rates, mu, logvar = model(x)
loss = pvae_loss(x, recon_rates, mu, logvar)
loss.backward()
optimizer.step()
PVAE Inference and Sampling
After training, the PVAE can be used to generate new samples from the learned latent space by sampling from the latent distribution and passing the samples through the decoder network.
Code:
def sample(model, num_samples, device):
z = torch.randn(num_samples, model.latent_dim).to(device)
recon_rates = model.decoder(z)
samples = Poisson(recon_rates).sample()
return samples
PVAE Applications
PVAEs have been successfully applied to various tasks involving count data, such as topic modeling, natural language processing, and recommendation systems. They offer a powerful generative approach for modeling discrete, non-negative data while capturing the underlying latent structure.
Code:
# Load pre-trained PVAE model
model = PVAE(input_dim, latent_dim)
model.load_state_dict(torch.load('pvae_model.pth'))
# Generate new samples
samples = sample(model, num_samples=100, device='cuda')
Handling Sparse Data
Count data, such as word frequencies or event counts, can often be highly sparse, with many zero entries. PVAEs can handle sparse data by incorporating additional techniques, such as dropout or adding a sparsity-inducing prior on the latent variables.
Code:
class SparseEncoder(nn.Module):
def __init__(self, input_dim, latent_dim, dropout_rate=0.2):
super(SparseEncoder, self).__init__()
self.dropout = nn.Dropout(dropout_rate)
# Define encoder network with dropout layers
def forward(self, x):
x = self.dropout(x)
# Encoder forward pass
return mu, logvar
PVAE for Topic Modeling
PVAEs have shown promising results in topic modeling, where the goal is to discover latent topics from a collection of documents. The PVAE can learn a low-dimensional representation of the document-word matrix, capturing the underlying topics and their word distributions.
Code:
# Preprocess document-word matrix
doc_word_matrix = ...
model = PVAE(input_dim=vocab_size, latent_dim=num_topics)
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
for epoch in range(num_epochs):
for doc in doc_word_matrix:
recon_rates, mu, logvar = model(doc)
loss = pvae_loss(doc, recon_rates, mu, logvar)
loss.backward()
optimizer.step()
PVAE for Recommendation Systems
PVAEs can be used in recommendation systems to model user-item interactions, such as purchase histories or movie ratings. The PVAE can learn a low-dimensional representation of user preferences and item characteristics, enabling personalized recommendations and capturing complex patterns in the data.
Code:
# Preprocess user-item interaction matrix
user_item_matrix = ...
model = PVAE(input_dim=num_items, latent_dim=latent_dim)
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
for epoch in range(num_epochs):
for user_vec in user_item_matrix:
recon_rates, mu, logvar = model(user_vec)
loss = pvae_loss(user_vec, recon_rates, mu, logvar)
loss.backward()
optimizer.step()
# Generate recommendations for a user
user_id = 42
user_vec = user_item_matrix[user_id]
recon_rates, mu, logvar = model(user_vec)
top_item_indices = torch.argsort(recon_rates, descending=True)[:k]
top_item_recommendations = [item_mapping[idx] for idx in top_item_indices]
PVAE for Anomaly Detection
PVAEs can be used for anomaly detection in count data by leveraging the reconstruction error. Instances with high reconstruction errors are considered anomalies, as they deviate significantly from the learned distribution.
Code:
def detect_anomalies(model, data, threshold):
anomalies = []
for x in data:
recon_rates, mu, logvar = model(x)
recon_error = pvae_loss(x, recon_rates, mu, logvar)
if recon_error > threshold:
anomalies.append(x)
return anomalies
Challenges and Future Directions
While PVAEs have shown promising results in various applications, there are still challenges and areas for future research. These include handling highly sparse data, improving scalability for large datasets, and exploring alternative prior distributions or regularization techniques for the latent space.
Code:
# Example of using a sparse prior for the latent variables
class SparseLatentPrior(nn.Module):
def __init__(self, latent_dim):
super(SparseLatentPrior, self).__init__()
self.latent_dim = latent_dim
self.mu = nn.Parameter(torch.zeros(latent_dim))
self.logvar = nn.Parameter(torch.zeros(latent_dim))
def forward(self, z):
# Compute sparse prior loss (e.g., L1 regularization)
sparse_loss = torch.sum(torch.abs(z - self.mu))
return sparse_loss
Slide 14 (Additional Resources):
Additional Resources
For further reading and exploration of Poisson Variational Autoencoders and related topics, here are some recommended resources from ArXiv.org:
- "Poisson Gamma Belief Network" by Zhou et al. (https://arxiv.org/abs/1711.07936)
- "Auto-Encoding Variational Bayes" by Kingma and Welling (https://arxiv.org/abs/1312.6114)
- "Variational Autoencoders for Collaborative Filtering" by Liang et al. (https://arxiv.org/abs/1802.05814)
- "Bayesian Poisson Tensor Factorization for Recommender Systems" by Liang et al. (https://arxiv.org/abs/1711.04269)
Please note that these resources are subject to availability and may change over time.