xbe.at

LLMs and generative AI / Vector databases and embeddings

Exploring Vector Databases

10 min read, 15 sections

Contents
  1. Introduction to Vector Databases
  2. Vector Representation of Data
  3. Vector Similarity Measures
  4. Indexing in Vector Databases
  5. Real-Life Example: Image Similarity Search
  6. Vector Database Architecture
  7. Scaling Vector Databases
  8. Query Processing in Vector Databases
  9. Real-Life Example: Recommendation Systems
  10. Handling Updates in Vector Databases
  11. Vector Databases vs. Traditional Databases
  12. Challenges and Future Directions
  13. Implementing a Simple Vector Database
  14. Performance Optimization Techniques
  15. Additional Resources

Introduction to Vector Databases

Vector databases are specialized systems designed to store, index, and query high-dimensional vectors. These vectors are mathematical representations of data points, capturing the semantic essence of information. Unlike traditional databases that focus on structured data, vector databases excel at handling complex, multidimensional data, making them ideal for similarity searches and AI-driven applications.

import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer

# Example: Converting text to vectors
texts = ["Vector databases are powerful", "Databases store information", "AI uses vector representations"]
vectorizer = TfidfVectorizer()
vectors = vectorizer.fit_transform(texts).toarray()

print("Text vectors:")
print(vectors)

Vector Representation of Data

Vector representation involves transforming various types of data (text, images, audio) into numerical vectors. These vectors capture the essential features and relationships within the data, allowing for efficient comparisons and analysis.

from sentence_transformers import SentenceTransformer

# Example: Using a pre-trained model to generate sentence embeddings
model = SentenceTransformer('paraphrase-MiniLM-L6-v2')
sentences = ["This is an example sentence", "Each sentence becomes a vector"]
embeddings = model.encode(sentences)

print("Sentence embeddings:")
print(embeddings)

Vector Similarity Measures

Vector databases rely on similarity measures to compare vectors. Common metrics include cosine similarity and Euclidean distance. These measures help determine how close or similar two vectors are in the high-dimensional space.

import numpy as np

def cosine_similarity(v1, v2):
    return np.dot(v1, v2) / (np.linalg.norm(v1) * np.linalg.norm(v2))

def euclidean_distance(v1, v2):
    return np.linalg.norm(v1 - v2)

v1 = np.array([1, 2, 3])
v2 = np.array([4, 5, 6])

print(f"Cosine similarity: {cosine_similarity(v1, v2):.4f}")
print(f"Euclidean distance: {euclidean_distance(v1, v2):.4f}")

Indexing in Vector Databases

Efficient indexing is crucial for vector databases to perform fast similarity searches. Techniques like Locality-Sensitive Hashing (LSH) and Hierarchical Navigable Small World (HNSW) graphs are commonly used to create index structures that enable quick nearest neighbor searches.

from sklearn.neighbors import NearestNeighbors
import numpy as np

# Generate random vectors
vectors = np.random.rand(1000, 128)

# Create an index using k-d tree
index = NearestNeighbors(n_neighbors=5, algorithm='kd_tree').fit(vectors)

# Perform a nearest neighbor search
query = np.random.rand(1, 128)
distances, indices = index.kneighbors(query)

print("Nearest neighbor indices:", indices[0])
print("Distances:", distances[0])

Real-Life Example: Image Similarity Search

Vector databases are widely used in image similarity search applications. By converting images into vector representations, we can efficiently find visually similar images across large datasets.

from PIL import Image
import numpy as np
from torchvision import transforms, models

# Load a pre-trained ResNet model
model = models.resnet50(pretrained=True)
model = model.eval()

# Define image transformation
transform = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])

def image_to_vector(image_path):
    img = Image.open(image_path)
    img_t = transform(img)
    batch_t = torch.unsqueeze(img_t, 0)
    features = model(batch_t)
    return features.detach().numpy().flatten()

# Example usage
image_vector = image_to_vector('path/to/image.jpg')
print("Image vector shape:", image_vector.shape)

Vector Database Architecture

Vector databases typically consist of several key components: a vector store for efficient data storage, an indexing mechanism for fast retrieval, a query processor for handling similarity searches, and often, a metadata store for additional information about the vectors.

import faiss
import numpy as np

class SimpleVectorDatabase:
    def __init__(self, dimension):
        self.index = faiss.IndexFlatL2(dimension)
        self.metadata = []

    def add_vectors(self, vectors, metadata):
        self.index.add(vectors)
        self.metadata.extend(metadata)

    def search(self, query_vector, k=5):
        distances, indices = self.index.search(query_vector.reshape(1, -1), k)
        return [(self.metadata[i], distances[0][j]) for j, i in enumerate(indices[0])]

# Usage example
db = SimpleVectorDatabase(128)
vectors = np.random.rand(100, 128).astype('float32')
metadata = [f"Item_{i}" for i in range(100)]
db.add_vectors(vectors, metadata)

query = np.random.rand(128).astype('float32')
results = db.search(query)
print("Search results:", results)

Scaling Vector Databases

As datasets grow, vector databases need to scale efficiently. Techniques like sharding, distributed indexing, and load balancing are employed to handle large-scale vector collections while maintaining query performance.

import numpy as np
from multiprocessing import Pool

class DistributedVectorDB:
    def __init__(self, num_shards):
        self.num_shards = num_shards
        self.shards = [[] for _ in range(num_shards)]

    def add_vector(self, vector):
        shard_id = hash(tuple(vector)) % self.num_shards
        self.shards[shard_id].append(vector)

    def parallel_search(self, query_vector, k=5):
        with Pool(processes=self.num_shards) as pool:
            results = pool.starmap(self._search_shard, 
                                   [(shard, query_vector, k) for shard in self.shards])
        return sorted(sum(results, []), key=lambda x: x[1])[:k]

    def _search_shard(self, shard, query_vector, k):
        return sorted([(v, np.linalg.norm(query_vector - v)) for v in shard], 
                      key=lambda x: x[1])[:k]

# Usage
db = DistributedVectorDB(num_shards=4)
for _ in range(1000):
    db.add_vector(np.random.rand(128))

query = np.random.rand(128)
results = db.parallel_search(query)
print("Distributed search results:", results[:5])

Query Processing in Vector Databases

Query processing in vector databases involves transforming the input query into a vector representation, performing similarity searches using the indexed data, and often applying additional filters or constraints based on metadata.

import numpy as np
from scipy.spatial.distance import cosine

class VectorDBWithMetadata:
    def __init__(self):
        self.vectors = []
        self.metadata = []

    def add_item(self, vector, metadata):
        self.vectors.append(vector)
        self.metadata.append(metadata)

    def query(self, query_vector, metadata_filter=None, k=5):
        similarities = [1 - cosine(query_vector, v) for v in self.vectors]
        results = list(zip(similarities, self.metadata))
        
        if metadata_filter:
            results = [r for r in results if metadata_filter(r[1])]
        
        return sorted(results, key=lambda x: x[0], reverse=True)[:k]

# Usage
db = VectorDBWithMetadata()
for i in range(100):
    db.add_item(np.random.rand(128), {"category": f"cat_{i%5}", "id": i})

query = np.random.rand(128)
results = db.query(query, metadata_filter=lambda m: m["category"] == "cat_2")
print("Query results with metadata filter:", results)

Real-Life Example: Recommendation Systems

Vector databases power many modern recommendation systems. By representing user preferences and item characteristics as vectors, these systems can quickly find and suggest relevant items to users.

import numpy as np
from sklearn.metrics.pairwise import cosine_similarity

class SimpleRecommendationSystem:
    def __init__(self, num_users, num_items, embedding_dim):
        self.user_embeddings = np.random.rand(num_users, embedding_dim)
        self.item_embeddings = np.random.rand(num_items, embedding_dim)

    def get_recommendations(self, user_id, top_k=5):
        user_vector = self.user_embeddings[user_id]
        similarities = cosine_similarity([user_vector], self.item_embeddings)[0]
        top_items = similarities.argsort()[-top_k:][::-1]
        return [(item_id, similarities[item_id]) for item_id in top_items]

# Usage
recommender = SimpleRecommendationSystem(num_users=1000, num_items=10000, embedding_dim=128)
user_id = 42
recommendations = recommender.get_recommendations(user_id)
print(f"Top recommendations for user {user_id}:")
for item_id, score in recommendations:
    print(f"Item {item_id}: Score {score:.4f}")

Handling Updates in Vector Databases

Vector databases need to efficiently handle updates to existing vectors and the addition of new vectors. This often involves strategies like batch updates, incremental indexing, and periodic reindexing to maintain performance.

import faiss
import numpy as np

class UpdatableVectorDB:
    def __init__(self, dimension, index_size=1000):
        self.dimension = dimension
        self.index_size = index_size
        self.index = faiss.IndexFlatL2(dimension)
        self.buffer = []

    def add_vector(self, vector):
        if len(self.buffer) >= self.index_size:
            self._merge_buffer()
        self.buffer.append(vector)

    def _merge_buffer(self):
        if self.buffer:
            buffer_array = np.array(self.buffer, dtype='float32')
            self.index.add(buffer_array)
            self.buffer.clear()

    def search(self, query_vector, k=5):
        self._merge_buffer()  # Ensure all vectors are searchable
        distances, indices = self.index.search(query_vector.reshape(1, -1), k)
        return list(zip(indices[0], distances[0]))

# Usage
db = UpdatableVectorDB(dimension=128)
for _ in range(1500):
    db.add_vector(np.random.rand(128).astype('float32'))

query = np.random.rand(128).astype('float32')
results = db.search(query)
print("Search results after updates:", results)

Vector Databases vs. Traditional Databases

Vector databases differ from traditional databases in their data model, indexing methods, and query capabilities. While traditional databases excel at exact matches and range queries, vector databases are optimized for similarity searches and semantic understanding.

import sqlite3
import numpy as np
from scipy.spatial.distance import cosine

# Traditional database (SQLite) for exact matches
conn = sqlite3.connect(':memory:')
c = conn.cursor()
c.execute('''CREATE TABLE items
             (id INTEGER PRIMARY KEY, name TEXT, category TEXT)''')
c.executemany('INSERT INTO items VALUES (?,?,?)', 
              [(i, f'item_{i}', f'cat_{i%5}') for i in range(100)])

# Vector database for similarity search
class SimpleVectorDB:
    def __init__(self):
        self.vectors = []
        self.metadata = []

    def add_item(self, vector, metadata):
        self.vectors.append(vector)
        self.metadata.append(metadata)

    def similarity_search(self, query_vector, k=5):
        similarities = [1 - cosine(query_vector, v) for v in self.vectors]
        top_k = sorted(enumerate(similarities), key=lambda x: x[1], reverse=True)[:k]
        return [(self.metadata[i], sim) for i, sim in top_k]

# Usage comparison
print("Traditional DB - Exact match:")
c.execute('SELECT * FROM items WHERE category = ?', ('cat_2',))
print(c.fetchall()[:5])

vector_db = SimpleVectorDB()
for i in range(100):
    vector_db.add_item(np.random.rand(128), f'item_{i}')

print("\nVector DB - Similarity search:")
query = np.random.rand(128)
print(vector_db.similarity_search(query))

conn.close()

Challenges and Future Directions

Vector databases face challenges such as the curse of dimensionality, maintaining accuracy with large-scale datasets, and handling dynamic data efficiently. Future directions include improved indexing algorithms, hardware acceleration, and integration with machine learning pipelines.

import numpy as np
from sklearn.decomposition import PCA

def demonstrate_curse_of_dimensionality():
    dimensions = [10, 100, 1000, 10000]
    num_points = 1000
    
    for dim in dimensions:
        points = np.random.rand(num_points, dim)
        distances = np.linalg.norm(points - points.mean(axis=0), axis=1)
        
        print(f"Dimension: {dim}")
        print(f"  Min distance: {distances.min():.4f}")
        print(f"  Max distance: {distances.max():.4f}")
        print(f"  Ratio max/min: {distances.max() / distances.min():.4f}")

def dimensionality_reduction_example():
    # Generate high-dimensional data
    dim = 1000
    num_points = 100
    data = np.random.rand(num_points, dim)
    
    # Apply PCA for dimensionality reduction
    pca = PCA(n_components=10)
    reduced_data = pca.fit_transform(data)
    
    print(f"Original shape: {data.shape}")
    print(f"Reduced shape: {reduced_data.shape}")
    print(f"Explained variance ratio: {pca.explained_variance_ratio_.sum():.4f}")

print("Demonstrating curse of dimensionality:")
demonstrate_curse_of_dimensionality()

print("\nDimensionality reduction example:")
dimensionality_reduction_example()

Implementing a Simple Vector Database

Let's implement a basic vector database using Python to demonstrate the core concepts we've discussed. This implementation will support adding vectors, searching for similar vectors, and basic metadata filtering.

import numpy as np
from scipy.spatial.distance import cosine

class SimpleVectorDatabase:
    def __init__(self):
        self.vectors = []
        self.metadata = []

    def add_vector(self, vector, metadata=None):
        self.vectors.append(vector)
        self.metadata.append(metadata)

    def search(self, query_vector, k=5, metadata_filter=None):
        similarities = [1 - cosine(query_vector, v) for v in self.vectors]
        results = list(zip(similarities, self.metadata, range(len(self.vectors))))
        
        if metadata_filter:
            results = [r for r in results if metadata_filter(r[1])]
        
        return sorted(results, key=lambda x: x[0], reverse=True)[:k]

    def update_vector(self, index, new_vector):
        if 0 <= index < len(self.vectors):
            self.vectors[index] = new_vector
        else:
            raise IndexError("Vector index out of range")

# Usage example
db = SimpleVectorDatabase()

# Add vectors
for i in range(100):
    db.add_vector(np.random.rand(128), {"id": i, "category": f"cat_{i % 5}"})

# Search
query = np.random.rand(128)
results = db.search(query, k=3, metadata_filter=lambda m: m["category"] == "cat_2")
print("Search results:", results)

# Update a vector
db.update_vector(0, np.random.rand(128))

Performance Optimization Techniques

Vector databases employ various techniques to optimize performance, especially for large-scale datasets. These include approximate nearest neighbor search algorithms, data compression, and caching strategies.

import numpy as np
from sklearn.neighbors import NearestNeighbors

class OptimizedVectorDB:
    def __init__(self, dim, n_trees=10):
        self.dim = dim
        self.vectors = []
        self.metadata = []
        self.index = NearestNeighbors(n_neighbors=5, algorithm='ball_tree', n_jobs=-1)
        self.n_trees = n_trees

    def add_vectors(self, vectors, metadata):
        self.vectors.extend(vectors)
        self.metadata.extend(metadata)
        self._rebuild_index()

    def _rebuild_index(self):
        self.index.fit(self.vectors)

    def search(self, query_vector, k=5):
        distances, indices = self.index.kneighbors([query_vector], n_neighbors=k)
        return [(self.metadata[i], dist) for i, dist in zip(indices[0], distances[0])]

# Usage
db = OptimizedVectorDB(dim=128)
vectors = np.random.rand(10000, 128)
metadata = [{"id": i} for i in range(10000)]
db.add_vectors(vectors, metadata)

query = np.random.rand(128)
results = db.search(query)
print("Optimized search results:", results)

Additional Resources

For those interested in delving deeper into vector databases and their applications, here are some valuable resources:

  1. "Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs" by Yu. A. Malkov and D. A. Yashunin (ArXiv:1603.09320) URL: https://arxiv.org/abs/1603.09320
  2. "FAISS: A Library for Efficient Similarity Search" by J. Johnson et al. (ArXiv:1702.08734) URL: https://arxiv.org/abs/1702.08734
  3. "ANN-Benchmarks: A Benchmarking Tool for Approximate Nearest Neighbor Algorithms" by M. Aumüller et al. (ArXiv:1807.05614) URL: https://arxiv.org/abs/1807.05614

These papers provide in-depth discussions on advanced algorithms and techniques used in modern vector databases.

More in Vector databases and embeddings

  1. Introduction to LLM-based Text-to-SQL Integration12 min

    Text-to-SQL integration using Large Language Models (LLMs) is a powerful approach to bridge natural language queries with database operations. This...

  2. Embeddings Re-ranking and Vector Databases in Python11 min

    Embeddings are dense vector representations of data that capture semantic meaning. They're crucial for various machine learning tasks, especially in natural...

  3. Introduction to Vector Databases for AI and Machine Learning in Python8 min

    Vector databases are a type of database designed to store and manipulate high-dimensional vectors efficiently. They are particularly useful in AI and Machine...

  4. Efficient Quantization for Large Language Models8 min

    Quantization is a technique to reduce model size by mapping floating-point values to lower-precision formats. This process involves determining the optimal...

  5. MatMul-Free LLMs for Enhanced Efficiency in Python11 min

    Traditional Large Language Models (LLMs) rely heavily on matrix multiplication operations, which can be computationally expensive. MatMul-Free LLMs aim to...

  6. Decoding Embedding in LLMs with Python11 min

    Embeddings are dense vector representations of words or tokens in a continuous vector space. They capture semantic relationships between words, allowing...