xbe.at

LLMs and generative AI / Vector databases and embeddings

Vector Databases Powering AI and Machine Learning

4 min read, 15 sections

Contents
  1. Introduction to Vector Databases
  2. What are Vectors?
  3. Code for Vector Representation
  4. Why Vector Databases?
  5. Source Code for Vector Distance Calculation
  6. Efficient Similarity Search
  7. Source Code for Simple LSH Implementation
  8. Real-Life Example: Recommendation Systems
  9. Source Code for Simple Recommendation System
  10. Real-Life Example: Image Similarity Search
  11. Source Code for Image Feature Extraction (Pseudo-code)
  12. Scalability and Performance
  13. Source Code for Vector Database Benchmark
  14. Future of Vector Databases
  15. Additional Resources

Introduction to Vector Databases

Vector databases are specialized systems designed to store and efficiently query high-dimensional vector data. They are becoming increasingly important in the era of AI and machine learning, where complex data representations are common.

What are Vectors?

Vectors are mathematical objects representing data points in multi-dimensional space. In the context of machine learning, vectors often represent features or embeddings of data.

Code for Vector Representation

# Vector representation in Python
class Vector:
    def __init__(self, *components):
        self.components = list(components)

    def __str__(self):
        return f"Vector({', '.join(map(str, self.components))})"

    def __len__(self):
        return len(self.components)

# Example usage
v1 = Vector(1, 2, 3)
v2 = Vector(4.5, 5.5, 6.5, 7.5)

print(f"v1: {v1}, dimension: {len(v1)}")
print(f"v2: {v2}, dimension: {len(v2)}")

Why Vector Databases?

Traditional databases struggle with high-dimensional data and similarity searches. Vector databases are optimized for these tasks, making them crucial for modern AI applications.

Source Code for Vector Distance Calculation

import math

def euclidean_distance(v1, v2):
    if len(v1) != len(v2):
        raise ValueError("Vectors must have the same dimension")
    
    return math.sqrt(sum((a - b) ** 2 for a, b in zip(v1, v2)))

# Example usage
vector1 = [1, 2, 3]
vector2 = [4, 5, 6]

distance = euclidean_distance(vector1, vector2)
print(f"Euclidean distance between {vector1} and {vector2}: {distance:.2f}")

Efficient Similarity Search

Vector databases use specialized indexing techniques like LSH (Locality-Sensitive Hashing) or HNSW (Hierarchical Navigable Small World) to perform fast similarity searches in high-dimensional spaces.

Source Code for Simple LSH Implementation

import random

class SimpleLSH:
    def __init__(self, dim, num_hashes):
        self.dim = dim
        self.num_hashes = num_hashes
        self.hash_functions = [self._random_vector() for _ in range(num_hashes)]

    def _random_vector(self):
        return [random.gauss(0, 1) for _ in range(self.dim)]

    def hash(self, vector):
        return [1 if sum(a*b for a, b in zip(v, vector)) >= 0 else 0
                for v in self.hash_functions]

# Example usage
lsh = SimpleLSH(dim=3, num_hashes=5)
vector = [1, 2, 3]
hash_code = lsh.hash(vector)
print(f"LSH hash code for {vector}: {hash_code}")

Real-Life Example: Recommendation Systems

Vector databases power recommendation systems by storing item and user embeddings. They enable quick retrieval of similar items or users based on their vector representations.

Source Code for Simple Recommendation System

def cosine_similarity(v1, v2):
    dot_product = sum(a*b for a, b in zip(v1, v2))
    magnitude1 = math.sqrt(sum(a*a for a in v1))
    magnitude2 = math.sqrt(sum(b*b for b in v2))
    return dot_product / (magnitude1 * magnitude2)

def recommend_items(user_vector, item_vectors, top_n=3):
    similarities = [(i, cosine_similarity(user_vector, item_vector))
                    for i, item_vector in enumerate(item_vectors)]
    return sorted(similarities, key=lambda x: x[1], reverse=True)[:top_n]

# Example usage
user_vector = [0.5, 0.3, 0.2]
item_vectors = [
    [0.4, 0.4, 0.2],
    [0.6, 0.2, 0.2],
    [0.3, 0.5, 0.2],
    [0.7, 0.1, 0.2]
]

recommendations = recommend_items(user_vector, item_vectors)
print("Top recommendations (item_id, similarity):")
for item_id, similarity in recommendations:
    print(f"Item {item_id}: {similarity:.2f}")

Real-Life Example: Image Similarity Search

Vector databases enable efficient image similarity search by storing and querying image feature vectors extracted from deep learning models.

Source Code for Image Feature Extraction (Pseudo-code)

# Note: This is pseudo-code and requires a pre-trained CNN model
def extract_image_features(image_path, model):
    image = load_image(image_path)
    preprocessed_image = preprocess(image)
    features = model.extract_features(preprocessed_image)
    return features

def find_similar_images(query_image, image_database, top_n=5):
    query_features = extract_image_features(query_image, model)
    similarities = []
    for image_id, features in image_database.items():
        similarity = cosine_similarity(query_features, features)
        similarities.append((image_id, similarity))
    return sorted(similarities, key=lambda x: x[1], reverse=True)[:top_n]

# Example usage (pseudo-code)
image_database = load_image_database()
query_image = "path/to/query/image.jpg"
similar_images = find_similar_images(query_image, image_database)
print("Similar images:", similar_images)

Scalability and Performance

Vector databases are designed to handle large-scale data and provide fast query responses, making them suitable for applications with millions or billions of vectors.

Source Code for Vector Database Benchmark

import time
import random

class SimpleVectorDB:
    def __init__(self):
        self.vectors = {}

    def insert(self, id, vector):
        self.vectors[id] = vector

    def search(self, query_vector, k=1):
        return sorted(
            [(id, cosine_similarity(query_vector, v)) for id, v in self.vectors.items()],
            key=lambda x: x[1],
            reverse=True
        )[:k]

# Benchmark function
def benchmark_vector_db(num_vectors, vector_dim, num_queries):
    db = SimpleVectorDB()
    
    # Insert vectors
    start_time = time.time()
    for i in range(num_vectors):
        vector = [random.random() for _ in range(vector_dim)]
        db.insert(i, vector)
    insert_time = time.time() - start_time
    
    # Perform queries
    start_time = time.time()
    for _ in range(num_queries):
        query_vector = [random.random() for _ in range(vector_dim)]
        db.search(query_vector, k=10)
    query_time = time.time() - start_time
    
    print(f"Insertion time for {num_vectors} vectors: {insert_time:.2f} seconds")
    print(f"Query time for {num_queries} queries: {query_time:.2f} seconds")

# Run benchmark
benchmark_vector_db(num_vectors=100000, vector_dim=128, num_queries=1000)

Future of Vector Databases

As AI continues to advance, vector databases will play an increasingly important role in managing and querying high-dimensional data, supporting applications in natural language processing, computer vision, and more.

Additional Resources

For more information on vector databases and their applications, consider exploring these resources:

  1. "Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs" by Yu. A. Malkov and D. A. Yashunin (2018). ArXiv: https://arxiv.org/abs/1603.09320
  2. "Billion-scale similarity search with GPUs" by Johnson et al. (2017). ArXiv: https://arxiv.org/abs/1702.08734

These papers provide in-depth discussions on efficient similarity search algorithms used in vector databases.

More in Vector databases and embeddings

  1. Vanna Adaptive Text-to-SQL Tool5 min

    Vanna represents a significant advancement in text-to-SQL technology, utilizing dynamic model adaptation and real-time learning capabilities. Its...

  2. Understanding Vector Databases Fundamentals and Applications14 min

    Vector databases operate on the principle of vector spaces where data points are represented as high-dimensional vectors. Understanding the mathematical...

  3. Tokens and Tokenization in Large Language Models in Python7 min

    Introduction to Tokens and Tokenization in LLMs

  4. Techniques for Compressing Large Language Models in Python7 min

    Model compression is crucial for deploying large language models (LLMs) in resource-constrained environments. It aims to reduce model size and computational...

  5. Running Open-Source LLMs Locally with Ollama10 min

    The first step involves configuring the Python environment to interact with Ollama's API endpoints. This allows seamless communication between Python...

  6. Running Open-Source Large Language Models Locally with Python5 min

    Running large language models (LLMs) locally can be a cost-effective and privacy-conscious approach for researchers, developers, and enthusiasts. This...