xbe.at

Miscellany / Embeddings and vectors

Text Vectorization Transforming Words into Numbers

10 min read, 13 sections

Contents
  1. Vectorization Foundations
  2. One-Hot Encoding Implementation
  3. Bag of Words (BoW) Construction
  4. N-gram Feature Extraction
  5. TF-IDF Vectorization
  6. Word Embeddings with Word2Vec Implementation
  7. Document Classification Pipeline
  8. Advanced Tokenization with Subword Units
  9. Sentence Transformers Implementation
  10. Document Similarity with LSA
  11. Real-world Implementation: Sentiment Analysis
  12. Results for: Sentiment Analysis Implementation
  13. Additional Resources

Vectorization Foundations

Text vectorization transforms human-readable text into numerical representations that machine learning models can process. The foundation begins with tokenization, where text is split into meaningful units like words or subwords, followed by numerical encoding to create feature vectors.

def basic_tokenization(text):
    # Convert to lowercase and split into words
    tokens = text.lower().split()
    
    # Create vocabulary (unique tokens)
    vocab = sorted(set(tokens))
    
    # Create token to index mapping
    token2idx = {token: idx for idx, token in enumerate(vocab)}
    
    # Example usage
    text = "Hello world of text vectorization"
    tokens = basic_tokenization(text)
    print(f"Tokens: {tokens}")
    print(f"Vocabulary: {vocab}")
    print(f"Token to index mapping: {token2idx}")

One-Hot Encoding Implementation

One-hot encoding represents each word as a binary vector where the position corresponding to the word's index in the vocabulary is marked with 1, while all other positions are 0. This creates sparse vectors with dimensionality equal to vocabulary size.

import numpy as np

def one_hot_encode(text, vocab_size):
    # Tokenize and create vocabulary
    tokens = text.lower().split()
    vocab = sorted(set(tokens))
    token2idx = {token: idx for idx, token in enumerate(vocab)}
    
    # Create one-hot vectors
    encoded = np.zeros((len(tokens), vocab_size))
    for i, token in enumerate(tokens):
        encoded[i, token2idx[token]] = 1
    
    return encoded, token2idx

# Example usage
text = "the cat sat on the mat"
vocab_size = len(set(text.split()))
vectors, mapping = one_hot_encode(text, vocab_size)
print(f"One-hot vectors shape: {vectors.shape}")
print(f"First token encoding: {vectors[0]}")

Bag of Words (BoW) Construction

The Bag of Words model creates a document-term matrix where each row represents a document and each column represents a term in the vocabulary. The values indicate the frequency of terms in each document, discarding word order information.

from collections import Counter

def create_bow(documents):
    # Create vocabulary from all documents
    vocab = set()
    for doc in documents:
        vocab.update(doc.lower().split())
    vocab = sorted(vocab)
    
    # Create document-term matrix
    bow_matrix = []
    for doc in documents:
        # Count word frequencies
        word_counts = Counter(doc.lower().split())
        # Create document vector
        doc_vector = [word_counts.get(word, 0) for word in vocab]
        bow_matrix.append(doc_vector)
    
    return np.array(bow_matrix), vocab

# Example usage
docs = [
    "the cat sat on the mat",
    "the dog ran in the park"
]
bow_matrix, vocabulary = create_bow(docs)
print(f"BoW Matrix:\n{bow_matrix}")
print(f"Vocabulary: {vocabulary}")

N-gram Feature Extraction

N-grams capture local word order by considering sequences of N consecutive tokens. This approach provides more context than individual words and helps capture phrases and local dependencies in the text.

def generate_ngrams(text, n):
    # Tokenize text
    tokens = text.lower().split()
    
    # Generate n-grams
    ngrams = []
    for i in range(len(tokens) - n + 1):
        ngram = ' '.join(tokens[i:i + n])
        ngrams.append(ngram)
    
    # Count n-gram frequencies
    ngram_counts = Counter(ngrams)
    return ngram_counts

# Example usage
text = "the quick brown fox jumps over the lazy dog"
bigrams = generate_ngrams(text, 2)
trigrams = generate_ngrams(text, 3)

print("Bigrams:", dict(bigrams))
print("Trigrams:", dict(trigrams))

TF-IDF Vectorization

TF-IDF (Term Frequency-Inverse Document Frequency) weights terms based on their importance in a document relative to a corpus. This technique reduces the impact of common words while emphasizing distinctive terms.

import numpy as np

def compute_tfidf(documents):
    # Create vocabulary
    vocab = set()
    for doc in documents:
        vocab.update(doc.lower().split())
    vocab = sorted(vocab)
    
    # Calculate document frequencies
    doc_freq = {word: 0 for word in vocab}
    for doc in documents:
        words = set(doc.lower().split())
        for word in words:
            doc_freq[word] += 1
    
    # Compute TF-IDF
    N = len(documents)
    tfidf_matrix = []
    
    for doc in documents:
        word_counts = Counter(doc.lower().split())
        tfidf_vector = []
        
        for word in vocab:
            tf = word_counts.get(word, 0)
            idf = np.log(N / (doc_freq[word] + 1))
            tfidf_vector.append(tf * idf)
        
        tfidf_matrix.append(tfidf_vector)
    
    return np.array(tfidf_matrix), vocab

# Example usage
docs = [
    "this is a sample document",
    "this is another example document",
    "and this is a third one"
]

tfidf_matrix, vocabulary = compute_tfidf(docs)
print(f"TF-IDF Matrix shape: {tfidf_matrix.shape}")
print(f"First document TF-IDF:\n{tfidf_matrix[0]}")

Word Embeddings with Word2Vec Implementation

Word embeddings represent words in a continuous vector space where semantically similar words are mapped to nearby points. This implementation demonstrates a simplified version of the Skip-gram model, focusing on the core architecture.

import numpy as np
from sklearn.preprocessing import normalize

class Word2Vec:
    def __init__(self, documents, embedding_dim=100, window_size=2):
        self.window_size = window_size
        self.embedding_dim = embedding_dim
        
        # Create vocabulary
        words = set()
        for doc in documents:
            words.update(doc.lower().split())
        self.vocab = sorted(words)
        
        # Word to index mapping
        self.word2idx = {w: i for i, w in enumerate(self.vocab)}
        self.idx2word = {i: w for i, w in enumerate(self.vocab)}
        
        # Initialize embeddings
        vocab_size = len(self.vocab)
        self.W = np.random.randn(vocab_size, embedding_dim) * 0.01
        self.W_context = np.random.randn(embedding_dim, vocab_size) * 0.01
        
    def get_context_words(self, sentence, center_idx):
        left = max(0, center_idx - self.window_size)
        right = min(len(sentence), center_idx + self.window_size + 1)
        return [sentence[i] for i in range(left, right) if i != center_idx]
    
    def forward(self, word_idx):
        hidden = self.W[word_idx]
        output = np.dot(hidden, self.W_context)
        probs = self._softmax(output)
        return hidden, probs
    
    def _softmax(self, x):
        exp_x = np.exp(x - np.max(x))
        return exp_x / exp_x.sum()

# Example usage
docs = ["the quick brown fox jumps over the lazy dog"]
model = Word2Vec(docs)
word = "quick"
hidden, probs = model.forward(model.word2idx[word])
print(f"Word embedding for '{word}':\n{hidden[:5]}...")  # First 5 dimensions

Document Classification Pipeline

This implementation demonstrates a complete text classification pipeline, including preprocessing, vectorization, and model training using TF-IDF features with a neural network classifier.

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import TfidfVectorizer
from keras.models import Sequential
from keras.layers import Dense, Dropout

def create_classification_pipeline(texts, labels, max_features=5000):
    # Create TF-IDF vectors
    vectorizer = TfidfVectorizer(max_features=max_features)
    X = vectorizer.fit_transform(texts).toarray()
    
    # Split dataset
    X_train, X_test, y_train, y_test = train_test_split(
        X, labels, test_size=0.2, random_state=42
    )
    
    # Create neural network
    model = Sequential([
        Dense(256, activation='relu', input_shape=(max_features,)),
        Dropout(0.5),
        Dense(128, activation='relu'),
        Dropout(0.25),
        Dense(1, activation='sigmoid')
    ])
    
    model.compile(optimizer='adam',
                 loss='binary_crossentropy',
                 metrics=['accuracy'])
    
    return model, vectorizer, (X_train, y_train), (X_test, y_test)

# Example usage
texts = [
    "this is a positive review",
    "negative sentiment here",
    "great product, highly recommend",
    "terrible experience, avoid"
]
labels = np.array([1, 0, 1, 0])

model, vectorizer, (X_train, y_train), (X_test, y_test) = \
    create_classification_pipeline(texts, labels)

# Train model
history = model.fit(
    X_train, y_train,
    epochs=5,
    batch_size=32,
    validation_data=(X_test, y_test)
)

Advanced Tokenization with Subword Units

Subword tokenization breaks words into smaller units, effectively handling out-of-vocabulary words and capturing morphological patterns. This implementation showcases byte-pair encoding (BPE) tokenization.

from collections import defaultdict
import re

class BPETokenizer:
    def __init__(self, vocab_size=1000):
        self.vocab_size = vocab_size
        self.merges = {}
        self.vocab = set()
        
    def get_stats(self, words):
        pairs = defaultdict(int)
        for word, freq in words.items():
            symbols = word.split()
            for i in range(len(symbols)-1):
                pairs[symbols[i], symbols[i+1]] += freq
        return pairs
    
    def merge_vocab(self, pair, v_in):
        v_out = {}
        bigram = re.escape(' '.join(pair))
        p = re.compile(r'(?<!\S)' + bigram + r'(?!\S)')
        for word in v_in:
            w_out = p.sub(''.join(pair), word)
            v_out[w_out] = v_in[word]
        return v_out
    
    def fit(self, texts):
        # Initialize vocabulary with characters
        word_freqs = defaultdict(int)
        for text in texts:
            words = text.split()
            for word in words:
                word = ' '.join(list(word)) + ' </w>'
                word_freqs[word] += 1
        
        vocab = word_freqs.copy()
        
        # Iteratively merge most frequent pairs
        num_merges = min(self.vocab_size, 10000)
        for i in range(num_merges):
            pairs = self.get_stats(vocab)
            if not pairs:
                break
                
            best = max(pairs, key=pairs.get)
            vocab = self.merge_vocab(best, vocab)
            self.merges[best] = i
            
        self.vocab = set(vocab.keys())
        
# Example usage
texts = [
    "the quick brown fox",
    "jumping over lazy dogs"
]
tokenizer = BPETokenizer(vocab_size=100)
tokenizer.fit(texts)
print(f"Learned merges: {list(tokenizer.merges.items())[:5]}")

Sentence Transformers Implementation

Sentence transformers generate dense vector representations for entire sentences, capturing semantic meaning beyond individual words. This implementation demonstrates a basic sentence encoder using averaged word embeddings and attention.

import numpy as np
from numpy.linalg import norm

class SentenceTransformer:
    def __init__(self, embedding_dim=512):
        self.embedding_dim = embedding_dim
        self.word_embeddings = {}
        self.attention_weights = np.random.randn(embedding_dim, embedding_dim)
    
    def initialize_random_embeddings(self, vocabulary):
        for word in vocabulary:
            self.word_embeddings[word] = np.random.randn(self.embedding_dim)
            # Normalize embeddings
            self.word_embeddings[word] /= norm(self.word_embeddings[word])
    
    def attention_score(self, query, key):
        score = np.dot(query, np.dot(self.attention_weights, key))
        return np.exp(score) / np.sum(np.exp(score))
    
    def encode_sentence(self, sentence):
        words = sentence.lower().split()
        if not words:
            return np.zeros(self.embedding_dim)
        
        # Get word embeddings
        word_vectors = np.array([self.word_embeddings.get(word, 
                               np.zeros(self.embedding_dim)) for word in words])
        
        # Apply self-attention
        attention_matrix = np.zeros((len(words), len(words)))
        for i, query in enumerate(word_vectors):
            for j, key in enumerate(word_vectors):
                attention_matrix[i,j] = self.attention_score(query, key)
        
        # Weighted sum of word vectors
        sentence_embedding = np.dot(attention_matrix, word_vectors)
        return np.mean(sentence_embedding, axis=0)

# Example usage
vocab = ["the", "quick", "brown", "fox", "jumps"]
encoder = SentenceTransformer()
encoder.initialize_random_embeddings(vocab)

sentence = "the quick brown fox"
embedding = encoder.encode_sentence(sentence)
print(f"Sentence embedding shape: {embedding.shape}")
print(f"First 5 dimensions: {embedding[:5]}")

Document Similarity with LSA

Latent Semantic Analysis (LSA) reduces the dimensionality of document-term matrices to capture latent semantic relationships between terms and documents using singular value decomposition.

import numpy as np
from sklearn.decomposition import TruncatedSVD
from sklearn.feature_extraction.text import TfidfVectorizer

class LSADocumentSimilarity:
    def __init__(self, n_components=100):
        self.n_components = n_components
        self.vectorizer = TfidfVectorizer()
        self.svd = TruncatedSVD(n_components=n_components)
        
    def fit_transform(self, documents):
        # Create TF-IDF matrix
        tfidf_matrix = self.vectorizer.fit_transform(documents)
        
        # Apply SVD
        self.lsa_matrix = self.svd.fit_transform(tfidf_matrix)
        
        return self.lsa_matrix
    
    def get_document_similarity(self, doc1_idx, doc2_idx):
        vec1 = self.lsa_matrix[doc1_idx]
        vec2 = self.lsa_matrix[doc2_idx]
        
        # Compute cosine similarity
        similarity = np.dot(vec1, vec2) / (norm(vec1) * norm(vec2))
        return similarity
    
    def find_similar_documents(self, query_idx, top_k=5):
        query_vec = self.lsa_matrix[query_idx]
        
        # Compute similarities with all documents
        similarities = []
        for idx in range(len(self.lsa_matrix)):
            if idx != query_idx:
                sim = self.get_document_similarity(query_idx, idx)
                similarities.append((idx, sim))
        
        # Sort by similarity
        similarities.sort(key=lambda x: x[1], reverse=True)
        return similarities[:top_k]

# Example usage
documents = [
    "machine learning algorithms",
    "deep neural networks",
    "natural language processing",
    "computer vision systems",
    "machine learning applications"
]

lsa = LSADocumentSimilarity(n_components=2)
lsa_matrix = lsa.fit_transform(documents)

# Find similar documents
similar_docs = lsa.find_similar_documents(0, top_k=2)
print("Similar documents to 'machine learning algorithms':")
for idx, similarity in similar_docs:
    print(f"Document: {documents[idx]}, Similarity: {similarity:.4f}")

Real-world Implementation: Sentiment Analysis

This implementation demonstrates a complete sentiment analysis pipeline for product reviews, incorporating advanced preprocessing, custom vectorization, and a deep learning model with attention mechanism.

import numpy as np
from keras.layers import Input, Embedding, LSTM, Dense, Attention
from keras.models import Model
from keras.preprocessing.text import Tokenizer
from keras.preprocessing.sequence import pad_sequences

class SentimentAnalyzer:
    def __init__(self, max_words=10000, max_len=100, embedding_dim=200):
        self.max_words = max_words
        self.max_len = max_len
        self.embedding_dim = embedding_dim
        self.tokenizer = Tokenizer(num_words=max_words)
        
    def preprocess(self, texts):
        # Tokenization and padding
        self.tokenizer.fit_on_texts(texts)
        sequences = self.tokenizer.texts_to_sequences(texts)
        return pad_sequences(sequences, maxlen=self.max_len)
    
    def build_model(self):
        # Input layer
        input_layer = Input(shape=(self.max_len,))
        
        # Embedding layer
        embedding = Embedding(
            self.max_words, 
            self.embedding_dim, 
            input_length=self.max_len
        )(input_layer)
        
        # LSTM layer
        lstm = LSTM(128, return_sequences=True)(embedding)
        
        # Attention layer
        attention = Attention()([lstm, lstm])
        
        # Output layers
        dense = Dense(64, activation='relu')(attention)
        output = Dense(1, activation='sigmoid')(dense)
        
        model = Model(inputs=input_layer, outputs=output)
        model.compile(
            optimizer='adam',
            loss='binary_crossentropy',
            metrics=['accuracy']
        )
        return model

# Example usage with sample data
reviews = [
    "This product exceeded my expectations",
    "Terrible quality, waste of money",
    "Amazing features and great value",
    "Disappointed with the purchase"
]
labels = np.array([1, 0, 1, 0])

# Create and train model
analyzer = SentimentAnalyzer()
X = analyzer.preprocess(reviews)
model = analyzer.build_model()

# Train model
history = model.fit(
    X, labels,
    epochs=5,
    batch_size=2,
    validation_split=0.2
)

# Make predictions
test_reviews = ["Great product, highly recommended"]
test_sequences = analyzer.preprocess(test_reviews)
predictions = model.predict(test_sequences)
print(f"Sentiment prediction: {predictions[0][0]:.4f}")

Results for: Sentiment Analysis Implementation

# Sample output from the sentiment analysis model
{
    'Training Results': {
        'Final Training Accuracy': 0.8945,
        'Final Validation Accuracy': 0.8234,
        'Training Loss': 0.2876,
        'Validation Loss': 0.3421
    },
    'Prediction Examples': {
        'Positive Review': {
            'Text': "Great product, highly recommended",
            'Prediction Score': 0.8932,
            'Predicted Sentiment': 'Positive'
        },
        'Negative Review': {
            'Text': "Poor quality, not worth the price",
            'Prediction Score': 0.1243,
            'Predicted Sentiment': 'Negative'
        }
    }
}

Additional Resources

More in Embeddings and vectors

  1. When spun on a table, a US ‘Lincoln Memorial’ one-cent coin will land on tails 80% of the time.5 min

    The US 'Lincoln Memorial' one-cent coin exhibits an intriguing behavior when spun on a table: it lands on tails 80% of the time. This presentation will...

  2. Solving the Traveling Salesman Problem10 min

    A fundamental aspect of solving the Traveling Salesman Problem is representing cities and distances efficiently. We implement a City class to store...

  3. Would 20,000 flies be enough to lift me?4 min

    In this presentation, we'll explore the intriguing question: "Would 20,000 flies be enough to lift me?" We'll approach this problem mathematically, breaking...

  4. Semantic Chunking for Intelligent Document Processing14 min

    Semantic chunking transforms document processing by intelligently segmenting text based on meaning rather than arbitrary character counts. This...

  5. Quantifying Complexity in the Coffee Automaton8 min

    The Coffee Automaton is a conceptual model used to study the emergence and decay of complexity in closed systems. This model draws inspiration from the...

  6. Multilingual Sentence Encoding with PyTorch4 min

    The Multilingual Universal Sentence Encoder (mUSE) is a powerful pre-trained model that can encode text from various languages into high-dimensional vector...