Embeddings

From symbols to vectors

Computers handle symbols, words, product IDs, categories, as discrete items with no notion of similarity. To a program, "dog" and "canine" are as different as "dog" and "carburetor." Machine learning needs a representation where related things are near each other, so that what the model learns about one generalizes to the other.

An embedding is exactly that: a mapping from each discrete item to a dense vector, a list of a few hundred or thousand numbers, learned from data. Items that appear in similar contexts end up with similar vectors. The geometry of the space carries meaning. This is the representation almost every neural network uses internally, and it is worth understanding on its own because the same idea powers search, recommendations, and retrieval systems.

Word2vec and the distributional idea

The modern embedding era started with word2vec, published by Mikolov and colleagues at Google in 2013. The underlying principle is older than computing: you shall know a word by the company it keeps. Words that show up in similar surroundings tend to mean similar things.

Word2vec trains a shallow network on one of two tasks. In the continuous bag-of-words setup, the network predicts a word from its neighbors. In skip-gram, it predicts the neighbors from the word. Neither task is useful on its own. The point is the side effect: to do well, the network's internal weight matrix must assign similar vectors to words that appear in similar contexts. After training, that matrix is thrown away as a network and kept as a lookup table. Each row is a word's embedding.

The famous party trick is vector arithmetic. Take the vector for "king," subtract "man," add "woman," and the nearest vector is "queen." This works because the training process arranges the space so that consistent relationships, like gendered pairs or capital cities, point in consistent directions. It does not work for every analogy, and cherry-picked examples made it look more reliable than it is, but the underlying regularity is genuine.

Static versus contextual

Word2vec gives each word one vector, always the same. That breaks on words with multiple senses. The vector for "bank" has to cover both rivers and finance, so it lands somewhere unhelpful between them.

Contextual embeddings fix this by making the vector depend on the surrounding text. Models like ELMo (2018) and BERT (2018) run the whole sentence through a network and produce a different vector for "bank" in "river bank" than in "bank account." This was a large part of why those models outperformed everything before them: they stopped pretending words have fixed meanings.

In a transformer, embeddings are just the starting point. Each token's static embedding goes into the first layer, and every layer refines it with context from the rest of the sequence. By the final layer, the vector for a token encodes its role in that specific sentence. The embedding matrix itself, of shape vocabulary size by model dimension, is learned alongside everything else during training.

Similarity and search

Once items are vectors, similarity becomes geometry. The standard measure is cosine similarity: the cosine of the angle between two vectors, ranging from -1 (opposite) through 0 (unrelated) to 1 (identical direction).

```python

import numpy as np

def cosine_similarity(a, b):

return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

# toy 3-dimensional embeddings

cat = np.array([0.9, 0.2, 0.1])

dog = np.array([0.8, 0.3, 0.2])

car = np.array([0.1, 0.9, 0.8])

print(cosine_similarity(cat, dog)) # high: related animals

print(cosine_similarity(cat, car)) # low: unrelated

```

This simple operation is the engine of semantic search. Embed a user's query, embed every document in a collection, and return the documents with the highest cosine similarity to the query. No keywords need to match. A query about "cheap places to stay" can retrieve a page about "affordable hotels" because their vectors point in similar directions. Retrieval-augmented generation builds directly on this.

Beyond words

The same trick applies to anything discrete. Recommendation systems embed users and products into one shared space, so that a user's vector sits near the products they would like. Image models embed pictures and captions together, which is how text-to-image search works. Code models embed functions by what they do rather than what they are named.

The embedding matrix is often the largest single parameter block in a model, since its size is vocabulary times dimension. It is also the model's interface with the world: every input passes through it on the way in, and the output layer is usually its mirror image on the way out.

Dimension tradeoffs

Embedding size is a setting, not a law of nature, and it involves a real tradeoff. More dimensions can capture finer distinctions: with 1,500 dimensions there is room for subtleties that 300 dimensions must blur together. But bigger vectors cost more memory, slow down similarity search, and need more training data to fill in meaningfully. Small models often use a few hundred dimensions; large ones use a few thousand. Matryoshka embeddings, a newer technique, train the vector so that its first N dimensions are useful on their own, letting users truncate to whatever size fits their budget without retraining. The right size is whatever captures the distinctions your task needs and no more.

What embeddings do not capture

An embedding space reflects the data it was trained on, including the data's biases and gaps. If the training text associates certain professions with one gender, the geometry will too, and downstream systems inherit it. Embeddings also compress ruthlessly: a thousand numbers cannot hold everything about a word, and fine distinctions get lost. They are a useful approximation of meaning, not a theory of it.