Part 2 of 7 · 3 min read
Embeddings: meaning as direction
How a computer can tell that "tea" and "coffee" are alike, and "tea" and "cat" are not.
In module 1 you saw that a model reads token numbers. But token number 4,017 is just a label. Nothing about the number says that it is close in meaning to token 9,312. To work with meaning, each token is replaced by a longer list of numbers where similar things get similar lists. This is called an embedding. Three words first.
Vector
A list of numbers, such as [0.9, 0.0, 0.0]. You can picture it as a point, or an arrow from the origin, in a space with as many dimensions as there are numbers in the list.
Embedding
A vector that represents a piece of data (a token, a word, a sentence, an image) so that things with similar meaning get nearby vectors. Embeddings are learned: the model adjusts them during training until useful patterns appear.
Cosine similarity
A score from -1 to 1 for how closely two vectors point in the same direction, ignoring their length. 1 means the same direction (very similar), 0 means at right angles (unrelated), and -1 means opposite. It is the standard way to compare embeddings.
Real embeddings have hundreds or thousands of numbers and no axis that a person could name. To see the idea, the example code (code/mini-attention) uses a hand-made table with three named axes: technology, animal and drink. These numbers are invented for teaching; no model produced them.
| Word | technology | animal | drink |
|---|---|---|---|
tea |
0.0 | 0.0 | 0.90 |
coffee |
0.0 | 0.0 | 0.95 |
cat |
0.0 | 0.90 | 0.0 |
code |
0.95 | 0.0 | 0.0 |
python |
0.60 | 0.55 | 0.0 |
java |
0.60 | 0.0 | 0.50 |
The function that compares two vectors is one line: multiply them element by element and add up (the dot product, a @ b), then divide by the two lengths so that only the direction counts:
def cosine_similarity(a, b):
return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))
Here are the real results:
| Pair | Cosine similarity | Reading |
|---|---|---|
tea, coffee |
1.00 | Both point only along the drink axis |
tea, cat |
0.00 | Different axes: unrelated |
tea, python |
0.00 | Different axes: unrelated |
python, code |
0.74 | Python is partly a technology word |
python, snake |
0.68 | Python is also partly an animal word |
python, java |
0.57 | Both are partly technology words |
python and java each sit between two meanings, because each word has more than one. The next section shows what the model does about that.
Inside a language model, each token number first looks up its row in a big table of embeddings, and those vectors are then refined layer by layer. Separately, you can use embedding models (smaller models you call through an API) whose only job is to turn a piece of text into one vector you can store and search. That is what Level 4 does.
Check yourself
- Why is cosine similarity 0 for
teaandcatin the table? - Why does a real embedding not have axes named "drink" or "animal"?