Embeddings and vector space

An embedding is a list of numbers that stands for a piece of text. You can read that list as the coordinates of a point in space, and the whole trick is that the space is arranged so that things close in meaning sit close together. Similar ideas land near each other; unrelated ideas land far apart. Meaning becomes location.

Turning meaning into numbers

A computer cannot compare two sentences for meaning the way a person does. It has no sense of what “a small dog” and “a little puppy” have in common. All it can do quickly and reliably is arithmetic on numbers. So if you want a machine to judge whether two pieces of text are about the same thing, you first have to turn each piece of text into numbers in a way that reflects its meaning. That is exactly what an embedding is: a procedure takes in some text and returns a fixed length list of numbers, and that list is built so its values carry the sense of the text.

The important word is “reflects.” Turning text into numbers is easy if you do not care what the numbers mean; you could just number the words alphabetically. The hard and useful thing is producing numbers where closeness in the numbers means closeness in meaning. An embedding for “cat” and an embedding for “kitten” should come out nearly the same, while an embedding for “cat” and one for “invoice” should come out clearly different. When the numbers behave that way, meaning has been made measurable.

A list of numbers is a point in space

The step that makes embeddings intuitive is realizing that a list of numbers is the same thing as a location. Two numbers, like 3 and 5, are coordinates on a flat map: go 3 across and 5 up, and you are at a point. Three numbers place a point in the room around you: across, up, and depth. The list does not have to stop at three. Any list of numbers picks out a single point in a space that has one direction for each number in the list.

A list of numbers used this way is called a vector, and the space it lives in is a vector space. Once you see an embedding as a point, the earlier claim becomes a statement about geography. Arranging the numbers so similar meanings get similar values is the same as arranging the space so similar meanings land at nearby points. “Cat” and “kitten” are neighbors. “Cat” and “invoice” are in different neighborhoods. The embedding is the address; the vector space is the city.

Making peace with many dimensions

Real embeddings are not two or three numbers long. They are typically hundreds or even thousands of numbers long. That means the space has hundreds or thousands of directions, and no one can picture that. You do not need to. The rules that matter carry over unchanged from the flat map you can picture.

On a flat map, each direction can capture something. Imagine one axis running from cold to hot and another from cheap to expensive; every product gets a spot, and spots near each other share both traits. A high dimensional space is just that idea with far more axes, so it can capture far more shades of meaning at once: how formal a phrase is, whether it concerns animals, whether it is a question, what topic it belongs to, and hundreds of subtler things all at the same time, each on its own direction. You cannot draw it, but you can reason about it. Two points are close when they agree along many of those directions, and the same distance arithmetic that works on a map works, unchanged, no matter how many directions there are.

You do not have to see a thousand dimensional space. You only have to trust that “close” still means “similar” in it, exactly as it does on a map.

Measuring how close two meanings are

If meaning is location, then judging similarity is measuring distance, and there is a specific measure that fits embeddings well. You could measure the plain straight line distance between two points, and that is sometimes used. More often the measure is cosine similarity, and the idea behind it is simpler than the name.

Picture each point not just as a dot but as an arrow from the center of the space out to that dot. Every piece of text is now an arrow pointing in some direction. Cosine similarity asks a single question: do these two arrows point the same way? If two arrows point in nearly the same direction, the texts are judged very similar, and the score is near one. If they point at right angles, unrelated, the score is near zero. If they point in opposite directions, the score goes negative. The appeal is that this cares about direction, not length. A short phrase and a long passage about the same topic point the same way even though one arrow is longer, so they score as similar, which is what you want. The one thing to hold onto is that a higher cosine score means more alike, and that this single number lets a machine rank any pile of texts by how close each is in meaning to some target.

A tiny worked example

Shrink the space down to something you can hold in your head. Suppose an embedding had only two numbers, and suppose, purely to see the shape of it, that the first number tracked how much a word is about animals and the second how much it is about money. “Kitten” might land near the point 9 and 1: very much an animal, almost nothing to do with money. “Puppy” might land near 9 and 0, right next to it. “Invoice” might land near 0 and 9: nothing to do with animals, very much about money. Plot those three points and “kitten” and “puppy” are all but on top of each other, while “invoice” is off in a far corner.

Now the similarity measure does real work. The arrows to “kitten” and “puppy” point in nearly the same direction, so their cosine similarity is close to one, and a machine reports them as strongly related without ever being told they are both young animals. The arrow to “invoice” points almost at a right angle to both, so its similarity to each is near zero, and the machine reports it as unrelated. Real embeddings do exactly this, only with hundreds of numbers instead of two, so instead of just animals and money the space can weigh hundreds of aspects at once. The arithmetic and the conclusion are the same; there are simply more directions feeding into how the arrows line up.

The famous king and queen example

There is one result that first showed people these spaces hold real structure, and it is worth walking through because it makes the whole idea vivid. Take the embedding for “king,” subtract the embedding for “man,” and add the embedding for “woman.” Because these are just lists of numbers, you can literally do that arithmetic, number by number. The point you land on comes out extremely close to the embedding for “queen.”

Read what that means. The direction you travel when you go from “man” to “woman” in the space is roughly the same direction you travel when you go from “king” to “queen.” The space has, without anyone designing it that way, laid out a consistent direction that corresponds to the idea of gender, and another that corresponds to royalty, so that these relationships become movements you can add and subtract. Similar patterns show up elsewhere: the step from a country to its capital tends to be a consistent direction too. Nobody labeled any of this. It fell out of arranging the space to reflect meaning. That is the moment the numbers stop looking arbitrary and start looking like a map of concepts.

How the arrangement gets learned

No person places these points by hand; there are far too many, and no one could decide the thousand coordinates of “king.” The arrangement is learned from how words are used, and the guiding principle is old and simple: words that appear in similar contexts tend to mean similar things. “Cat” and “dog” show up around the same neighbors, “pet,” “feed,” “vet,” “fur,” while “cat” and “invoice” almost never share a sentence.

A training procedure turns that principle into positions. It starts every piece of text at a random point, then works through a mountain of real text, and each time it sees which words actually keep company with which, it nudges the points: words that share contexts get pulled a little closer, words that do not get pushed a little apart. Repeat that across enormous amounts of text and the points settle into an arrangement where distance tracks meaning, and where the consistent directions like the gender step emerge on their own. The same idea extends past single words. A model can embed a whole sentence, a paragraph, or a chunk of source code, placing the entire passage at one point according to its overall meaning. That is what makes embeddings practical for real documents rather than just single words.

What embeddings capture, and what they miss

Embeddings are strong at a specific thing: rough, overall topical and semantic closeness. They will happily tell you that a paragraph about training a puppy is near a paragraph about dog obedience and far from a paragraph about tax law, even when the two dog paragraphs share almost no exact words. That ability to see past the exact wording to the gist is the whole reason they are useful, and it is genuinely hard to get any other way.

They are also lossy, and it helps to be honest about how. Compressing a passage into a few hundred numbers throws detail away. Fine distinctions can blur, so two texts that are broadly on the same subject but disagree on a crucial detail may still sit close together, because the embedding caught the topic and lost the disagreement. A negation can be treated as similar to the thing it negates, since “the deploy succeeded” and “the deploy did not succeed” share almost every word and most of their context. And an embedding reflects only what its training text showed; it has no idea about facts or code specific to your private world that it never saw. Embeddings give you a reliable sense of “these are in the same area,” not a guarantee of “these say the same thing.”

This is worth keeping in mind whenever embeddings are used on source code, where the difference between “in the same area” and “the same thing” can be sharp. Two functions can embed close together because they are written in a similar style and talk about similar things, and still do opposite work. Two functions that do nearly identical work can embed farther apart because one is written in a different language or uses different names. An embedding sees the surface of the text and its learned sense of topic. It does not run the code, follow a call from one function to another, or know that a rename elsewhere just changed what a line means. Those are relationships of structure and behavior, not of wording, and they are the kind of thing an explicit map of the code has to supply, because an embedding, by its nature, will not.

The substrate for finding things by meaning

Step back and the payoff is clear. Once every piece of text you care about has been placed at a point in the same space, a question you could never answer by matching words becomes easy. Embed a query, then look for the nearest points. Because near means similar in meaning, the nearest points are the passages most related to your query, even if they use entirely different words. You have turned “find text that means roughly this” into “find the closest points,” which is just arithmetic a machine does fast.

That move, from meaning to coordinates to nearest neighbors, is the foundation under a great deal of what modern systems do with text and code, and it is the reason embeddings show up again and again in this hub. Doing the nearest point lookup quickly across millions of embeddings, and knowing where it shines and where plain exact matching still wins, is a subject of its own, which is vector search and similarity. And the very same notion of turning meaning into positions and then weighing what relates to what sits at the heart of the architecture that powers today’s models, which is the transformer and attention.