The transformer and attention

The transformer is the design that nearly all modern language models are built on. Its central idea, called attention, is a way of letting every word in a passage look at every other word and decide, for itself, which of them matter for understanding it. That one idea is most of why these models handle long, tangled text as well as they do.

What an architecture even is

Before the transformer, a word about the word “architecture,” because it is used loosely. In this setting an architecture is the blueprint of the model: the arrangement of parts, what each part does, and how information flows from one to the next. It is not the knowledge the model ends up with, which lives in the trained numbers, the parameters. It is the shape those numbers are poured into. The same architecture can be trained on different text to get different models, the way the same floor plan can be built into different houses. When people say a model is “a transformer,” they mean it is laid out according to this particular blueprint.

Why the blueprint matters so much is that it decides what the model can even attempt. A design that can only look at words one at a time, in order, will struggle with meaning that jumps across a long passage, no matter how much you train it. A design that lets any word draw on any other word has a fighting chance at that. The transformer is the second kind, and understanding why that was the breakthrough means understanding the problem it was built to solve.

The problem: meaning depends on faraway words

Language is full of connections that reach across a sentence or a whole document. Take the sentence: “The trophy did not fit in the suitcase because it was too big.” What does “it” refer to, the trophy or the suitcase? A person knows instantly it is the trophy, because a thing that is too big is the thing that fails to fit. But the word “it” is nowhere near the word “trophy,” and the answer depends on holding several words in mind at once and relating them. Change one word, “because it was too small,” and now “it” is the suitcase. The meaning of a single small word hinges on words scattered elsewhere in the sentence.

This is the general problem, and it has a name: long range dependencies. To understand any word, you often need context that sits far away from it, sometimes paragraphs away. Earlier designs handled text like a person reading through a straw, one word at a time, carrying a running summary forward. That works for nearby context, but the summary blurs as it travels, and by the time you are far downstream the early words have faded. Important connections across long distances got lost. The question the transformer answers is direct: how can every word reach any other word it needs, no matter how far away, without the signal fading on the way?

Attention as weighted looking

Attention is the answer, and the plainest way to describe it is that each word gets to look at all the other words at once and pull in what it finds relevant, ignoring the rest. Not one at a time, not through a fading summary. All of them, directly, in parallel.

An analogy makes the mechanism concrete. Imagine you are trying to understand one word in a sentence, and you can ask every other word a question: “are you relevant to me?” Each other word answers with a strength, high if it matters to you, low if it does not. Back at “it” in the trophy sentence, the word “trophy” answers loudly, “too” and “big” answer loudly, and words like “the” and “in” answer softly. You then build your understanding of “it” as a blend of the other words, weighted by how loudly each answered. The loud ones shape your sense of “it” a lot; the quiet ones barely at all. That weighted blend is attention. Every word does this for itself, at the same time, so the whole passage rewires itself around what actually relates to what.

The weights are not fixed rules. They are computed from the words themselves, using the trained parameters, and they change with every passage. In one sentence “it” leans on “trophy”; in the next it leans on “suitcase.” Nobody wrote “pronouns look back to nouns.” The model learned, from mountains of text, how to set these weights so that the relevant words light up. And crucially, a far away word is no harder to reach than a nearby one. Distance does not weaken the signal, because every word is looked at directly. The straw is gone.

Attention lets every word ask every other word how much it matters, and rebuild its own meaning from the answers. Nearness in the text stops being what decides nearness in meaning.

Many kinds of looking at once

There is a refinement to the picture that matters, because a single weighted blend is too blunt on its own. When you read a word, you are not tracking just one kind of relationship. You are tracking several at the same time: which noun a pronoun points back to, which verb an object belongs to, what the overall topic is, whether the tone is a question or a statement. One set of attention weights can only express one of these patterns at a time, since it is a single answer to “which words matter to me.”

So the transformer runs several attention operations side by side rather than one, and each is free to specialize. In the trophy sentence, one of them might learn to link pronouns to the nouns they stand for, so at “it” that one lights up “trophy.” Another might learn to connect a description like “too big” to the thing being described. Another might mostly track the subject of the sentence. Each produces its own weighted blend, and their results are combined, so a single word ends up informed by several different kinds of relationship at once. This is called multi head attention, where each parallel operation is a “head.” You do not need the term to keep going. The idea to hold is that the model does not look in one way; it looks in many ways at the same time and merges what it finds, which is a large part of why one round of attention can capture something as tangled as real language.

One look is not enough: stacking layers

A single round of this weighted looking is useful but shallow. It can catch that “it” relates to “trophy,” but real understanding needs several passes that build on each other. So the transformer stacks the operation. The output of one round of attention, an updated sense of every word colored by its relevant neighbors, becomes the input to another round, and that to another, dozens of times. These repetitions are the model’s layers.

You can think of the layers as understanding forming in stages. An early layer might just sort out which words group together and what refers to what. A middle layer, working on those cleaned up pieces, might assemble the meaning of a phrase or track who did what to whom. A later layer, working on those, might grasp the point of the whole sentence and what should come next. Each layer looks again with fresh weights at the enriched version the previous layer produced, so understanding deepens the further up the stack you go. The number of layers is a big part of what people mean by a model’s depth, and deeper stacks, given enough training, tend to capture more.

Why this design could be scaled

A quiet feature of attention turned out to be decisive for its success: it does its work on all the words at the same time, not in sequence. Because each word looks at every other word in one simultaneous operation, the heavy computation can be spread across many processors working in parallel. The older, read through a straw designs could not do this, since each step depended on the summary from the step before, forcing the work into a strict order that no amount of hardware could speed past.

This matters because it is what let these models be trained at all at the sizes that made them capable. The specialized chips used for training are fast precisely when a task can be split into many pieces that run at once. The transformer is that kind of task. So as people built bigger models on more text, the design kept the hardware busy and the training stayed practical. The earlier idea that scaling data and parameters together improves a model needed an architecture that could actually be scaled that far, and the transformer’s parallel nature is a large part of why it, and not some other design, is what got scaled.

The cost hidden in “every word looks at every word”

The strength of attention carries a price, and it is worth understanding because it explains a real limit you will run into. If every word looks at every other word, then the amount of looking grows faster than the length of the text. Double the number of words and you do not double the work; you roughly quadruple it, because each of the now twice as many words has twice as many others to look at. Ten words means on the order of a hundred pairwise looks; a thousand words means on the order of a million; ten thousand words, a hundred million. The work climbs with the square of the length.

This is why a language model cannot simply be handed unlimited text. There is a ceiling on how many tokens it can attend to at once, its context window, and the steep growth of attention cost is a central reason that ceiling exists and why raising it is expensive rather than free. The very mechanism that lets any word reach any other is the same mechanism that makes very long inputs costly, because reaching everything means comparing everything. Engineers have found clever ways to soften this cost, but the basic pressure remains: attention is powerful because it is all to all, and it is expensive for exactly the same reason.

Putting the pieces together

Trace the path once from end to end. Text arrives and is cut into tokens. Each token is turned into a list of numbers that places it in a space of meaning, an embedding, so the model starts with a numeric sense of each piece. Then the stack of transformer layers goes to work: layer after layer of attention, each letting every token look at every other and update its meaning from the relevant ones, understanding deepening as it climbs. At the top, the model uses this richly contextual picture to do the one thing it was trained for, predict the next token, and the loop begins again.

That is the transformer: an architecture whose one distinctive move is letting everything attend to everything, stacked deep and run in parallel. It is why modern models can hold a thread across a long passage, and, through the cost of all that looking, it is also why they cannot hold an unlimited one. The ceiling that cost creates, how much a model can keep in view at once and why it matters so much for working with real code, is the subject of the context window. And the process that sets all the weights inside this blueprint in the first place, turning a blank architecture into a model that knows things, is the subject of training and inference.

It is worth being clear about one thing attention does not do, since the name invites the wrong idea. Attention lets a word draw on any other word in the text placed in front of the model right now. It does not let the model reach outside that text to your files, your history, or the world. Everything a word can attend to has to already be present in the input. That is why so much of the effort in working with these models goes into deciding what to put in front of them, and it is the thread that runs from here into the ideas about the context window and about giving a model facts it never learned.