Training and inference

Training is the slow, expensive process of fitting a model to a huge pile of data, done once and then occasionally repeated. Inference is the fast, cheap process of using the finished model to answer a request, done millions of times. They are two entirely different activities, and confusing them is behind most of the misunderstandings people have about what a model can and cannot know.

Two activities, not one

When people say a model learned something, they usually blur two moments together that could not be more different. The first moment is training: the model is built by exposing it to an enormous amount of text and adjusting it, gradually, until it gets good at predicting that text. The second moment is inference: the finished model is handed a new input and produces an output. Training makes the model. Inference uses it.

A useful comparison is a printed dictionary. Writing and printing the dictionary is an immense effort: lexicographers research every word, editors argue over definitions, the whole thing is typeset and pressed. That is training. Looking a word up in the finished dictionary is instant, and it does not change the book. That is inference. Every reader who opens the same dictionary gets the same definitions, because the content was fixed at printing time. Nothing a reader does while looking things up gets written back into the pages.

This is the single most important consequence to hold on to, and we will return to it: using the model does not teach the model. Inference is a read. It leaves the model exactly as it was.

How training works, at an intuitive level

A model is, underneath, a very large collection of numbers called parameters, often billions of them. At the start of training these numbers are essentially random, and the model is useless: given some text and asked to predict what comes next, it produces nonsense. Training is the process of nudging those billions of numbers, over and over, until the predictions stop being nonsense and start being good.

The nudging follows a simple loop. Show the model a snippet of real text with the next piece hidden. Let it predict. Compare its prediction to the piece that was actually there. Measure how wrong it was, a single number called the loss. Then adjust every parameter a tiny amount in the direction that would have made the loss smaller. Repeat with the next snippet, and the next, across a staggering amount of text.

The method for deciding which direction to nudge each parameter is called gradient descent. The intuition is a hiker trying to reach the bottom of a valley in thick fog. They cannot see the valley floor, but they can feel which way the ground slopes under their feet, and they take a small step downhill. Feel the slope again, step again. Each step alone is tiny and local, but millions of small downhill steps carry them a long way down. In training, downhill means lower loss, better predictions, and each step adjusts the parameters by a small amount. The gradient is just the local slope: which way, and how steeply, the error changes if you nudge each number.

Do this enough times, over enough text, and something remarkable happens. To predict the next word well across billions of examples, the model is forced to internalize the patterns behind the text: grammar, facts that recur, the shape of an argument, the conventions of code. Nobody programs those patterns in by hand. They are squeezed out of the data by the relentless pressure to predict a little better on each step.

It is worth sitting with how indirect this is. At no point does anyone tell the model a rule of grammar or a fact of history. The only instruction it ever gets, billions of times over, is predict the next piece of text a little better than last time. Everything the model appears to know is a byproduct of getting good at that one narrow task. This is why its knowledge is broad but fuzzy, and why it has no clean line between what it knows for certain and what it is merely inclined to say. There was never a step that sorted solid facts from likely guesses. There was only the slope, and the many small steps down it.

Why training is a one-time, giant cost

Training a serious model is one of the most expensive things done in computing. It runs across thousands of specialized processors working in parallel for weeks or months, consuming an amount of electricity and hardware that costs a great deal of money. The reason it is so costly is the sheer scale: billions of parameters, each nudged across a colossal number of text snippets, every nudge requiring calculation.

Crucially, this cost is paid up front and only occasionally repeated. You train the model once, and then it is done, frozen, ready to use. Later you might train a new version to bring in newer data or improve quality, but that is a fresh, scheduled, expensive project, not something that happens continuously. Between those periodic efforts, the model sits fixed. It is far more like printing a new edition of the dictionary every so often than like a notebook you jot in daily.

Training is paid once and rarely repeated. Inference is paid on every request. The model you use today is frozen at the moment its training ended.

Why inference is a per-request cost

Inference is the opposite shape. Each time you send a request, the model runs a single forward pass over your input to produce an output, then it is done with you and ready for the next request. That pass is cheap and quick compared to training, but you pay it every single time, and the bill scales with how much you use it. A service answering millions of requests pays the inference cost millions of times over, while it paid the training cost just once.

This is why the two costs are budgeted and reasoned about completely separately. Training is a capital project. Inference is an ongoing operating expense, priced by how much text goes in and comes out. It also explains why keeping the input lean matters so much in practice: training is already spent and cannot be changed by you, but inference cost is paid on your text, on every call.

The frozen-knowledge consequence

Now the payoff. Because a model is fixed at the moment its training ends, its built-in knowledge is frozen at that moment too. Everything it absorbed came from the data it was trained on, and that data has a cutoff date. Anything that happened after the cutoff simply is not in the model. It did not learn about last week’s news, because last week had not happened when the parameters were frozen. Ask it about a recent event and, from its own knowledge, it cannot know.

The same logic applies, even more sharply, to anything private. Your company’s internal codebase was never in the training data. Your unreleased product, your team’s decisions, the function you wrote this morning: none of it was present when the model was trained, so none of it is in the model. This is not a gap that using the model more will close, because, as we established, inference does not teach the model. You could ask it a thousand questions about your code and it would end each conversation knowing exactly as much about your code as it did at the start, which is nothing.

So a model has two hard blind spots that follow directly from how it is built: it cannot know what happened after its training cutoff, and it cannot know anything private that was never in its training data. Both blind spots are permanent features of a frozen model, not bugs to be patched by clever wording.

A worked example of the split

Walk through a single realistic moment to see both activities in their places. Some months ago, a company trained a model. Over weeks, across thousands of processors, gradient descent nudged its billions of parameters down the loss valley until it predicted text well. At the end of that run the parameters were frozen and shipped. That whole effort was training, and it happened once, before you ever touched the model.

Today you open your editor and ask an assistant built on that model to explain a function you wrote this morning. The model runs a forward pass over your request and produces an explanation. That is inference. It took a second, it cost a fraction of a cent, and it changed nothing about the model. The parameters that came out of that months-old training run are the exact same parameters after your request as before it.

Two facts follow immediately and they often confuse people. First, the model could explain your brand new function even though your function did not exist when the model was trained. It managed that only because the function’s text was in your request, at inference time, for it to read. It did not know your function. It read your function. Second, if you close the editor and a colleague asks the same model about the same function tomorrow, the model will not remember a thing about it, because your inference call taught it nothing. Every request stands alone against the same frozen parameters.

Two ways to add knowledge: fine-tuning and prompting

If the model does not know something you need, there are two broad ways to get that knowledge to it, and they map exactly onto the two activities.

The first is fine-tuning, which means doing more training. You take the finished model and run additional training steps on new material, nudging the parameters again so the new patterns get baked in. This changes the model itself. It is powerful for teaching a durable style or skill, but it inherits the character of training: it is slow, it needs real expertise and a curated dataset, it costs money, and the moment you do it the result is frozen again. Fine-tune a model on your codebase today and tomorrow’s changes are already outside it. For knowledge that moves, fine-tuning is a treadmill.

The second is prompting, which means supplying the facts at inference time, inside the request itself. You do not touch the parameters. You simply include the relevant information in the input so the model can read it while it answers. This changes nothing about the model and everything about the answer, because the model works only from what is in front of it. The knowledge lives in the request, is used for that request, and is gone again after, ready to be different next time.

Why you feed facts in rather than retrain

Line the two up against the blind spots and the choice is usually obvious. The knowledge that matters most for real work is the knowledge that moves: today’s events, this week’s code, the decision made an hour ago. Fine-tuning bakes knowledge in and then freezes it, so for anything that changes it is always behind. Prompting delivers knowledge fresh on every request, so it is always current by construction.

That is the deep reason modern systems lean so heavily on feeding facts at inference time instead of retraining. It is faster, it is cheaper, and above all it stays current. When a coding assistant seems to understand your specific project, it is almost never because the model was trained on your project. It is because the relevant slice of your project was placed into the request before the model answered. The model supplies the general ability. The request supplies the specific facts.

The model brings general ability. The request brings the specific, current facts. Good results come from getting the right facts into the request, not from retraining the model.

This reframes the whole practical problem. If the answer to a frozen model’s blind spots is to supply the right facts at inference time, then the pressing questions become: which facts, and how do they get chosen and delivered on time? That is exactly what the wider picture is about. Understanding what a large language model is underneath makes the frozen-parameter story concrete, and the transformer and attention explain the mechanics of a single forward pass. From there, the honest limits of a fluent predictor lead into hallucination and grounding, and the challenge of giving an agent lasting, current knowledge across many requests is the problem taken up by memory for agents.