There are two common ways to explain a language model badly. The first is to say it is just autocomplete, which is technically close but makes it sound trivial. The second is to say it understands language, which sounds explanatory but quietly smuggles in a claim nobody can defend.
This piece takes the middle route: describe the mechanism accurately, then use it to predict where the thing will be brilliant and where it will fall over.
The one-sentence version
A large language model is a mathematical function with billions of adjustable numbers, tuned so that when you give it a stretch of text, it produces a probability distribution over what comes next.
That is the whole job. Everything else — answering questions, writing code, translating, summarising — is that same operation applied repeatedly.
Step one: text becomes numbers
Models do not see letters. They see tokens, which are chunks of text somewhere between a character and a word. Common words are usually one token. Unusual words break into pieces.
The sentence "unblocking the server" might become five tokens: "un", "block", "ing", " the", " server". This matters more than it sounds:
- It explains why models are unreliable at counting letters or reversing words. They cannot see inside a token easily.
- It explains why some languages cost more to process. If a language tokenises less efficiently, the same meaning takes more tokens.
- It explains pricing. You are billed per token, not per word.
Each token becomes a long list of numbers — a vector — and those vectors are what the model actually operates on.
Step two: attention
The architecture underneath almost every modern model is called a transformer, and its central idea is attention.
At each layer, every token gets to look at every other token in the input and decide how much each one matters for interpreting it. In "the console overheated because its fan had failed", the token for "its" needs to resolve to "console". Attention is the mechanism that lets it, and it does so by learned relevance rather than fixed grammatical rules.
Stack a hundred of those layers and you get something that captures a startling amount of structure: syntax, factual association, tone, argument shape, code semantics. Nobody programmed any of it in. It is all a side effect of getting very good at the next-token task across a huge amount of text.
Predicting the next word well, across essentially everything humans have written, turns out to require internal machinery that looks a lot like knowledge. Not because knowledge was the goal, but because it is the cheapest way to lower the prediction error.
Step three: three stages of training
Almost every useful model goes through the same pipeline.
Pre-training. Show it an enormous corpus and have it predict the next token, over and over. This is where the capability comes from, and where nearly all the compute is spent. The result is a model that can continue text but has no particular inclination to be helpful.
Instruction tuning. Fine-tune on examples of instructions paired with good responses. This is what turns a text continuer into something that answers a question rather than inventing five more questions.
Preference tuning. Show the model pairs of responses ranked by human judgement and train it toward the preferred one. This shapes tone, refusal behaviour, formatting and general helpfulness. It also introduces the characteristic failure modes: sycophancy, over-hedging, and the tendency to produce confident-sounding answers because confident answers were rated well.
Why they make things up
Hallucination is not a malfunction. It is the mechanism operating exactly as designed on a question it lacks the information to answer.
The model produces a plausible continuation. When the true answer is well represented in what it learned, the plausible continuation is the correct one. When it is not, the model does not have a mechanism that says "no data here". It generates the most statistically reasonable-looking text, which for a factual question means a fluent, well-formatted, wrong answer.
This is why the fixes are external rather than internal:
- Retrieval. Fetch the real documents and put them in the prompt, so the answer is a summarisation task rather than a recall task.
- Tools. Let it call a calculator, a database, a search engine. Do not ask it to be a database.
- Citations. Require sources so a human can check.
- Verification. For anything consequential, have a second system or a person confirm.
The context window, and what "memory" really means
The context window is how much text the model can look at in one go. Everything outside it does not exist as far as the model is concerned.
A base model has no memory between conversations. When a product appears to remember you, one of two things is happening: previous messages are being pasted back in as part of the prompt, or facts about you were saved to a store and retrieved when relevant. Both are engineering around the model, not properties of it.
Very long context windows help, but they are not free. Attention cost grows quickly with length, and models reliably attend better to the beginning and end of a long input than to the middle. Putting the important instruction at the top of a 200-page document is not a good plan.
What they are reliably good and bad at
| Task | Reliability | Why |
|---|---|---|
| Rewriting, summarising, translating | High | The information is in the prompt |
| Drafting and brainstorming | High | Plausibility is the goal, not accuracy |
| Code in common languages | Good with tests | Testable, and heavily represented in training |
| Structured extraction | Good | Pattern matching over supplied text |
| Precise arithmetic | Poor alone | Use a tool |
| Obscure facts, exact citations | Poor | Recall without a source is guessing |
| Anything after the training cutoff | Poor without search | It simply was not there |
The pattern is consistent: strong when the information is in front of it, weak when it has to remember.
Reasoning models, briefly
Newer systems are trained to produce a long chain of intermediate steps before answering, and to be rewarded on whether the final answer was correct. This helps substantially on maths, logic and multi-step planning, because the model gets to use generated text as working memory.
Two things worth knowing. First, it costs more, because those intermediate tokens are real tokens. Second, the visible reasoning is not a reliable audit trail — a model can reach the right answer with flawed stated steps, or the reverse. Treat it as scratch paper, not as an explanation.
A practical way to use them well
- Supply the source material. Nearly every quality problem is a context problem.
- Ask for structure. A specified format is easier to get right and far easier to check.
- Separate generation from verification. Never let the same pass do both.
- Give one clear task per request. Compound instructions produce compound failures.
- Assume any specific number, name, date or citation is wrong until checked. This single habit prevents most real-world embarrassment.
The honest summary
A language model is a compression of an enormous amount of human text into a function that continues patterns. That is a genuinely remarkable thing to have built, and it is also a bounded thing. It does not know what it does not know, it has no persistent memory of you, and it cannot check its own work.
Used as a fast, tireless, occasionally wrong assistant with a good memory for form and a poor one for fact, it is enormously useful. Used as an oracle, it will eventually embarrass you. The difference is entirely in how you wire it up.