Guides
How Large Language Models Work, Explained for Lawyers
How LLMs work, explained for lawyers: tokens, prediction, training versus inference, context windows, knowledge cutoffs, reasoning modes and the September 2026 model line-up.
On this page
- Tokens: what the model actually reads
- Training and inference: two different events
- The knowledge cutoff: the date the trainee stopped reading
- The context window: the trainee's desk
- Why the model is confident and wrong
- Reasoning modes: when the trainee is allowed to think first
- Multimodal input: reading scans, tables and images
- What each mechanic means for your practice
- A setup that works
Think of a large language model as a brilliant trainee who has read most of the public internet up to a fixed date, has no access to your file, and answers every question by writing the most plausible next sentence. That is how LLMs work: they predict text, token by token, from patterns learned in training. Everything useful and everything dangerous about them for legal work follows from that mechanism.
Tokens: what the model actually reads
A model does not see words, pages or paragraphs. It sees tokens, fragments of text that are usually a short word, part of a longer word or a piece of punctuation. "Indemnification" is several tokens; "the" is one. Two consequences matter in practice.
First, everything is priced and limited in tokens. As of September 2026 Anthropic charges $10 per million input tokens and $50 per million output tokens for Claude Fable 5.1, $5/$25 for Claude Opus 5 and $2/$10 for Claude Sonnet 5; OpenAI charges $10/$50 for GPT-6 Astra; Google charges $2/$12 for Gemini 3.1 Pro (Anthropic model overview). Anthropic says one million tokens is roughly 555,000 words, so a 40-page share purchase agreement is around 20,000 tokens and a 2,000-document data room might be tens of millions.
Second, tokenisation is why models sometimes stumble on citations, section numbers and unusual names. A citation such as "[2025] EWHC 1383 (Admin)" is a string of odd tokens with weak statistical associations, which is one reason invented citations look so convincing: the model has learned the shape without the substance. The glossary entry on tokens has worked examples.
Training and inference: two different events
Training happens once, at enormous cost, before you ever see the model. The vendor feeds it trillions of tokens of text and adjusts billions of internal weights until it becomes good at predicting what comes next. Then the vendor runs further rounds of human feedback and reinforcement to make it helpful, polite and (mostly) safe. After that the weights are frozen.
Inference is what happens when you type. The frozen model reads your prompt plus everything already in the conversation, and generates a reply one token at a time, each token chosen from a probability distribution over the vocabulary. Nothing you say during inference changes the weights. When a vendor says a consumer chat "may be used to improve the model", it means your transcript may be collected for a future training run, not that the model is learning from you live; the confidentiality checklist sets out which plans do that.
The trainee analogy holds here too. Training is law school and everything the trainee read before joining you. Inference is the answer they give in the meeting. If the file is not on the table, the answer is a guess.
The knowledge cutoff: the date the trainee stopped reading
Because training is fixed, every model has a knowledge cutoff. As of September 2026:
| Model | Released | Knowledge cutoff | Context window | Max output |
|---|---|---|---|---|
| Claude Fable 5.1 (Anthropic) | 1 September 2026 | June 2026 | 1M tokens | 128K |
| Claude Opus 5 (Anthropic) | 24 July 2026 | May 2026 | 1M tokens | 128K |
| Claude Sonnet 5 (Anthropic) | 30 June 2026 | not stated | 1M tokens | 128K |
| GPT-6 Astra (OpenAI) | 4 September 2026 (public to paid users) | 30 April 2026 | ~1.05M tokens | 128K |
| GPT-5.6 Luna/Terra/Sol (OpenAI) | 9 July 2026 | not stated | not stated | not stated |
| Gemini 3.1 Pro (Google) | 19 February 2026 | not stated | 1M tokens | 64K |
| Grok 4.6 (xAI) | 12 August 2026 | not stated | 500K tokens | not stated |
| Muse Spark 1.3 (Meta) | 2 September 2026 | not stated | 1M tokens | not stated |
Sources: Anthropic model overview; GPT-6 Astra; vendor announcements as collected in September 2026.
A model with a June 2026 cutoff does not know that the Digital Omnibus on AI entered into force on 27 July 2026, that GPT-6 Astra exists, or what a court decided last week. It will still answer questions about those things, fluently, because answering fluently is what it does. Two fixes exist: give it a search tool (Perplexity, ChatGPT search, Westlaw or Lexis connectors) or paste the source in. Both put the fact into the context window, which is the only place the model can "know" anything new.
The context window: the trainee's desk
The context window is everything the model can see in one conversation: your instructions, the documents you upload, the conversation so far and its own replies. Claude Fable 5.1, Opus 5 and Sonnet 5, GPT-6 Astra and Gemini 3.1 Pro all offer around one million tokens, enough for a full lease portfolio or a deposition bundle in one go. Claude Haiku 4.5 offers 200K; Grok 4.6 offers 500K.
Three practical points. A long context is not a database: models are measurably worse at finding a detail on page 380 of a 500-page upload than on page 3, so ask specific questions and require page references. Cost scales with context, and Gemini 3.1 Pro doubles its price above 200K tokens. And the context is where confidentiality risk lives: the model only sees what you put there, so the discipline of redaction and matter-scoped uploads is the discipline that matters. Claude Projects and Gemini Notebook are, at bottom, ways to manage what sits in the window; setting up Claude Projects for a matter shows the pattern. See also context window.
Why the model is confident and wrong
Fluency is not evidence. A model produces the most probable continuation, and a probable continuation of "The leading authority on this point is" is a real-sounding case name, a real-sounding citation and a holding that matches your question. Nothing in the mechanism checks the case exists. OpenAI's own researchers published a paper on 5 September 2025 arguing that training and evaluation reward guessing over saying "I don't know", so models learn to bluff like a student who never leaves an exam question blank (OpenAI, Why Language Models Hallucinate).
The legal consequences are documented at length: 2,041 court decisions worldwide involving hallucinated material by 14 September 2026, 811 of them involving lawyers. The mechanism, the numbers and a verification routine are in AI hallucinations in legal practice. Two habits follow directly from the mechanism: never ask a general model for citations you have not supplied, and treat any answer that is more confident than you would be as a flag rather than a comfort. Stanford's 2026 AI Index adds a subtler failure: when a false premise is framed as the user's own belief, accuracy collapses, in one test from 98% to 64% (Stanford HAI AI Index 2026). Models agree with you. Draft your prompts so that agreement is not the easy answer, a point developed in prompt engineering for lawyers.
Reasoning modes: when the trainee is allowed to think first
Since 2025 the frontier models can "think" before answering: they generate an internal chain of working, sometimes thousands of tokens long, and only then write the reply. On Claude this is "adaptive" thinking with an effort setting from low to max; on Fable 5.1 thinking is always on and the raw chain is never shown. GPT-6 Astra uses a "recurrent depth" technique that OpenAI acknowledges "obscures some or all of the AI's reasoning" (GPT-6 Astra).
For lawyers the trade is simple: reasoning modes are slower and cost more, and they are markedly better at multi-step tasks (tracing a defined term through a facility agreement, reconciling a chronology, checking a redline against a playbook). They are not markedly better at recalling facts they never had. One behavioural change is worth knowing: Anthropic's release notes for Opus 5 (24 July 2026) say it "verifies its own work without being told to", writes longer deliverables and delegates to sub-agents more readily (Anthropic, what's new in Opus 5). Prompts written in 2025 that end "now add a final verification step" can now produce over-verification and padding. The reasoning model entry covers how to choose an effort level.
Multimodal input: reading scans, tables and images
Current models accept images and PDFs as well as text. In practice this means a scanned lease, a photographed whiteboard or a chart in a board pack can be read directly. Quality varies: for poor scans a dedicated OCR pass before upload is still the safer route, and reports from 2026 suggest Claude is more likely to reply "illegible" where ChatGPT will guess a number, a difference that matters when the number is a rent figure. Extracting data from PDFs and scans sets out a workflow.
What each mechanic means for your practice
| Mechanic | What it means at your desk | Habit to adopt |
|---|---|---|
| Next-token prediction | Fluent output is not checked output | Verify every fact and citation you did not supply |
| Frozen weights | The model cannot learn your matter between chats | Use Projects, Notebooks or a firm knowledge base to re-supply context |
| Knowledge cutoff | Anything after the cutoff is invisible | Provide recent authority; use search tools for currency |
| 1M-token context | Whole bundles fit, but recall degrades with depth | Ask targeted questions; demand page references |
| Reasoning effort | Better on multi-step logic, dearer and slower | Use high effort for analysis, low for formatting |
| Multimodal input | Scans and images can be read, imperfectly | OCR first for anything with numbers |
The same mechanics explain why the vendor market has split the way it has: Harvey, Legora and CoCounsel are largely systems for managing what goes into the context window and what checks run on the way out. That is also why a firm can get a long way with a general model plus discipline, a comparison taken up in legal-specific AI vs general-purpose AI and in the reviews of Claude and ChatGPT for legal work.
A setup that works
Use a reasoning model at high effort for analysis and a fast model for formatting. Keep a matter-scoped project with the key documents uploaded once. Write every research prompt so the model must work only from the authorities you supply. Ask for page references and for the strongest counter-argument. Then check the two or three things you would check in a trainee's memo: does the authority exist, does it say that, and is it still good law. Module 1 of the AI Lab for Lawyers course spends its first session on exactly these mechanics, because every later technique, from prompting to agents, is an application of them.
Frequently asked questions
What is a token in an AI model?
A token is the unit of text a model reads and writes, roughly three-quarters of an English word or a few characters of a citation. Models are priced per million tokens in and out, and context windows are measured in tokens. Anthropic states that one million tokens is about 555,000 words on its current tokenizer, so a 40-page contract is roughly 20,000 tokens.
Why do large language models make things up?
Because they generate the most probable continuation of the text, not a retrieved fact. A plausible case name, citation and holding are all high-probability sequences, so the model produces them fluently whether or not the case exists. OpenAI's own September 2025 paper adds that training and benchmarks reward guessing over admitting uncertainty, which makes confident errors more likely.
What does the knowledge cutoff mean for legal research?
The cutoff is the last date covered by the model's training data. Claude Opus 5 is May 2026, Claude Fable 5.1 June 2026 and GPT-6 Astra 30 April 2026. The model knows nothing after that unless a search tool or document you provide puts it in the context window, so recent judgments, amendments and rate changes must be supplied, not assumed.
What is a context window and why does it matter for lawyers?
The context window is the working memory of a single conversation: everything you paste, upload or the model writes back. Claude Fable 5.1, Opus 5, Sonnet 5, GPT-6 Astra and Gemini 3.1 Pro all offer about one million tokens. That fits a large document set, but accuracy on details buried deep in a long context still degrades, so cite page numbers and ask targeted questions.
Put this into practice
AI Lab for Lawyers: a live, hands-on course for practising lawyers
4 live sessions (8 hours), recordings included, taught by Dr. Niklas Schmidt of Wolf Theiss. Rated 4.7/5 from 381 ratings; alumni from Norton Rose Fulbright, Paul Hastings, Pillsbury and Taylor Wessing.
Related reading
Glossary
Large Language Model (LLM) — Definition for Lawyers
A large language model is a neural network trained to predict the next token, and it powers every legal AI tool. How LLMs are built, the models of September 2026 and what follows for reliability and confidentiality.
4 min read
Glossary
Token (AI) — Definition for Lawyers
A token is the unit of text a language model reads and writes, usually part of a word. Why limits, prices and speed are counted in tokens, not pages or words.
3 min read
Glossary
Context Window — Definition for Lawyers
A context window is the text an AI model can hold at once. As of September 2026 it is 1 million tokens on Claude, GPT-6 and Gemini, about 555,000 words.
4 min read
Glossary
Reasoning Model — Definition for Lawyers
A reasoning model works through intermediate steps before it answers. What adaptive thinking and effort settings mean for legal analysis, cost, speed and hidden reasoning.
4 min read
Guides
AI Hallucinations in Legal Practice: Why They Happen and How to Catch Them
AI hallucinations in legal practice: why models invent cases, what 2,041 court decisions and the HAQQ benchmark show, and a verification routine that holds.
9 min read
Tools
Claude for Lawyers: An Honest Review of Anthropic's Models for Legal Work
Claude for lawyers, reviewed as of September 2026: Fable 5.1, Opus 5 and Sonnet 5, plan prices, data retention by plan, legal plugins, benchmarks, weaknesses.
9 min read