Glossary
Fine-Tuning — Definition for Lawyers
Fine-tuning retrains an AI model's weights on your data. How it differs from RAG and prompting, why Latham and Harvey do it, why Spellbook refuses, and the client-data problem.
On this page
Fine-tuning is the further training of an existing AI model on a chosen dataset so that its internal weights change and it behaves differently from then on. It is the expensive option among three ways to shape a model: prompting tells it what to do at run-time, retrieval-augmented generation hands it documents at run-time, fine-tuning rewrites the model itself. As of September 2026, almost no law firm should do it.
Prompting, RAG and fine-tuning compared
A frontier model arrives pre-trained on a large slice of the internet and then post-trained by its vendor to follow instructions. Fine-tuning is a further round of that post-training, run by you or a vendor on your data: thousands of example inputs paired with the outputs you want, pushed through the model until its weights shift. The result is a new model that has absorbed a style, a vocabulary or a task format. It has not absorbed a live library, and it has not learned to stop inventing citations.
| Prompting | Retrieval-augmented generation (RAG) | Fine-tuning | |
|---|---|---|---|
| What changes | Nothing in the model; only the instructions | Nothing in the model; documents are fetched into the context at run-time | The model's weights |
| Best at | Format, tone, task framing | Facts that change: precedents, statutes, matter files | Consistent style and structure; domain vocabulary; a fixed output format |
| Weak at | Long instruction sets the model drifts from | Anything the index does not contain | New facts; it cannot cite what it absorbed |
| Effort | Minutes | Days to build a knowledge base | Weeks, GPUs, machine-learning staff, evaluation |
| Where client data ends up | The context window, then the vendor's retention rules | A retrieval index you control | Inside the weights, irreversibly |
| Who does it, September 2026 | Every lawyer | Firms building a knowledge base; every legal vendor | Latham & Watkins; Harvey |
Who fine-tunes, and who refuses to
Latham & Watkins, with about $8.3bn in revenue and around 100 of its 900 technology staff working on AI, bought Nvidia H200 servers housed where only its own people can go and is fine-tuning Nvidia's open-weight Nemotron 3 models on them, according to reports of 10 and 11 September 2026; the firm still pays for Harvey, Legora and the frontier labs. Harvey shipped its own trained model, Harvey Tenet, a post-trained open-weight legal model, in research preview on 20 August 2026 inside Harvey II. Thomson Reuters' next-generation CoCounsel Legal, generally available in the US since 20 August 2026, pairs Anthropic's Claude Agent SDK with TR's own legal LLM.
Then there is Spellbook, with roughly 4,000 customers in 80-plus countries, whose chief executive Scott Stevenson has said fine-tuning "encourages hallucinations"; Spellbook grounds GPT-5-class frontier models on the user's documents rather than training its own. Litera makes a similar point from the other direction, keeping a deterministic redline engine under its generative layer. The build-versus-buy guide works through where a firm sits between those positions; for almost everyone below Latham's scale the answer is a knowledge base and retrieval-augmented generation, not a training run.
The benchmark data sides with the sceptics. In HAQQ's June 2026 test, 24% of 3,000 answers cited or misapplied law that did not support the claim, and Mistral Large, a strong open-weight base, produced citation errors on 64% of questions. Fine-tuning on a corpus of judgments makes a model sound like a judgment; it does not make the citations real.
The data-protection problem fine-tuning creates
Every bar rule on AI draws the same line, and fine-tuning sits on the wrong side of it unless you control what goes in. Germany's BRAK guidance of December 2024 says tools that store or forward client data for model training breach § 43a Abs. 2 BRAO. The Austrian bar's four practice rules from its March 2026 KI-Tag end with "no client data for training". ABA Formal Opinion 512 requires informed consent where a self-learning tool may expose client information, and California's proposed comment to Rule 1.6 would treat exposure to AI as "revealing" client data wherever there is "a material risk of unauthorized access, retention, or use". Weights are retention in its purest form: a model fine-tuned on a client's contracts cannot forget that client on request, and an erasure demand under the GDPR has no clean answer.
The EDPB's Opinion 28/2024 adds a second layer. A model is anonymous only if extraction of personal data from it is "insignificant", and a firm deploying a model must assess whether the upstream training was lawful. Fine-tune on matter files and your firm becomes the party whose training has to survive that test. The GDPR and AI tools page and the Austrian and German bar guidance page have the detail.
A checklist before anyone proposes a training run
- Name the behaviour you want. If it is "know our precedents", you want retrieval. If it is "draft in house style", a fine-tune might help, and a detailed system prompt usually gets most of the way there first.
- Confirm the base model may be fine-tuned: an open-weight model under MIT or Apache 2.0, or a vendor's fine-tuning service with a written no-retention term.
- Strip client-identifying data from the training set or obtain informed consent from each client whose material is included, and document the decision.
- Budget for evaluation. Without a held-out test set marked by lawyers, you will not know whether the tuned model is better or merely different.
- Write down who may query the model and where its outputs are logged.
Module 4 of the AI Lab for Lawyers course builds the retrieval alternative live, a firm knowledge base wired to a frontier model without code, which is where most "we should fine-tune" conversations end.
Frequently asked questions
What is the difference between fine-tuning and RAG?
Fine-tuning further trains a model on your examples so that its weights change and it permanently behaves differently; it is good at style, structure and vocabulary. Retrieval-augmented generation leaves the model alone and fetches relevant documents into the prompt at the moment of the question, so the model can quote and cite them. Facts that change, such as precedents, statutes and matter files, belong in retrieval. A fixed house style might belong in a fine-tune, but a detailed system prompt usually gets most of the way there.
Do law firms fine-tune their own AI models?
A handful do. Latham & Watkins, with about 900 technology staff and around 100 on AI, bought Nvidia H200 servers held where only its staff have access and is fine-tuning Nvidia's open-weight Nemotron 3 models, according to reports of 10 and 11 September 2026, while still paying for Harvey, Legora and the frontier labs. Harvey shipped its post-trained open-weight model Tenet in research preview on 20 August 2026. Almost every other firm builds a knowledge base on a frontier model instead.
Is fine-tuning an AI model on client data allowed?
Only with care that most firms cannot yet show. Germany's BRAK guidance says tools that store or forward client data for model training breach section 43a of the BRAO; the Austrian bar's practice rules say no client data for training; ABA Formal Opinion 512 requires informed consent where a self-learning tool may expose client information. Data absorbed into weights cannot be erased for one client later, which makes GDPR erasure and rectification requests hard to answer. Strip identifying data or obtain consent, and document both.
Put this into practice
AI Lab for Lawyers: a live, hands-on course for practising lawyers
4 live sessions (8 hours), recordings included, taught by Dr. Niklas Schmidt of Wolf Theiss. Rated 4.7/5 from 381 ratings; alumni from Norton Rose Fulbright, Paul Hastings, Pillsbury and Taylor Wessing.
Related reading
Comparisons
Open-Weight vs Closed Models for Law Firms: Control, Cost and Confidentiality
Open-weight vs closed AI models for law firms: what Latham's own GPUs buy, Harvey Tenet, DeepSeek V4, Mistral, the legal benchmark gap, when self-hosting pays.
10 min read
Guides
Build or Buy Legal AI? Latham's GPUs, Harvey's Seats and the Firms in Between
Build vs buy legal AI in 2026: rent Harvey or Legora, assemble on Claude or Astra with MCP, or build like Latham on own GPUs. Costs, control, decision tree.
11 min read
Glossary
Retrieval-Augmented Generation (RAG) — Definition for Lawyers
Retrieval-augmented generation (RAG) searches a document set first and drafts from what it finds. Every grounded legal tool uses it; here is why errors still get through.
4 min read
Regulation & Ethics
GDPR and AI Tools for Lawyers: Lawful Basis, DPIAs and Vendor Terms
How GDPR applies when law firms use Claude, ChatGPT or Copilot: controller and processor roles, lawful basis, DPIA triggers, transfers and vendor terms.
11 min read
Playbooks
Build a Firm Knowledge Base for AI: Curation, Retrieval and Permissions
A playbook for a law firm knowledge base AI can retrieve from: curating precedents, metadata, permission mapping, platform choice and a 30-question test.
11 min read
Glossary
Open-Weight Model — Definition for Lawyers
An open-weight model publishes its trained weights for anyone to run. Open weights vs open source, DeepSeek, Mistral and Muse Glimmer licences, self-hosting and government bans.
5 min read