AI Agents for Lawyers

Glossary

Fine-Tuning — Definition for Lawyers

Fine-tuning retrains an AI model's weights on your data. How it differs from RAG and prompting, why Latham and Harvey do it, why Spellbook refuses, and the client-data problem.

By Dr. Niklas SchmidtUpdated 13 November 20265 min read
On this page
  1. Prompting, RAG and fine-tuning compared
  2. Who fine-tunes, and who refuses to
  3. The data-protection problem fine-tuning creates
  4. A checklist before anyone proposes a training run

Fine-tuning is the further training of an existing AI model on a chosen dataset so that its internal weights change and it behaves differently from then on. It is the expensive option among three ways to shape a model: prompting tells it what to do at run-time, retrieval-augmented generation hands it documents at run-time, fine-tuning rewrites the model itself. As of September 2026, almost no law firm should do it.

Prompting, RAG and fine-tuning compared

A frontier model arrives pre-trained on a large slice of the internet and then post-trained by its vendor to follow instructions. Fine-tuning is a further round of that post-training, run by you or a vendor on your data: thousands of example inputs paired with the outputs you want, pushed through the model until its weights shift. The result is a new model that has absorbed a style, a vocabulary or a task format. It has not absorbed a live library, and it has not learned to stop inventing citations.

PromptingRetrieval-augmented generation (RAG)Fine-tuning
What changesNothing in the model; only the instructionsNothing in the model; documents are fetched into the context at run-timeThe model's weights
Best atFormat, tone, task framingFacts that change: precedents, statutes, matter filesConsistent style and structure; domain vocabulary; a fixed output format
Weak atLong instruction sets the model drifts fromAnything the index does not containNew facts; it cannot cite what it absorbed
EffortMinutesDays to build a knowledge baseWeeks, GPUs, machine-learning staff, evaluation
Where client data ends upThe context window, then the vendor's retention rulesA retrieval index you controlInside the weights, irreversibly
Who does it, September 2026Every lawyerFirms building a knowledge base; every legal vendorLatham & Watkins; Harvey

Who fine-tunes, and who refuses to

Latham & Watkins, with about $8.3bn in revenue and around 100 of its 900 technology staff working on AI, bought Nvidia H200 servers housed where only its own people can go and is fine-tuning Nvidia's open-weight Nemotron 3 models on them, according to reports of 10 and 11 September 2026; the firm still pays for Harvey, Legora and the frontier labs. Harvey shipped its own trained model, Harvey Tenet, a post-trained open-weight legal model, in research preview on 20 August 2026 inside Harvey II. Thomson Reuters' next-generation CoCounsel Legal, generally available in the US since 20 August 2026, pairs Anthropic's Claude Agent SDK with TR's own legal LLM.

Then there is Spellbook, with roughly 4,000 customers in 80-plus countries, whose chief executive Scott Stevenson has said fine-tuning "encourages hallucinations"; Spellbook grounds GPT-5-class frontier models on the user's documents rather than training its own. Litera makes a similar point from the other direction, keeping a deterministic redline engine under its generative layer. The build-versus-buy guide works through where a firm sits between those positions; for almost everyone below Latham's scale the answer is a knowledge base and retrieval-augmented generation, not a training run.

The benchmark data sides with the sceptics. In HAQQ's June 2026 test, 24% of 3,000 answers cited or misapplied law that did not support the claim, and Mistral Large, a strong open-weight base, produced citation errors on 64% of questions. Fine-tuning on a corpus of judgments makes a model sound like a judgment; it does not make the citations real.

The data-protection problem fine-tuning creates

Every bar rule on AI draws the same line, and fine-tuning sits on the wrong side of it unless you control what goes in. Germany's BRAK guidance of December 2024 says tools that store or forward client data for model training breach § 43a Abs. 2 BRAO. The Austrian bar's four practice rules from its March 2026 KI-Tag end with "no client data for training". ABA Formal Opinion 512 requires informed consent where a self-learning tool may expose client information, and California's proposed comment to Rule 1.6 would treat exposure to AI as "revealing" client data wherever there is "a material risk of unauthorized access, retention, or use". Weights are retention in its purest form: a model fine-tuned on a client's contracts cannot forget that client on request, and an erasure demand under the GDPR has no clean answer.

The EDPB's Opinion 28/2024 adds a second layer. A model is anonymous only if extraction of personal data from it is "insignificant", and a firm deploying a model must assess whether the upstream training was lawful. Fine-tune on matter files and your firm becomes the party whose training has to survive that test. The GDPR and AI tools page and the Austrian and German bar guidance page have the detail.

A checklist before anyone proposes a training run

  1. Name the behaviour you want. If it is "know our precedents", you want retrieval. If it is "draft in house style", a fine-tune might help, and a detailed system prompt usually gets most of the way there first.
  2. Confirm the base model may be fine-tuned: an open-weight model under MIT or Apache 2.0, or a vendor's fine-tuning service with a written no-retention term.
  3. Strip client-identifying data from the training set or obtain informed consent from each client whose material is included, and document the decision.
  4. Budget for evaluation. Without a held-out test set marked by lawyers, you will not know whether the tuned model is better or merely different.
  5. Write down who may query the model and where its outputs are logged.

Module 4 of the AI Lab for Lawyers course builds the retrieval alternative live, a firm knowledge base wired to a frontier model without code, which is where most "we should fine-tune" conversations end.

Frequently asked questions

What is the difference between fine-tuning and RAG?

Fine-tuning further trains a model on your examples so that its weights change and it permanently behaves differently; it is good at style, structure and vocabulary. Retrieval-augmented generation leaves the model alone and fetches relevant documents into the prompt at the moment of the question, so the model can quote and cite them. Facts that change, such as precedents, statutes and matter files, belong in retrieval. A fixed house style might belong in a fine-tune, but a detailed system prompt usually gets most of the way there.

Do law firms fine-tune their own AI models?

A handful do. Latham & Watkins, with about 900 technology staff and around 100 on AI, bought Nvidia H200 servers held where only its staff have access and is fine-tuning Nvidia's open-weight Nemotron 3 models, according to reports of 10 and 11 September 2026, while still paying for Harvey, Legora and the frontier labs. Harvey shipped its post-trained open-weight model Tenet in research preview on 20 August 2026. Almost every other firm builds a knowledge base on a frontier model instead.

Is fine-tuning an AI model on client data allowed?

Only with care that most firms cannot yet show. Germany's BRAK guidance says tools that store or forward client data for model training breach section 43a of the BRAO; the Austrian bar's practice rules say no client data for training; ABA Formal Opinion 512 requires informed consent where a self-learning tool may expose client information. Data absorbed into weights cannot be erased for one client later, which makes GDPR erasure and rectification requests hard to answer. Strip identifying data or obtain consent, and document both.

Put this into practice

AI Lab for Lawyers: a live, hands-on course for practising lawyers

4 live sessions (8 hours), recordings included, taught by Dr. Niklas Schmidt of Wolf Theiss. Rated 4.7/5 from 381 ratings; alumni from Norton Rose Fulbright, Paul Hastings, Pillsbury and Taylor Wessing.

Reserve a seat · €200 Maven money-back guarantee · certificate of completion