AI Agents for Lawyers

Glossary

Retrieval-Augmented Generation (RAG) — Definition for Lawyers

Retrieval-augmented generation (RAG) searches a document set first and drafts from what it finds. Every grounded legal tool uses it; here is why errors still get through.

By Dr. Niklas SchmidtUpdated 13 November 20264 min read
On this page
  1. How it works, on one research question
  2. Where lawyers meet it
  3. Why grounding cuts errors but does not end them

Retrieval-augmented generation (RAG) is a technique in which an AI system first searches a defined body of documents for passages relevant to your question, then hands those passages to the language model together with the question, so that the answer is drafted from retrieved text rather than from the model's memory. Every "grounded" legal research tool, from CoCounsel to Lexis+ Protégé, relies on it.

How it works, on one research question

Take a question a litigator might ask a research tool: whether an English landlord must serve a section 146 notice before forfeiting a commercial lease for non-payment of rent. Without retrieval, the model answers from whatever it absorbed in training, and if it never absorbed the right case it produces the case most likely to exist. With retrieval, five things happen in sequence:

  1. Search. The retriever turns the question into a query. That may be keyword search, or an embedding compared against a vector database of the corpus, or both. It returns the top matches: perhaps twenty paragraphs from judgments and a practice note.
  2. Assembly. Those passages are placed in the model's context window with an instruction along the lines of "answer only from these sources and cite them by paragraph".
  3. Drafting. The model writes the answer with citations to the passages it was given.
  4. Verification layer. The better products then check the cited authorities against a citator: Lexis+ with Protégé describes Shepard's-grounded verification, Thomson Reuters ships Deep Research Verify in CoCounsel Legal.
  5. Presentation. You see an answer with links to the sources. The links are real because they came from the search index, not from the model.

Two consequences follow. The model never had to remember a case, so inventing one from nothing becomes rarer. And every one of the five steps can fail silently.

Where lawyers meet it

Product (as of September 2026)What it retrieves fromWhat the vendor says about grounding
CoCounsel Legal (Thomson Reuters)WestlawDeep Research Verify; next-generation version GA in the US on 20 August 2026
Lexis+ with ProtégéLexis contentShepard's-grounded verification; US GA 24 February 2026
Astra for Law (OpenAI)Legal Search Index of 230 million-plus URLs of US law, built with CourtListener and the Free Law Project54% on the Vals Legal Research Bench against 38.7% for GPT-6 with plain web search (OpenAI-reported, September 2026; treat as days old)
Harvey Vault and Knowledge; Legora Tabular ReviewYour uploaded matter documents, up to 100,000 in Legora's caseTabular output with per-cell citations
Gemini Notebook (formerly NotebookLM)Only the sources you upload, 50 free or 300 paidClosed universe; cannot browse the open web
Claude Projects, a firm knowledge baseFiles you or the firm curateNo citator; you supply the corpus and the verification

The last row is the one most firms end up building themselves, and the firm knowledge base playbook covers the curation and permission questions that decide whether it works.

Why grounding cuts errors but does not end them

Stanford's RegLab put the vendors' "hallucination-free" claims to the test and measured 17% hallucination for Lexis+ AI and 33% for Westlaw AI-Assisted Research on May 2024 versions, against 43% for ungrounded GPT-4. Better, not fixed. The sanctions record says the same: the 16 fabricated quotations in Fletcher v. Experian (5th Cir. 2026) were traced to vLex and CoCounsel, and the $31,100 in Lacey v. State Farm (C.D. Cal. 2025) followed a brief built with CoCounsel, Westlaw Precision and Gemini. A 2026 academic benchmark of legal RAG still finds state-of-the-art systems hallucinating at the level of individual claims.

Failure modeWhat happensHow you catch it
Retrieval missThe leading authority is not among the top passages, so the model answers from what it was given: a plausible but partial answerRun your own search for the case you expected to see; ask the tool what it excluded
MisreadingThe retrieved case is real but the model states a holding it does not contain; on the HAQQ benchmark (June 2026), 24% of answers misapplied the law they citedOpen the source and read the paragraph; nothing else works
Stale or walled corpusFirm RAG fails less because of the retriever than because legal knowledge is fragmented, walled by ethical screens and unevenly updated (Inside Practice, January 2026)Ask when the corpus was last updated and what your permissions exclude
Fabricated reference despite searchA University of Pennsylvania study (April 2026) found search-augmented deep-research agents fabricating 3% to 13% of cited URLsClick every link before it goes in a filing
Over-refusalIn Vals' February 2025 study, Vincent AI "refused to answer, rather than hallucinate"; the safer failure, and still a failureRephrase, narrow the jurisdiction, or search manually

The practical shift is this. An ungrounded chatbot makes you ask whether the case exists. A grounded tool makes you ask whether the case says that, and the hallucination statistics show the second question is where lawyers now get caught. The workflow for asking it properly, tool by tool, is in the AI legal research page; Module 2 of the AI Lab for Lawyers course covers legal research and document analysis, where grounded tools do most of the work.

Frequently asked questions

Does RAG stop AI hallucinations?

No. It reduces them, mainly the wholesale invention of cases, because the model is drafting from passages in front of it. Stanford's RegLab study still measured 17% hallucination for Lexis+ AI and 33% for Westlaw AI-Assisted Research on 2024 versions, and a 2026 academic benchmark of legal RAG systems finds claim-level errors in state-of-the-art systems. The residual errors are real cases misread, which are harder to spot than fakes.

What is the difference between RAG and fine-tuning?

Fine-tuning changes the model itself by training it further on examples, which is expensive and freezes knowledge at training time. RAG leaves the model unchanged and supplies documents at the moment of the question, so the corpus can be updated daily and every answer can point to its source. Spellbook grounds frontier models rather than fine-tuning them; Harvey's Tenet is a post-trained legal model. Most legal products use both.

Is Gemini Notebook (NotebookLM) a RAG tool?

Yes, in its purest form. It answers only from the sources you upload, 50 per notebook on the free tier and 300 on paid plans, cites the passage it relied on, and cannot search the open web. That makes it a closed-universe tool: reliable for questions about the file in front of it, useless for finding authority that is not in the notebook. Use it in a Workspace or enterprise account, not a personal one.

Put this into practice

AI Lab for Lawyers: a live, hands-on course for practising lawyers

4 live sessions (8 hours), recordings included, taught by Dr. Niklas Schmidt of Wolf Theiss. Rated 4.7/5 from 381 ratings; alumni from Norton Rose Fulbright, Paul Hastings, Pillsbury and Taylor Wessing.

Reserve a seat · €200 Maven money-back guarantee · certificate of completion