Guides
Legal AI Benchmarks Explained: Vals, BigLaw Bench, LAB, HAQQ and LegalBench
Legal AI benchmarks explained: what Vals, BigLaw Bench, Harvey LAB, HAQQ and LegalBench measure, the September 2026 leaders, and how to run your own evaluation.
On this page
Legal AI benchmarks measure different things with different scoring rules, which is why GPT-5.6 Sol scored 92.7% on Harvey's BigLaw Bench and 2.5% on Harvey's own LAB in the same month. As of September 2026 the five worth reading are the Vals suite, BigLaw Bench, LAB, HAQQ and LegalBench. None is independent enough to buy on; all are useful for knowing what to test yourself.
Five benchmarks in one table
| Benchmark | Run by | Independent? | What it measures | Scoring | Leaders, September 2026 |
|---|---|---|---|---|---|
| Vals Index; Vals Legal Research Bench; Vals Harvey Legal Agent Benchmark | Vals.ai | Third-party evaluator; studies often commissioned by participants | Broad model ranking; US legal research questions; LAB tasks re-run across 62 models | Weighted accuracy; strict and criteria-level rates | Index (15 Sep, 58 models): Claude Fable 5.1, Opus 5, GPT-6 Astra. Legal Research Bench: Muse Spark 1.3 Max, Opus 5, Fable 5.1 |
| BigLaw Bench | Harvey | No | Practical legal tasks against expert rubrics | Partial credit | GPT-5.6 Sol 92.7% (Jul 2026); GPT-5.5 91.7%; GPT-5.4 91.0%; Claude Opus 4.7 90.9% (May 2026) |
| LAB (Legal Agent Benchmark) | Harvey, open source, launched 6 May 2026 | No, but the tasks and rubrics are public | 1,200+ long-horizon agent tasks, 24 practice areas, 75,000+ rubric criteria | Strict: every criterion must pass | Aggregator leaderboard (18 Sep): Fable 5 13.3%, Opus 5 11.7%, Opus 4.7 7.1%, GPT-5.5 2.1%, Gemini 3.1 Pro 0.0% |
| HAQQ | HAQQ.ai, 5 Jun 2026 | No | 3,000 graded answers, 300 tasks, 51 practice areas, 20+ jurisdictions; accuracy, hallucination, cost, speed | Points out of 35; separate hallucination rate | Claude Opus 4.8 30.02; Grok 4.3 second; GPT-5.5 lowest hallucination at 3% |
| LegalBench | Academic consortium; run on current models by Vals (16 Sep 2026) | Yes | Legal reasoning tasks, largely classification | Accuracy | Fable 5 88.56%; Fable 5.1 88.51%; Gemini 3.1 Pro 87.40% |
Two vendor benchmarks deserve a footnote. GC AI's In-House Legal Bench (May 2026) scores GC AI 86.8%, GPT-5.5 79.8%, Claude Opus 4.7 68.4% and Gemini 3.1 Pro 57.5%, and its RedlineBench rose from 47.9 to 57.0 with Fable 5.1 (GC AI); a vendor topping its own bench tells you the harness works, not that the model is weak. And the only independent test of paid legal research products remains Stanford RegLab's 2024 study, in which Lexis+ AI hallucinated 17% of the time and Westlaw AI-Assisted Research 33%, on products that have since been replaced (Stanford RegLab). No vendor has submitted to a comparable public test since.
Vals: the closest thing to a referee
Vals.ai is the only recurring evaluator that is not selling a legal product. Its February 2025 Legal AI Report compared Harvey, CoCounsel, Vincent AI and Oliver against a lawyer baseline on seven tasks: Harvey scored 94.8% on document question-answering against 70.1% for lawyers, CoCounsel 89.6%, and lawyers won redlining (79.7% against Harvey's 65.0%) and EDGAR research; Lexis+ AI withdrew before publication; Vincent was noted for refusing to answer rather than hallucinating (Vals, 27 Feb 2025).
The October 2025 legal research study (14 Oct 2025; 199 US questions) is the one most quoted, and it needs its metric labelled. On weighted accuracy, ChatGPT Team scored 80%, Counsel Stack 78%, Alexi and Midpage 74–77%, and the lawyer control group 69%; LawSites' summary quotes slightly different per-tool figures on another metric, with lawyers at 71% (LawSites, Oct 2025). Two findings matter more than the ranking: every system lost roughly 11–14 points on multi-jurisdiction questions, and lawyers took about 23 minutes per question against seconds for the tools. Thomson Reuters, LexisNexis and vLex declined to participate.
In September 2026 OpenAI reported that Astra for Law scored 54% on the Vals Legal Research Bench against 38.7% for GPT-6 with plain web search (Artificial Lawyer, 18 Sep 2026). A Hacker News commenter's reaction, "How is this a product that you are selling?", is a fair question about the absolute number; the fair answer is that it is a hard, adversarially written test and the comparison is with the same model without the legal index. Both readings are correct, which is the point of this page. The product itself is reviewed at Astra for Law.
BigLaw Bench and LAB: one vendor, two answers
Harvey publishes a BigLaw Bench score for each new frontier model within days of release (Harvey on GPT-5.6 Sol). Scores are rubric-based with partial credit, so a draft that gets most of the way earns most of the marks, and the leaders sit within two points of each other. One critic's objection is structural: BigLaw Bench "does not actually test case law research" (Sabaio), which is the task lawyers worry about most.
LAB, released as open source on 6 May 2026 without a leaderboard, applies the opposite rule: a task passes only if every one of its rubric criteria passes. Harvey's own initial run (26 May 2026) put Claude Opus 4.7 at 7.1%, Sonnet 4.6 at 5.4%, GPT-5.5 at 2.1% and Gemini 3.5 Flash at 0.8%, at a cost of about $50.90 and 22 minutes per task for Opus 4.7. Behaviour mattered: verify-and-revise added 1.5 points, drafting without review cost 1.2, and firing five or more parallel tool calls per turn cost 0.5 (Harvey, 26 May 2026). By 18 September 2026 an aggregator leaderboard showed Claude Fable 5 at 13.3% and Opus 5 at 11.7% (LLM Stats); criteria-level pass rates for the same runs sit around 83–88%.
Bob Ambrogi's description of LAB as "a benchmark built by a market participant" is right, and Harvey's own numbers are still the most useful single fact on this page for a supervising partner: the best agent on the market completes about one long legal task in eight without a single rubric miss (LawSites, May 2026). A Reddit analysis argued the all-pass metric "should probably be considered marginal at best" as a predictor of real performance, because a human associate would also fail a 60-criterion rubric somewhere. That is true, and it is also why associates have supervisors. What the numbers imply for delegation is worked through in what AI agents are.
HAQQ and the hallucination question
HAQQ's June 2026 benchmark is vendor-run but unusually transparent about cost and time. Across 3,000 graded answers, Claude Opus 4.8 won 130 of 300 tasks and finished first at 30.02 out of 35; Grok 4.3 was second at $0.003 per task; GPT-5.5 had the lowest citation-hallucination rate at 3% but was the slowest (134 seconds) and the most expensive ($0.082 per task); o3 hallucinated 32% of the time; Mistral Large had a 64% citation error rate; the cost spread across models was 90-fold. The line to remember: 24% of all answers "cited or misapplied law that didn't support claims", and every model, including the leaders, did it somewhere (HAQQ, 5 Jun 2026).
Independent work points the same way. A University of Pennsylvania study (3 April 2026) found search-augmented models and deep-research agents fabricated 3–13% of the URLs they cited, with agents hallucinating more because they cite more (arXiv). Vectara's summarisation leaderboard has the best models at roughly 1.8–3.3% for a much easier task. Rates have fallen; none is zero; a court requires zero. The practical consequence is a verification routine, not a model choice, and it is set out in AI hallucinations in legal practice and in the short answer to which AI has the lowest hallucination rate.
LegalBench: the academic baseline
LegalBench is the collaboratively built academic suite of legal reasoning tasks, most of them classification: does this clause do X, does this fact pattern satisfy Y. It is the only independent benchmark on the list, and it has reached a ceiling. Vals' September 2026 run has Fable 5, Fable 5.1 and Gemini 3.1 Pro within 1.2 points of each other, with 145 models scored (Vals benchmarks). It confirms that frontier models read law competently; it says nothing about drafting, research or agentic work, which is where the money and the risk are.
Why the results contradict each other
- Scoring rule. Partial credit produces 92.7%; all-pass produces 2.5%; both were GPT-5.6 Sol in July 2026. Ask which rule a number uses before comparing it with another.
- Task mix. BigLaw Bench skips case-law research; LegalBench is classification; HAQQ is citation-heavy; LAB is long-horizon agent work. A model can lead one and trail another honestly.
- Harness versus model. Astra for Law's 54% against raw GPT-6's 38.7%, and GC AI's 86.8% against raw Opus 4.7's 68.4%, measure the product wrapped around the model. So does a 30-lawyer firm's August 2026 bake-off in which only Claude Opus on maximum effort caught a disqualifying issue that CoCounsel, Harvey and Legora missed (r/legaltech, 30 Aug 2026). Effort settings, retrieval and prompts move scores more than model generations do.
- Who takes part. Westlaw, Lexis and vLex declined the Vals research study; Lexis+ AI withdrew from the earlier report. A leaderboard ranks the willing.
- Version churn. Anthropic shipped Opus 4.6 (February), 4.7 (April), 4.8 (May), Opus 5 (July) and Fable 5.1 (September) in 2026 alone; GPT-5.5 retires from ChatGPT on 14 October 2026. A score older than a quarter describes a model you may no longer be able to buy. Anthropic's claim that Opus 5's first-turn NDA redlines are "nearly double Opus 4.8" is a vendor claim about a vendor test.
- Cost and time are off the chart. $0.082 versus $0.0009 per task, 134 seconds versus a few, $50.90 and 22 minutes per LAB task: numbers that decide whether a workflow is viable rarely appear on a leaderboard.
Thomson Reuters' chief technology officer put the vendor view plainly: "The control point isn't where work starts. It's whether the output is accurate, grounded in authoritative sources." That is also the buyer's view, and the reason to test grounding on your own documents rather than trust any of the six numbers above.
Run your own 20-task evaluation
A firm evaluation does not need 1,200 tasks. It needs 20 real ones, a rubric written before anyone sees an output, and blind scoring. UBS's legal team started with 80 candidate use cases and cut to three with a 140% ROI hurdle; Addleshaw Goddard evaluated Legora for nearly a year before adopting it. A week is enough for the first pass.
- Choose 20 tasks from last quarter's real work, five in each of four categories: extraction and document question-answering; drafting or redlining against a playbook; a research memo where you supply the verified authorities; and summarisation or chronology. Use closed or anonymised matters.
- Write the rubric first. Three to six binary criteria per task, one of which is always "no authority, quotation or fact that is not in the supplied material". Have a second lawyer sign off the rubric before any tool runs.
- Fix the prompt. Use the same six-element prompt for every candidate; the templates in prompt engineering for lawyers are built for this. Record model, effort setting, cost and wall-clock time for every run.
- Score blind. Two lawyers, outputs de-identified and shuffled, disagreements resolved by a third.
- Report three numbers per tool, never one. All-pass rate, criteria pass rate, and count of citation or fact errors. Add cost and time.
- Decide with weights you chose beforehand, and keep the 20 tasks: they are your regression test for the next model release.
| Task | Category | Criteria | Tool A all-pass | Tool A criteria | Tool A errors | Tool B all-pass | Tool B criteria | Tool B errors | Cost / time |
|---|---|---|---|---|---|---|---|---|---|
| 1. Extract 12 fields from 5 leases | Extraction | 6 | Yes | 6/6 | 0 | No | 5/6 | 1 | $0.40 / 55 s vs $0.90 / 130 s |
| 2. Redline NDA v. playbook | Drafting | 5 | No | 4/5 | 0 | No | 3/5 | 0 | |
| 3. Memo from 4 supplied cases | Research | 5 | No | 4/5 | 1 | Yes | 5/5 | 0 | |
| ... |
Expect strict all-pass rates far below the criteria rates, as Harvey found. The gap is your supervision budget. The full procedure, including how to size the pilot and what to put in the vendor's contract, is in run an AI pilot in your firm, and the decision framework that follows the pilot is in how to choose a legal AI tool.
Where this leaves you
Read the leaderboards for direction, not decisions: the top three models are close on every test, Meta's Muse Spark leads two of them while its promised open weights remain unpublished, and the products wrapped around the models move results more than the models do. Buy on your own 20 tasks, re-run them every quarter, and treat any "hallucination-free" claim as a scoring choice until you have seen the rubric. Module 2 of the AI Lab for Lawyers course runs this kind of side-by-side on research and drafting tasks live, so that the first evaluation you score is not the one your firm is paying for. For a head-to-head on the two general models most firms shortlist, see Claude vs ChatGPT for lawyers.
Frequently asked questions
What is BigLaw Bench?
BigLaw Bench is Harvey's benchmark of legal tasks scored with partial credit against expert rubrics. Harvey publishes new model scores at each launch: GPT-5.6 Sol 92.7% in July 2026, GPT-5.5 91.7% in April, Claude Opus 4.7 90.9% in May. It is vendor-run, does not test case-law research, and its partial-credit scoring is why scores cluster in the low nineties.
Which AI model is best for legal work in September 2026?
It depends on the test. The Vals Index puts Claude Fable 5.1 first, Opus 5 second and GPT-6 Astra third; the Vals Legal Research Bench puts Meta's Muse Spark 1.3 Max first; HAQQ put Claude Opus 4.8 first overall but GPT-5.5 lowest for hallucination; Harvey's LAB puts Fable 5 first at 13.3% strict. The honest answer is that the top three models are close and the harness, prompt and verification matter more.
Are legal AI benchmarks independent?
Mostly not. BigLaw Bench and LAB are Harvey's, HAQQ and the GC AI In-House Legal Bench are run by the vendors they favour, and Vals studies are often commissioned by participants. LegalBench is academic. Thomson Reuters, LexisNexis and vLex declined to take part in the 2025 Vals research study. Treat every published score as a claim to test, not a finding to rely on.
How should a law firm evaluate AI tools itself?
Pick 20 real tasks from recent matters across extraction, drafting, research with supplied authorities and summarisation. Write a binary rubric for each before running anything, including a criterion for invented authority. Run every candidate with the same prompt, record cost and time, score blind with two lawyers, and report all-pass rate, criteria pass rate and citation errors separately.
Put this into practice
AI Lab for Lawyers: a live, hands-on course for practising lawyers
4 live sessions (8 hours), recordings included, taught by Dr. Niklas Schmidt of Wolf Theiss. Rated 4.7/5 from 381 ratings; alumni from Norton Rose Fulbright, Paul Hastings, Pillsbury and Taylor Wessing.
Related reading
Guides
How to Choose an AI Tool for Your Law Firm: A Decision Framework
How to choose legal AI for your law firm: four routes, six vendor questions, a weighted scoring matrix and a six-week pilot, with September 2026 prices.
11 min read
Playbooks
Run a Six-Week AI Pilot in Your Firm: Design, Metrics and the Go/No-Go
Run a law firm AI pilot in six weeks: three use cases, ten users, a 20-task evaluation set, baseline metrics, security sign-off and a go/no-go scorecard.
11 min read
Questions
Which AI Has the Lowest Hallucination Rate for Legal Work?
The lowest hallucination rate AI for legal work as of September 2026: GPT-5.5 at 3% on HAQQ, Claude Opus 4.8 best overall, and why no model reaches the zero courts demand.
6 min read
Guides
AI Hallucinations in Legal Practice: Why They Happen and How to Catch Them
AI hallucinations in legal practice: why models invent cases, what 2,041 court decisions and the HAQQ benchmark show, and a verification routine that holds.
9 min read
Comparisons
Claude vs ChatGPT for Lawyers (2026): Which Should You Use?
Claude vs ChatGPT for lawyers in 2026, compared task by task: drafting, research, agents, data terms by plan, prices and benchmarks, plus a pick by firm type.
10 min read
Glossary
Reasoning Model — Definition for Lawyers
A reasoning model works through intermediate steps before it answers. What adaptive thinking and effort settings mean for legal analysis, cost, speed and hidden reasoning.
4 min read