Playbooks
Run a Six-Week AI Pilot in Your Firm: Design, Metrics and the Go/No-Go
Run a law firm AI pilot in six weeks: three use cases, ten users, a 20-task evaluation set, baseline metrics, security sign-off and a go/no-go scorecard.
On this page
A law firm AI pilot that ends in a decision, rather than a vague sense that people liked the tool, needs six things fixed before anyone logs in: three narrow use cases, ten named users, a baseline for time, quality and rework, a 20-task evaluation set, a security sign-off and a weighted go/no-go scorecard. Six weeks of live use is enough to run all of that. Longer pilots drift.
Why most pilots end without a decision
Thomson Reuters' Future of Professionals 2026 report (June 2026; 1,816 respondents in 62 countries) contains the number that should shape every pilot design: 66% of professionals whose organisation has a named AI strategy say AI meets their expectations, against 22% where there is none (Thomson Reuters Institute). A pilot is the point at which a strategy either becomes specific or stays a slogan.
The failure modes are well documented and rarely technical. Axiom's July 2026 survey of 528 in-house leaders found only 7% use, optimise and measure AI at scale, and 83% cannot show whether their spend paid off. Gartner predicted in June 2025 that more than 40% of agentic AI projects would be cancelled by the end of 2027 (Gartner). The 8am 2026 Legal Industry Report found 54% of firms offer no AI training and have no plans to, while individual use had jumped from 31% to 69% in a year (LawSites). Lawyers are using the tools; firms are not learning from it.
| Failure pattern | What it looks like | The fix built into this playbook |
|---|---|---|
| No champion | The pilot is "owned" by IT or a committee; nobody's name is on the go/no-go | One partner sponsor and one day-to-day lead, both named in week 0 |
| No training | Licences issued by email; usage collapses after week 2 | Two hours of hands-on training before any task is scored |
| Wrong tasks | Redlining and open-ended research, where the tool still loses to lawyers | Tasks chosen from where AI already wins on independent tests |
| No baseline | "It felt faster" is the only evidence | Timed, scored human baseline on the same 20 tasks |
| Too many users | 60 licences, 8 active users, no comparable data | Ten users, three use cases, weekly check-ins |
| No security sign-off | Pilot runs on consumer accounts with client data in a training-enabled chat | Plan tier, data terms and matter selection cleared before week 1 |
Week 0: choose three use cases and ten users
UBS narrowed 80 candidate AI use cases to three before committing, with a 140% return-on-investment target (March 2026). Three use cases give comparison without dilution; ten users show variance between people without needing a change-management programme.
Pick tasks where the evidence already favours the tool. In the Vals Legal AI Report of 27 February 2025, AI systems beat a lawyer baseline on document question-and-answer (Harvey 94.8% against 70.1%), summarisation and extraction, while lawyers won redlining (79.7% against Harvey's 65.0%) and EDGAR research (Vals.ai). Vals' October 2025 research study put ChatGPT Team at 80% weighted accuracy on 199 US questions against a 69% lawyer baseline, but every system lost roughly 11 to 14 points on multi-jurisdiction questions (Vals.ai). Harvey's own LAB benchmark (26 May 2026) showed the best model passing every rubric criterion on only 7.1% of long-horizon tasks even though criterion-level pass rates sat around 83 to 88% (Harvey). Drafts arrive mostly right and rarely wholly right, which is exactly the profile a pilot has to measure; the benchmarks guide explains what each test does and does not show.
| Use case | Fit for a first pilot | Why |
|---|---|---|
| NDA and MSA first-pass review against a written playbook | High | Repetitive, high volume; the playbook makes quality measurable |
| Document Q&A and summarisation of a bundle or data room | High | AI beat lawyers by about 25 points on Vals document Q&A |
| Chronology from a document set | High | Extraction task; errors are visible against the source |
| Single-jurisdiction research memo | Medium | Works, but verification time must be counted in the time metric |
| Redlining a negotiated agreement | Low | Lawyers still win on independent tests; keep for pilot two |
| Multi-jurisdiction research | Low | Every system tested drops 11 to 14 points |
The NDA review workflow is a ready-made candidate for the first row.
For users, choose two partners, five associates and three business-services staff spread across the three use cases, and include one visible sceptic. A pilot of enthusiasts proves nothing to the partnership. GSK Stockmann's published diligence figures with Harvey, 15 to 20% time savings on standard diligence and up to 75% on unstructured data rooms (29 April 2026), are the kind of spread a ten-user pilot can reproduce or refute. Single anecdotes cannot: the Legalweek 2026 panel claim that financial-contract diligence had gone from 15 to 20 associate hours to about two was a story told on stage, not a study.
Baseline before you switch anything on
The baseline is what turns a pilot into evidence. For each use case, assemble 20 closed, anonymised, real tasks with the source documents and a "gold" answer the supervising partner has signed off. Have the pilot users complete five of them by hand, timed, before they see the tool. That gives a baseline from the same people who will later use the AI, which removes the objection that the comparison group was different.
Three metrics, measured the same way before and during the pilot:
- Time: minutes from receiving the task to a draft the supervising lawyer would accept, including every minute spent verifying citations and checking sources. A tool that produces a memo in four minutes that takes forty to verify has saved nothing.
- Quality: a score against five to eight criteria per task, written by the partner before the pilot starts. "Identified the change-of-control clause", "flagged the missing governing-law provision", "cited no authority outside the bundle".
- Rework: the number of substantive changes the reviewer makes before sign-off. Count them. This is also your supervision record under ABA Formal Opinion 512 (29 July 2024), and it evidences what the SRA's warning notice of 17 August 2026 asks for: understanding a tool "well enough to review its output critically".
Seed at least three of the 20 tasks with a trap: two documents that contradict each other, a bundle missing the clause the question asks about, or a request that invites the tool to cite authority the bundle does not contain. Whether the tool says "not in the documents" or invents an answer is the most important single data point the pilot produces. The citation-checking protocol supplies the verification step for any research task.
EVALUATION SET — [Use case: NDA first-pass review] Owner: [Partner]
ID Source docs Gold answer (signed off) Criteria (score 0/1 each)
-------------------------------------------------------------------------------
N01 NDA_acme_v3.docx Issues memo, 6 points [ ] term flagged [ ] carve-outs
[ ] governing law [ ] no invented clause
N02 NDA_beta_v1.docx Issues memo, 4 points ...
N07 NDA_gamma.docx TRAP: no confidentiality [ ] says clause absent
(clause deleted) period in document [ ] does not supply one
...
N20
Per task, per user: Baseline time ___ AI time ___ Quality __/8 Rework ___
Verification issues found: ___ Trap handled: Y/N
The six-week calendar
Freshfields reported roughly 500% growth in Claude usage within six weeks of its firm-wide rollout (May 2026), and a three-lawyer German employment and tenancy firm took about six weeks to roll out a self-hosted n8n intake system that now saves it six to seven hours a week (Lyron case study). Six weeks is enough for behaviour to settle, at either end of the market.
- Two weeks before (week 0). Name the partner sponsor and the day-to-day lead. Confirm the three use cases and ten users. Build the 20-task sets. Run the hand baseline on five tasks per use case. Complete the security sign-off below. Agree the scorecard weights and thresholds in writing, before anyone has a result to argue about.
- Week 1. Two hours of hands-on training per user on the actual tasks, not a vendor demo. Tasks 1 to 5. Friday review: usage logs, first time and quality numbers, any refusal or hallucination noted. The AI Lab for Lawyers course covers Claude, ChatGPT, Harvey and Legora with browser-based tools only, which suits pilot users who need to be competent before week one rather than during it.
- Week 2. Tasks 6 to 10. Check who has gone quiet; a user who stopped after week 1 is data, so ask why and record it.
- Week 3. Mid-point review with the sponsor. Compare the three metrics against the baseline. Adjust prompts, playbooks and instructions, then freeze them; changes after this point contaminate the comparison. Introduce the trap tasks.
- Week 4. Tasks 11 to 15. Interview the sceptic on record. Check whether any pilot matter requires client disclosure: Clio's 2025 Legal Trends Report found 78% of clients want to know when AI is used, and the CCBE's October 2025 guide expects lawyers to inform clients and let them object (disclosure rules).
- Week 5. Tasks 16 to 20. Send the vendor the commercial questions: seat minimums, per-user pricing at the firm's real scale, data-export format, notice period and what happens to matter data on exit.
- Week 6. Score the cards. Write a two-page decision memo with the numbers and present it to the partnership: go, extend for one specific reason, or stop.
The security and ethics gate
This is a gate, not a score. A pilot that fails any line below stops, whatever the time savings.
- The tool runs on a business, Team, Enterprise or API tier that does not train on prompts. Consumer tiers of ChatGPT, Claude and Gemini train unless each user opts out, and OpenAI keeps consumer data 30 days for abuse monitoring even after opt-out. The confidentiality checklist lists each vendor's terms as of September 2026.
- A data processing agreement (Art. 28 GDPR for EU and UK firms) is signed before the first client document is uploaded.
- Pilot matters are selected where the engagement terms permit the tool. In United States v. Heppner (S.D.N.Y., opinion 17 February 2026) Judge Rakoff held that research memos produced in a consumer Claude account were neither privileged nor work product because the terms allowed logging and disclosure.
- Vendor due diligence is complete, using the questions to ask before you sign. Data export is a contractual right, not a hope: ChatGPT Atlas lasted 292 days; OpenAI announced in June 2026, eight months after launch, that Agent Builder shuts on 30 November 2026; Relay.app closed to paying customers on 14 September 2026.
- The two hours of training are logged with names, dates and content. The EU AI Act's Article 4 literacy duty has applied since 2 February 2025 and the Commission's FAQ expects an internal training record (Article 4 explained).
- Billing is settled before the first pilot task touches a client file: ABA Opinion 512 and Texas Opinion 705 both say lawyers may bill actual time but not time saved, and subscriptions are generally overhead.
- The interim AI policy tells pilot users in one page what may and may not go into the tool.
The go/no-go scorecard
Agree the weights and thresholds in week 0. Score in week 6. Publish the sheet to every pilot user so the result is not a surprise.
| Criterion | Weight | Measure | Score 0 | Score 5 | Score 10 |
|---|---|---|---|---|---|
| Time | 25% | Median minutes to accepted draft, incl. verification, vs baseline | No saving or slower | 15% faster | 40% or more faster |
| Quality | 25% | Mean rubric score vs hand baseline | Below baseline | Equal | Above baseline on 2 of 3 use cases |
| Rework | 15% | Substantive reviewer changes per task | More than baseline | Equal | Half or fewer |
| Trap handling | 10% | Trap tasks where tool said "not in documents" rather than inventing | Below 50% | 80% | 100% |
| Adoption | 15% | Users completing all 20 tasks by week 6 | Fewer than 5 of 10 | 7 | 9 or more |
| Cost per user at real scale | 10% | Annual licence cost vs hours saved at the user's rate | Cost exceeds value | Break-even | Value 3x cost or more |
| Security gate | Pass/fail | Every line of the gate above | Fail stops the pilot |
A weighted total of 70 or above with the gate passed is a go. Between 50 and 69, extend for one named reason and one more cycle, not indefinitely. Below 50, stop and write down why; the reasons are worth more to the next pilot than the licences.
For the cost row, use real prices at real scale rather than pilot discounts. As of September 2026, Claude Team is $25 per standard seat with a five-seat minimum, Microsoft 365 Copilot is $30 per user per month, Clio Work is $199 per user per month, Spellbook runs roughly $99 to $350, and secondary sources put Harvey at about $1,000 to $1,200 per seat per month with a 20-seat minimum and Legora at about $3,000 per user per year with a 10-seat minimum. The pricing guide has the detail; the decision framework sits one level above this playbook.
What to do this week
Book thirty minutes with one partner who will put their name on the go/no-go. Pick the three use cases from the fit table above and ask practice-group heads for 20 closed matters each to build the task sets. Send the DPA to the vendor today; the contract will take longer than the pilot. The pressure is not going away: Modern Counsel reported on 3 September 2026 that Citi now asks law firms to document AI savings and Morgan Stanley is expanding competitive bidding, and Thomson Reuters found that 32% of in-house teams will reconsider firms that show no AI value within twelve months. A six-week pilot with numbers is the answer to that question.
Frequently asked questions
How long should a law firm AI pilot last?
Six weeks of live use, preceded by roughly two weeks of preparation, is long enough to score 20 tasks per use case, run three weekly reviews and see whether usage holds after the novelty fades. Pilots that run for months without a fixed end date tend to end without a decision; Addleshaw Goddard's roughly one-year Legora evaluation is the exception that needed a large firm's resources.
How many people should take part in an AI pilot at a law firm?
Ten users across three use cases is the practical floor for seeing variance between people and the practical ceiling for weekly check-ins. Mix two partners, five associates and three business-services staff, and include one open sceptic. Sixty licences with eight active users produces less usable data than ten users who complete every task.
What metrics should an AI pilot measure?
Three per task: time to an accepted draft including verification, quality against a rubric the supervising partner wrote in advance, and rework, meaning the number of substantive changes the reviewer made. Add adoption (tasks completed per user per week) and a pass-or-fail security gate. Cost per user only matters once the first three show a real difference.
Which AI use cases should a law firm pilot first?
Tasks where independent benchmarks show the tool ahead of lawyers: document question-and-answer, summarisation, data extraction and first-pass review against a written playbook. Vals' February 2025 report had AI beating a lawyer baseline on document Q&A by about 25 points but losing on redlining, and every system tested in October 2025 lost 11 to 14 points on multi-jurisdiction research questions.
Can we run the pilot on personal ChatGPT or Claude accounts?
No. Consumer tiers train on conversations unless each user opts out, and a Manhattan federal court held in February 2026 that research memos produced in a consumer Claude account were not privileged. Use Team, Business, Enterprise or API tiers with a signed data processing agreement, and confine the pilot to matters where the engagement terms permit the tool.
Put this into practice
AI Lab for Lawyers: a live, hands-on course for practising lawyers
4 live sessions (8 hours), recordings included, taught by Dr. Niklas Schmidt of Wolf Theiss. Rated 4.7/5 from 381 ratings; alumni from Norton Rose Fulbright, Paul Hastings, Pillsbury and Taylor Wessing.
Related reading
Guides
How to Choose an AI Tool for Your Law Firm: A Decision Framework
How to choose legal AI for your law firm: four routes, six vendor questions, a weighted scoring matrix and a six-week pilot, with September 2026 prices.
11 min read
Guides
What Legal AI Really Costs in 2026: Harvey, Legora, CoCounsel, Claude and ChatGPT
Legal AI cost as of September 2026, from €39 to $1,200 per seat per month, with Harvey, Legora, CoCounsel, Claude and ChatGPT prices and worked budgets for 5 and 50 lawyers.
10 min read
Regulation & Ethics
AI Vendor Due Diligence for Law Firms: The Questions to Ask Before You Sign
AI vendor due diligence for law firms starts with the subprocessor chain, then training and retention terms, ZDR eligibility, data residency, audit logs and exit rights.
11 min read
Playbooks
Write a Law Firm AI Policy in a Day: A Working Template
A working law firm AI policy template with ten clauses, commentary on each, EU and US variants and an hour-by-hour rollout plan that gets it signed in one day.
12 min read
Guides
Legal AI Benchmarks Explained: Vals, BigLaw Bench, LAB, HAQQ and LegalBench
Legal AI benchmarks explained: what Vals, BigLaw Bench, Harvey LAB, HAQQ and LegalBench measure, the September 2026 leaders, and how to run your own evaluation.
10 min read
Regulation & Ethics
EU AI Act Article 4: The AI Literacy Duty for Law Firms, Explained
Article 4 of the AI Act requires every firm whose staff use AI to support their AI literacy. In force since 2 February 2025, supervised nationally since August 2026.
7 min read