AI Agents for Lawyers

Use Cases

AI Contract Review: How Agents Redline Against a Playbook

AI contract review works when an agent applies your playbook of positions, fallbacks and escalations. Tools, accuracy numbers, setup and failure modes for 2026.

By Dr. Niklas SchmidtUpdated 13 November 20268 min read
On this page
  1. What "review against a playbook" actually means
  2. The tools, as of September 2026
  3. How accurate is it, honestly
  4. A setup that works: five steps
  5. Where agents fail on contracts
  6. Who signs off
  7. What to do this week

AI contract review means an agent reads a counterparty's draft clause by clause, compares each clause with your playbook of preferred positions and fallbacks, and returns tracked changes with a comment explaining every edit. As of September 2026 it is reliable enough to produce a first-turn redline of an NDA or a standard MSA, and not yet reliable enough to send without a lawyer reading the result.

What "review against a playbook" actually means

The model is the least important part of the setup. What matters is the contract playbook: a table with one row per clause that says what you want, what you will accept, and when a human has to decide. Agents that redline well are agents that have been given that table. Agents that redline badly are usually agents that were asked to "review this for risk" with nothing else.

Three columns do the work:

ClausePreferred positionFallbackEscalate when
Limitation of liabilityCap at 12 months' fees; indirect loss excludedCap at 24 months' fees if the cap is mutualUncapped liability requested, or carve-outs beyond IP, confidentiality and data
Governing law and forumAustrian law, courts of ViennaEnglish law and LCIA arbitration for non-EU counterpartiesAny US state law, or a seat outside the EU/UK
AssignmentCounterparty needs consent; we may assign to affiliatesConsent not to be unreasonably withheldFree assignment to any third party
Term and renewal12-month terms, 90 days' notice60 days' noticeEvergreen term with under 30 days' notice
Data protectionOur DPA attached; SCCs where transfers occurCounterparty DPA if it passes our ten-point checklistProcessing outside the EEA/UK with no transfer mechanism

Anthropic's open-source Claude for Legal commercial-legal plugin shows how vendors now encode this. Every plugin starts with a "cold-start interview" that writes a practice profile; the review skill then triages clauses RED, YELLOW or GREEN, writes MSA, NDA and SaaS redlines to Word tracked changes, and runs a "playbook-monitor" agent that flags when negotiated outcomes drift from the written positions (repo, 18 Sep 2026). The playbook automation playbook on this site walks through building the table itself.

The tools, as of September 2026

ToolWhere it runsWhat it does with a playbookPrice signal
Microsoft Word Legal AgentWord (Frontier preview from 30 Apr 2026; US and Windows first)Clause-by-clause comparison, tracked changes with explanatory comments via a "purpose-built insertion algorithm"; built by the ex-Robin AI team; needs Anthropic enabled as a subprocessorInside Microsoft 365 Copilot at $30/user/month
SpellbookWord add-inReview and redline; Autonomous Contract Management in early access since 30 Jun 2026About $180/month; ~4,000 customers
Claude for Legal (commercial-legal plugin)Cowork, Claude Code, Managed Agents, Microsoft 365MSA/NDA/SaaS review to tracked changes; RED/YELLOW/GREEN triage; 90-day renewal trackerNo extra charge on paid Claude plans; Apache 2.0
HarveyPlatform, Word and Outlook add-insAgentic playbook review (May 2026); 500+ pre-built agents; Agent BuilderNot published; about $1,000–1,200 per seat per month with a 20-seat minimum (secondary sources)
Legora aOSPlatform, Word and OutlookOvernight counterparty redlines "reviewed, flagged and responded to by the time lawyers arrive"From about $3,000/user/year, 10-seat minimum (secondary source)
Litera Kira / LitoPlatformHybrid GenAI plus proprietary engine since Jan 2026; deterministic redline comparisonEnterprise
IroncladCLM; "Jurist" inside Word since Jun 2026Redlining Agent and Intake Agent (13 Nov 2025)Enterprise; ~$200m ARR (secondary)
LegalOnPlatformPlaybook and Triage agents; Triage auto-approves low-risk contracts or escalatesAbout $550/month per comparison sites

Sources: Legal IT Insider on the Word Legal Agent (1 May 2026), Law.com on Spellbook ACM (30 Jun 2026), Artificial Lawyer on Legora aOS (7 May 2026), LegalOn (10 Feb 2026). Reviews of the Word Legal Agent, Spellbook, Harvey and Legora go deeper on each.

The surface matters more than the brand. If your lawyers negotiate in Word, a tool that writes real tracked changes with comments beats a platform that returns a PDF report. If you handle hundreds of contracts a month, a platform with intake, triage and a renewal tracker beats an add-in. The CLM comparison covers the case where you already own Ironclad or a similar system and are deciding whether to layer an agent on top.

How accurate is it, honestly

Four numbers frame the answer, and they point in the same direction:

  • RedlineBench 57.0. GC AI's contract-redlining benchmark rose from 47.9 to 57.0 when Claude Fable 5.1 shipped on 1 September 2026 (GC AI). Vendor-run, but a real jump.
  • Lawyers 79.7%, best tool 65.0%. In Vals AI's Legal AI Report of 27 February 2025, lawyers beat every tested tool on redlining, while Harvey scored 94.8% on document Q&A against a lawyer baseline of 70.1% (Vals AI). Reading contracts is solved; deciding what to change is not.
  • "Nearly double Opus 4.8." Anthropic says Claude Opus 5 (24 July 2026) scored the highest of any model it tested on first-turn NDA redlines (Anthropic). No absolute figure was published.
  • 13.3% all-pass. On Harvey's LAB benchmark, where every rubric criterion must pass, the best model (Claude Fable 5) completed 13.3% of long-horizon tasks flawlessly as of 18 September 2026, with criteria-level pass rates around 83–88% (LLM Stats).

Translated: the agent will find the clause, quote it correctly and propose a change consistent with the playbook most of the time. About one edit in five to one in three needs a lawyer's hand, and you cannot tell which without reading. The benchmarks guide explains why the headline scores and the strict scores diverge so much.

A setup that works: five steps

  1. Write the playbook in the agent's format. Twenty to forty rows for your most common paper (NDA, MSA, DPA, SaaS order form). Positions, fallbacks and escalation triggers in plain sentences, no internal shorthand. If the row cannot be written, the agent cannot apply it.
  2. Choose the surface. Word add-in, platform or open-source plugin, using the table above. Check the data terms before the features: the Word Legal Agent, for instance, requires an admin to enable Anthropic as a subprocessor.
  3. Calibrate on ten closed deals. Run the agent on the counterparty's first draft from ten past negotiations and compare its redline with the concessions you actually made. Adjust rows where the agent was harsher or softer than practice.
  4. Set the gates. Green: changes the agent may apply without comment (notice periods, boilerplate). Yellow: changes it proposes with a comment. Red: clauses it must never edit, only flag (liability caps, IP assignment, governing law, exclusivity).
  5. Name the reviewer and log the run. Every outgoing redline carries a lawyer's name. Keep the agent's run log with the matter file, and review the playbook-monitor output monthly.

Module 4 of the AI Lab for Lawyers course builds exactly this kind of playbook agent live, from the first row to the first tracked change.

Where agents fail on contracts

The errors that get past reviewers are rarely spelling or tone. They are structural:

  • Cross-references. The agent tightens clause 14.3, and clause 9.2 still says "subject to clause 14.3", which now says something else. Renumbering after an insertion is worse: check every "clause X" reference after the run.
  • Defined terms. "Services" becomes "the services"; a new capitalised term appears in the body with no definition; "Confidential Information" is narrowed in the definition and widened in the operative clause. Ask the agent for a defined-terms table as a separate output and compare it with the definitions clause.
  • Governing-law traps. A playbook written for English or Delaware paper contains positions that are pointless or unenforceable under Austrian or German law, such as excluding liability for gross negligence in standard terms, which § 309 no. 7 BGB blocks in German standard business terms. Keep one playbook per governing law, or at least a jurisdiction column.
  • Silent deletions. Some setups delete a sentence rather than striking it through. Insist on tracked changes; the Word Legal Agent's insertion algorithm preserves lists, tables and existing tracked changes, which is why Microsoft mentions it (Chris Menard's walkthrough, 13 May 2026).
  • Schedules and order of precedence. The agent reviews the body and ignores the SOW, the order form or the counterparty's terms incorporated by reference. Feed the whole bundle, and tell it which document wins on conflict.
  • Instructions hidden in the draft. A counterparty document can carry text the agent reads as an instruction. Treat every inbound draft as untrusted input; the agent security guide explains the controls.

Who signs off

Legora's aOS pitch is that counterparty redlines arrive overnight, already answered. The question that follows is who signed them off. Nothing in the professional rules has moved: ABA Formal Opinion 512 keeps supervision and competence with the lawyer, the Austrian bar's four practice rules say the lawyer keeps final responsibility, and Clifford Chance's briefing of 9 February 2026 on "the liability gap your contracts may not cover" lists hallucinated commitments and agency questions as live risks (Clifford Chance).

The commercial pressure runs the other way. Crosby, a New York firm of about 30 lawyers plus agents, had reviewed 13,000 contracts by March 2026, bills $250 to $1,000 per contract with no hourly element, and raised $60m at a reported $400m valuation (Forbes, 31 Mar 2026). Clients who see that price list will ask why an NDA still costs four hours. The NDA-at-scale workflow is the answer for the highest-volume paper, and the drafting page covers the first draft rather than the response.

What to do this week

  • Pull your last twenty negotiated NDAs and write fifteen playbook rows from the concessions you actually made. That table is the asset; the tool is replaceable.
  • Run one tool from the table above on the counterparty's first draft of a closed deal. Compare its redline with yours, clause by clause, and note where it was harsher, softer or wrong.
  • Decide the red list before anyone runs anything on live paper: the clauses an agent may flag but never edit.
  • Put a named reviewer and a run log into the process. If a redline leaves the firm, a lawyer's name is on it.

Frequently asked questions

What is AI contract review and how does it work?

An AI contract review agent reads a counterparty draft clause by clause, compares each clause with your firm's playbook of preferred positions and fallbacks, and returns tracked changes with a comment explaining each edit. The lawyer then accepts, rejects or escalates each change. The playbook, not the model, does most of the work.

How accurate is AI contract review in 2026?

Good enough for a first-turn redline of standard paper, not good enough to send unread. GC AI's RedlineBench reached 57.0 with Claude Fable 5.1 in September 2026, up from 47.9. In Vals AI's February 2025 test, lawyers scored 79.7% on redlining against 65.0% for the best tool, while AI beat lawyers comfortably on document questions.

Which tools review contracts against a playbook?

As of September 2026 the main options are Microsoft's Word Legal Agent, Spellbook, Anthropic's open-source Claude for Legal commercial plugin, Harvey, Legora's aOS, Litera Kira, Ironclad's Redlining Agent and LegalOn. They differ mainly in where they run, whether they write real tracked changes and what they cost per seat.

Can an AI agent sign off on a contract without a lawyer?

Not in any regulated practice. ABA Formal Opinion 512, the Austrian bar's four practice rules and the SRA's 2026 warning notice all keep final responsibility with the lawyer. Agents can auto-accept green-listed changes under a written policy, but every outgoing redline should carry a named reviewer.

Put this into practice

AI Lab for Lawyers: a live, hands-on course for practising lawyers

4 live sessions (8 hours), recordings included, taught by Dr. Niklas Schmidt of Wolf Theiss. Rated 4.7/5 from 381 ratings; alumni from Norton Rose Fulbright, Paul Hastings, Pillsbury and Taylor Wessing.

Reserve a seat · €200 Maven money-back guarantee · certificate of completion