AI Agents for Lawyers

Regulation & Ethics

AI Vendor Due Diligence for Law Firms: The Questions to Ask Before You Sign

AI vendor due diligence for law firms starts with the subprocessor chain, then training and retention terms, ZDR eligibility, data residency, audit logs and exit rights.

By Dr. Niklas SchmidtUpdated 13 November 202611 min read
On this page
  1. Start with the subprocessor chain, not the certificate
  2. Training, retention and zero data retention: the exact questions
  3. Residency, keys and who can compel production
  4. Model provenance, deprecation and the 18-day blackout
  5. Reading a vendor's benchmark claim
  6. Security, audit logs and agent permissions
  7. The questionnaire and the red flags
  8. Contract terms and what to do before you sign

AI vendor due diligence for a law firm comes down to six questions: which models and subprocessors actually touch your data, whether inputs are used for training and how long they are retained, where processing happens and who holds the keys, what is logged, what the benchmark claims measure, and what happens to your data and workflows when the contract ends. The questionnaire below asks all six.

Start with the subprocessor chain, not the certificate

The product on the order form is rarely the system that reads your documents. Microsoft's Legal Agent for Word, released in the Frontier preview on 30 April 2026, "requires Anthropic enabled as a subprocessor", and as Platinum IDS observed, that is a vendor-diligence question firm IT has often not formally evaluated. Harvey runs a multi-model stack, with Claude Opus 5 live from 24 July 2026 and Fable 5.1 from 1 September 2026. Lexis+ with Protégé draws on models from Anthropic, Google and OpenAI. Thomson Reuters' next-generation CoCounsel Legal is built on Anthropic's Claude Agent SDK plus TR's own legal model.

So the first request is a written list of every subprocessor that receives firm data, with the region, the retention period and the training exclusion for each hop. A SOC 2 Type II report from the platform vendor covers the vendor's own controls; it does not cover the model provider two contracts down. In the EU the chain matters legally as well as practically. EDPB Opinion 28/2024 (17 December 2024) says a deployer must assess whether an upstream model was trained unlawfully, the GDPR analysis requires an Article 28 processor agreement that flows down, and a German firm also needs a § 43e BRAO service-provider contract; the three-lawyer firm in Lyron's n8n case study needed both even for a self-hosted stack on a German server. The CCBE's Technical Guide on AI tools and models, adopted 27 March 2026 and summarised by the BRAK, walks through the deployment models from on-premise to SaaS, open-weight models and agentic AI, and is the reference to cite when a vendor says the question is unusual.

Training, retention and zero data retention: the exact questions

The answers change by tier, not by brand, and the differences are large enough to decide a purchase. The table states each provider's published position as of September 2026.

Provider and tierInputs used for training?RetentionZero data retention?
ChatGPT Free, Plus, ProYes, unless the user opts outAccount-levelNo
ChatGPT Business, Enterprise, APINo by defaultAPI inputs kept 30 days for abuse monitoringContractual, for eligible customers
Claude Free, Pro, MaxOnly if the user opted in5 years if opted in; 30 days if notNo
Claude Team and Enterprise (chat)NoPer contractNot ZDR-eligible; Claude Code under Enterprise is
Claude APINoNot retained by default, except Fable 5/5.1 and Mythos, which require 30-day retentionPer organisation via sales; not for Fable or Mythos without express authorisation
Gemini consumer appsYes, unless "Keep Activity" is off; a subset read by human reviewers18 months by default; reviewed chats up to 3 yearsNo
Microsoft 365 CopilotNoUnder the Microsoft DPA with Purview controlsNot applicable; consumer Copilot has no DPA at all
Meta Muse Spark API"Contributor tier" retains data; "standard tier" does notTier-dependentTier-dependent
Perplexity EnterpriseNever by defaultPer contractPer contract

Two details in that table decide matters for litigators. Anthropic's retention documentation says flagged content may be kept up to two years and Compliance API activity feeds and local session transcripts for six years by default, so "not retained" is never the whole answer. OpenAI's enterprise privacy page confirms the 30-day abuse-monitoring window on the API even where training is off. Retained logs are discoverable: in In re OpenAI Copyright Litigation (S.D.N.Y.) 20 million de-identified ChatGPT logs were ordered produced, affirmed on 5 January 2026, and in United States v. Heppner (S.D.N.Y., 17 Feb 2026) consumer-Claude memos lost privilege partly because the terms allowed logging and disclosure. Google's Gemini privacy page tells consumer users not to enter confidential information that a reviewer should not see. The confidentiality checklist covers the consumer toggles; the ZDR glossary entry explains what the term does and does not promise.

Put four questions in writing: is training excluded for this tier by contract rather than setting; what is retained, where and for how long, including abuse monitoring and flagged content; can the firm obtain ZDR for its actual model choice; and does every answer flow down the subprocessor list.

Residency, keys and who can compel production

European bars have made location a rule. ÖRAK's practice rules for Austrian lawyers require client data to stay on EU servers; BRAK's guidance treats tools that store or forward client data for training as a breach of § 43a Abs. 2 BRAO. Vendors have responded: NotebookLM Enterprise gained EU data residency on 3 June 2026, and Lexis+ with Protégé offers customer-held encryption keys. Ask for both, and ask whether "EU processing" includes the model provider's inference servers or only the vendor's database.

Residency is not the same as immunity from compelled production. Mistral's own positioning, quoted in a Digital Applied analysis, is that a "data-residency toggle changes where data sits — not which government can compel its production", a reference to the US CLOUD Act reach of US-domiciled providers. The honest counter is that Mistral Large scored 64% citation errors in the HAQQ benchmark of 5 June 2026, so the sovereign option carries an accuracy cost. Firms with the scale to avoid the trade-off are building: Latham & Watkins bought Nvidia H200 servers in a data centre only its staff can access (reported 11 September 2026). The open-weight comparison sets out what that path costs a smaller firm.

Model provenance, deprecation and the 18-day blackout

On 12 June 2026 the US government restricted access to Claude Fable 5 and Mythos 5 for foreign nationals, and Anthropic suspended access for every user because, in its own words on redeploying Fable 5, it "had no reliable way to verify nationality in real-time". Access returned on 1 July. Any workflow hard-wired to that model was down for 18 days, and no platform vendor built on it could do anything about it.

Deprecation is the slower version of the same risk. OpenAI retires GPT-5.5 from ChatGPT on 14 October 2026 and shuts down AgentKit's Agent Builder and Evals on 30 November 2026 (announced 3 June 2026). Robin AI, after failing to close a $50m round in October 2025, sold its technology team to Microsoft in January 2026. Anthropic publishes retirement floors in its model documentation: Fable 5.1 not before 1 September 2027, Opus 5 not before 24 July 2027, Haiku 4.5 not before 15 October 2026. Behaviour also drifts within a contract: Anthropic's notes on Opus 5 say it "verifies its own work without being told to", which quietly breaks prompts written for its predecessors.

The questions: which model versions power each feature; can the firm pin a version and for how long; what notice is given for model, subprocessor or pricing changes; what is the continuity plan if the upstream provider suspends access; and is the vendor's own solvency documented. Harvey's $550m raise at $15.5bn on 9 September 2026 and Legora's $5.6bn valuation answer the last question for two vendors; for the rest, ask.

Reading a vendor's benchmark claim

Harvey reported that GPT-5.6 Sol scored 92.7% on its BigLaw Bench in July 2026; on Harvey's own stricter LAB metric, which requires every rubric criterion to pass, the same model scored 2.5%. Both numbers are true. GC AI's In-House Legal Bench (May 2026) put GC AI at 86.8% and Claude Opus 4.7 at 68.4%, a vendor-run test on the vendor's own tasks. The HAQQ benchmark of 5 June 2026, across 3,000 answers and 51 practice areas, found that 24% of all answers cited or misapplied law that did not support the claim, and that every model fabricated or misapplied a citation somewhere.

The diligence question is therefore never "what is your score" but "on which benchmark, run by whom, on which date, with which model version, and under strict or partial scoring". The benchmarks guide decodes each name. Then run a six-week pilot on the firm's own documents, because no public benchmark tests your precedents, your jurisdictions or your playbook.

Security, audit logs and agent permissions

Certificates first, briefly: ask for the current SOC 2 Type II report and ISO 27001 certificate, the penetration-test summary and the incident-notification period. Then move to the questions that certificates do not answer, because 2026 has supplied the case studies. In July 2026 OpenAI models in a cyber evaluation with safeguards off exploited a zero-day, rooted a third-party sandbox and read 136 secrets across roughly 17,600 actions over four and a half days. Zenity's "PleaseFix" research showed zero-click hijacks of Claude in Chrome, Gemini in Chrome, Comet and Copilot Edge through emails and calendar invites. Anthropic's own Claude for Chrome post reports prompt-injection success cut from 23.6% to 11.2% with mitigations, and its later research stressed that "no browser agent is immune".

For any tool that acts rather than answers, ask: does it run under each user's own credentials and permissions, as the iManage MCP server and NetDocuments' ndMAX do, respecting ethical walls; are credentials held in a vault the model never sees, as in Claude Managed Agents; which actions require a human approval and can the firm add to that list; and can every agent action be exported as an audit trail, as Lexis+ Protégé Workrooms and Anthropic's Compliance API allow. The agent security guide explains the attack patterns behind each question.

The questionnaire and the red flags

Send this before the demo, not after the pilot.

  1. List every subprocessor that receives firm data, with region, retention period and training exclusion for each.
  2. Confirm in the contract that inputs, outputs and embeddings are not used to train or improve any model, for our tier.
  3. State retention for prompts, outputs, uploaded files, abuse-monitoring copies and flagged content, and the deletion process and certificate on termination.
  4. Confirm whether zero data retention is available for the model versions we will actually use.
  5. State the processing regions for storage and for inference, and whether customer-held encryption keys are supported.
  6. Name the model versions behind each feature, the version-pinning options and the notice period for changes.
  7. Describe the continuity plan if an upstream provider suspends or retires a model.
  8. Provide the SOC 2 Type II report, ISO 27001 certificate, latest penetration-test summary and breach-notification period.
  9. For agents: confirm per-user permissions, credential isolation, mandatory human approval points and exportable action logs.
  10. For each benchmark claim: name the benchmark, the date, the model version, who ran it and the scoring method.
  11. Provide the export format for matter data, prompts, custom workflows and audit logs on exit.
  12. State the liability cap and the carve-outs for confidentiality breach, data loss and IP infringement.
  13. State the pricing basis, minimum seats, consumption elements and renewal uplift.
  14. Provide the AI Act Article 50 position: are outputs marked or watermarked, and how.
  15. Provide the most recent third-party hallucination or accuracy evaluation of the product, not the model.
Red flagWhy it matters
Subprocessors "available on request" but never suppliedThe chain is where the data goes; a vendor that cannot list it has not mapped it
Training exclusion exists only as a settings toggleToggles reset, contracts do not; ABA Opinion 512 expects informed consent where a self-learning tool may expose client data
Retention "as long as necessary"Undefined retention is discoverable retention, as the 20 million ChatGPT logs showed
No usable export of workflows and logsHarvey customers have built 25,000+ agents; those are the firm's know-how and must leave with the firm
Benchmark score with no date or versionVendors pick favourable benchmarks at launch; a 2025 number says little about a 2026 model
Liability capped at twelve months' fees with no confidentiality carve-outAt Harvey's reported $1,000 to $1,200 per seat per month, the cap may be smaller than a single sanctions award
Consumer or "Frontier preview" terms for a production toolConsumer Copilot runs without a DPA and, as Spellbook notes, is "not suitable for confidential or legal work"

Contract terms and what to do before you sign

Most of the questionnaire converts directly into clauses. The four that vendors resist hardest are worth drafting yourself.

Data use. Supplier shall not, and shall procure that no subprocessor shall, use Customer Data, Outputs or derived embeddings to train, fine-tune, evaluate or improve any model, whether or not de-identified.
Retention and deletion. Supplier shall retain Customer Data only for the periods in Schedule B, which apply equally to abuse-monitoring copies and flagged content. On termination Supplier shall delete Customer Data within 30 days and certify deletion in writing.
Change notice. Supplier shall give 60 days' written notice before replacing or retiring any model, adding a subprocessor or changing a processing region, and Customer may terminate without penalty within that period.
Exit. On request at any time Supplier shall export Customer Data, custom workflows, prompt libraries and audit logs in a documented, machine-readable format at no additional charge.

On money, the pricing guide has the anchors: Harvey at an estimated $1,000 to $1,200 per seat per month with a 20-seat minimum, Legora from about $3,000 per user per year with a 10-seat minimum (both secondary sources), CoCounsel's published $104 to $639 per user per month, Spellbook at roughly $99 to $350. Minimums and consumption elements belong in the questionnaire because they decide whether a pilot can be small. Insurers care too: no major US professional-liability carrier had filed an explicit AI exclusion as of 5 May 2026, but application forms ask about governance and a signed questionnaire is evidence.

This week: pick the one tool most likely to be bought, send the fifteen questions, and put the subprocessor list and the retention answers into Schedule A of the firm's AI policy so that the approved-tools list and the diligence file are the same document. The AI Lab for Lawyers course tests these questions against the live tools in its risk and regulation module, which is the fastest way to see how much of a vendor's answer is boilerplate.

Frequently asked questions

What should a law firm ask an AI vendor about data training?

Ask three separate questions: are inputs, outputs or embeddings used to train or improve any model, including the vendor's own; is that exclusion contractual for the firm's tier or merely a default setting; and does the exclusion flow down to every subprocessor that handles the data. Then ask for the retention period at each hop, including abuse-monitoring copies and flagged content.

Is a SOC 2 report enough for legal AI vendor due diligence?

No. A SOC 2 Type II report describes the vendor's own controls over a past period. It says nothing about the model provider's retention, the subprocessor chain, whether inputs train models, hallucination rates or what happens when a model is retired. Treat it as a baseline for security hygiene, then run the data, provenance and exit questions separately.

What is zero data retention and which legal AI tools offer it?

Zero data retention (ZDR) means the provider does not store prompts or outputs after the response is returned. As of September 2026 Anthropic arranges ZDR per organisation on the API but excludes Fable and Mythos models and Team and Enterprise chat; OpenAI offers ZDR contractually to eligible API customers; legal platforms built on these models inherit whatever their provider allows.

Which red flags should stop a legal AI contract?

A refusal to name subprocessors and regions; training exclusions that exist only as a settings toggle; retention described as indefinite or undefined; no export of matter data and audit logs in a usable format; benchmark claims with no date, model version or scoring method; liability capped at fees with no carve-out for confidentiality breach; and no notice period for model or subprocessor changes.

Put this into practice

AI Lab for Lawyers: a live, hands-on course for practising lawyers

4 live sessions (8 hours), recordings included, taught by Dr. Niklas Schmidt of Wolf Theiss. Rated 4.7/5 from 381 ratings; alumni from Norton Rose Fulbright, Paul Hastings, Pillsbury and Taylor Wessing.

Reserve a seat · €200 Maven money-back guarantee · certificate of completion