Vendor marketing pages routinely claim their AI tool is “built for legal accuracy” without publishing an actual measured error rate. The most rigorous independent measurement available — the Stanford RegLab/HAI benchmark study (“Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools,” peer-reviewed and published in the Journal of Empirical Legal Studies) — actually tested several major tools head to head, and the results are worth knowing before choosing one.
What the study actually measured
Researchers ran a consistent set of legal research queries against multiple AI systems and evaluated whether the responses were accurate, misgrounded (citing a real source that did not actually support the claim), or fully hallucinated (fabricated). This is a meaningfully more rigorous test than most vendor demos, which tend to showcase cherry-picked queries the tool handles well.
The headline numbers
The study found Lexis+ AI produced incorrect or misgrounded responses on roughly 17% of queries tested. Westlaw's AI-Assisted Research product hallucinated at roughly 33% — close to double the Lexis+ AI rate. General-purpose GPT-4, tested without any legal-specific grounding, hallucinated at roughly 43%, the highest rate among the systems compared.
What this ranking means, and what it doesn't
The ranking (Lexis+ AI lowest measured rate, Westlaw meaningfully higher, general-purpose GPT-4 highest) is a real, measured data point, not marketing. But “lowest measured rate among the tools tested” is not the same as “safe to use without verification” — 17% is still roughly one in six queries producing an unreliable answer, which is far too high a rate to skip a separate verification step regardless of which tool you use.
Why premium pricing does not track with accuracy
Both Lexis+ AI and Westlaw's AI-Assisted Research are premium, well-established, expensive legal research platforms, and both still showed hallucination rates measured in the double digits by an independent academic study. This is worth internalizing specifically because it is counterintuitive: price and brand reputation are poor proxies for measured accuracy in this specific category, and the study is direct evidence of that, not a hypothetical warning.
What about newer entrants like Harvey and CoCounsel?
Newer legal AI platforms marketed toward large firms (Harvey, Thomson Reuters' CoCounsel) have not, as of this writing, published the same kind of independently peer-reviewed, methodology-disclosed benchmark the Stanford study ran on Lexis+ AI and Westlaw. Absence of an independent study is not evidence either way about their accuracy — but it does mean claims about their hallucination rate should be treated as unverified marketing until an independent, disclosed methodology exists, the same standard that should apply to any vendor's own self-reported numbers.
How to apply this when choosing a tool
Ask any vendor directly: has your hallucination or error rate been measured by an independent party, using a disclosed methodology, and is the number current? If the answer is no, treat the tool's accuracy as unknown rather than assuming “legal-specific” branding implies a low rate — the Stanford study's own findings on Westlaw and Lexis+ AI, both established legal-specific platforms, demonstrate that legal branding alone does not predict a low hallucination rate.
The verification layer matters regardless of which drafting tool you pick
Because even the best-measured tool in the Stanford study still hallucinated on roughly one in six queries, tool selection for drafting or research should be treated as a separate decision from verification. Whichever drafting tool you use, a dedicated, deterministic verification step before filing remains necessary — this is the entire premise behind purpose-built verification products like the one we built, Citation Safe, which checks citations against CourtListener directly rather than asking any generative model to self-assess.
How the study's methodology avoided the most common benchmark pitfall
A common criticism of AI benchmark studies generally is that vendors can optimize their systems specifically to perform well on whatever test questions become public, inflating scores in a way that does not reflect real-world performance. The Stanford RegLab/HAI researchers addressed this by using a broad, realistic set of legal research queries designed to reflect actual practitioner use, rather than a narrow set of questions any vendor could specifically tune against, which is part of why the study has been treated as unusually credible within the legal research community.
What "misgrounded" means and why it matters as its own category
The study distinguished between fully fabricated citations and “misgrounded” responses — answers that cite a real, existing source but mischaracterize what that source actually says or holds. This distinction matters practically because a misgrounded citation passes a simple existence check cleanly, meaning tools or manual processes that only verify existence will systematically miss this entire category of error, which the study found to be a meaningful share of the overall hallucination rate across every tool tested.
How to interpret hallucination-rate data as it continues to evolve
AI legal research products are updated frequently, and a hallucination rate measured at one point in time may not reflect a tool's current performance months or years later. Treat any specific percentage figure, including the ones cited in this post, as a snapshot rather than a permanent ranking, and look for the most recent independent study available before making a significant procurement decision, rather than relying on a single data point indefinitely.
How to request comparable data directly from a vendor
When asking a vendor about their tool's accuracy, ask specifically whether any independent third party has tested it, using what methodology, on what date, and whether the results are publicly available for you to review yourself. A vendor that can answer all four parts of that question specifically and verifiably is offering meaningfully more than one that responds only with a general accuracy percentage without methodology or independent verification behind it.
Why hallucination rate alone doesn't capture the full risk picture
A tool's overall hallucination rate is a useful single number, but it can obscure meaningful variation across query types — a tool might perform well on straightforward, well-settled legal questions and considerably worse on novel or narrow ones. Ask, where possible, whether accuracy data is broken down by query complexity or practice area, since your own actual use case may not match the average query profile a general benchmark study tested.
A common question
Does a lower hallucination rate in a benchmark study mean I can skip manual verification for that tool?
No. Even the lowest rate the Stanford study measured (roughly 17% for Lexis+ AI) is far too high to treat as safe without verification — that is a documented, non-trivial failure rate on ordinary legal research queries, not an edge case. The practical implication of the study is which tool to prefer if you must choose, not a signal that verification becomes optional at any measured rate above zero.