Citation Safe vs. Using ChatGPT Directly

In June 2023, a New York lawyer named Steven Schwartz asked ChatGPT to find case law supporting a client's claim against Avianca airlines. ChatGPT produced six confident, detailed citations — complete with fabricated quotes and invented case numbers — none of which existed. Judge P. Kevin Castel sanctioned Schwartz, his co-counsel, and their firm $5,000. That case is now the canonical example of the failure mode this entire product category exists to prevent, and the underlying behavior — a general-purpose language model generating plausible, confident, entirely fabricated legal citations — has not gone away just because the case is now well known.

A public database maintained by legal researcher Damien Charlotin tracks roughly 1,490 court decisions worldwide (more than 1,000 in the US) as of May 2026 where a party relied on AI-hallucinated material and a court responded. Within that tracked dataset, ChatGPT appears in more sanctioned matters than any other single tool — reported at roughly 50 cases, well above other AI products. Penalties have escalated well past the original $5,000: a federal appellate court has issued a $15,000 sanction against a single attorney, and in May 2026 a federal judge in Oregon fined two lawyers a combined $110,000 for 23 fabricated citations and eight invented quotations in one filing — the largest AI hallucination penalty in US legal history to date.

It is not that lawyers using ChatGPT are unaware of the risk — Mata v. Avianca is now a required-reading cautionary tale in legal ethics CLE. The problem is structural: a general-purpose language model is trained to produce fluent, plausible-sounding text, and a fabricated case citation is, by construction, exactly as fluent and plausible as a real one. There is no visual or stylistic tell that distinguishes a hallucinated citation from a genuine one inside ChatGPT's output — the only way to know the difference is to independently check it against a primary source, which is precisely the step that gets skipped under deadline pressure.

A 2024 Stanford RegLab study ("Large Legal Fictions") tested 2023-era general-purpose models on over 800,000 verifiable legal questions and found GPT-4 hallucinating 58% of the time, GPT-3.5 at 69%, and Llama 2 at 88%. A follow-up 2025 Stanford study testing newer tooling found GPT-4 at 43% when evaluated alongside retrieval-grounded commercial legal research tools (which scored better — 17% for Lexis+ AI, 33% for Westlaw AI-Assisted Research — but still meaningfully above zero). Whichever specific figure you cite, every measured rate for direct, ungrounded use of a general-purpose model sits well above the threshold where independent verification is optional.

As of early 2026, at least 25 federal district courts have adopted standing orders requiring attorneys to certify whether AI was used in preparing a filing and to confirm a human reviewed all content before submission. This is a meaningful shift: courts are no longer treating AI-assisted drafting as a novelty to be handled case by case, but as a standard risk requiring a standard disclosure and verification practice. An independent, deterministic verification step is the most direct way to satisfy that certification honestly, rather than simply asserting review happened.

ChatGPT is a genuinely capable drafting and brainstorming tool, and plenty of lawyers use it responsibly as part of a broader workflow that includes independent verification. The point of this page isn't "never use ChatGPT" — it's that ChatGPT was never built, and OpenAI has never marketed it, as a citation verification tool, and treating its output as pre-verified is the specific mistake that has produced the largest, costliest sanctions on record. Citation Safe's deterministic existence and quote-match layers are built to sit downstream of exactly this kind of drafting, checking whatever came out the other end before it reaches a filing.

ChatGPT is not a citation verifier and has never claimed to be one — and it has appeared in more AI-hallucination sanctions cases than any other tool tracked. Citation Safe is the independent, deterministic check that belongs downstream of any ChatGPT-assisted draft before it reaches a filing.