The Argus Standard for AI-Safe Legal Citation Verification

1. Introduction: a crisis with a public scoreboard

As of this writing, legal researcher Damien Charlotin maintains a public, continuously updated database of court cases in which a filing was found to contain AI-hallucinated legal citations — fabricated case names, invented quotations, or citations that misrepresent what a real case actually held. That database has grown past 1,700 documented cases, spanning federal and state courts, criminal and civil matters, solo practitioners and large firms, across nearly every US jurisdiction. It is not a hypothetical risk. It is a running tally, updated most weeks, of lawyers who filed something a court later struck, sanctioned, or referred to a disciplinary body because an AI tool invented a case that does not exist, or invented a quotation a real case never contained.

This is the founding fact behind the Argus Standard. Every other section of this document follows from a single observation: courts do not care whether a hallucination came from a general-purpose chatbot or a purpose-built legal AI product. Rule 11 and its state equivalents impose the same duty of reasonable inquiry regardless of tooling. The lawyer signs the filing. The lawyer is sanctioned. The tool is, at most, a footnote in the order. That asymmetry — the person who benefits from AI-assisted drafting speed bears 100% of the liability for AI-assisted drafting errors — is the actual problem this standard is built to address, and no amount of "our AI is smarter now" marketing changes the math, because the math is about verification, not drafting quality.

1.1 Who actually bears this risk

The seminal case that put AI hallucination on the legal profession’s radar, Mata v. Avianca, Inc., No. 22-cv-1461 (S.D.N.Y. 2023), involved a solo-adjacent practice using ChatGPT to research aviation-law precedent, citing several cases that turned out not to exist, and being sanctioned once opposing counsel and the court could not locate them. That case is now more than two years old and is treated, incorrectly, as a historical curiosity rather than a live pattern. The Charlotin database shows otherwise: new entries continue to appear on a near-weekly cadence, across criminal defense, family law, bankruptcy, employment, immigration, and civil litigation, in courts ranging from small-claims-adjacent state courts to federal district courts and circuit courts of appeal.

The pattern that recurs across the database is not "reckless lawyers." It is time-pressured lawyers — often solo practitioners or small firms without a research department, filing under a real deadline, using a tool that produces confident, well-formatted prose faster than any associate could draft it by hand. The tool does not announce which sentences it invented. The lawyer, reading their own filing under deadline pressure, has no structural reason to distrust a citation that is formatted exactly like every real citation around it. This is precisely the population the Argus Standard is written for: not enterprise firms with research departments and KeyCite subscriptions, but the solo and small-firm lawyer who adopted AI drafting tools because they had to, and who needs a verification step that is fast enough to actually run at 11pm the night before a filing deadline, not a workflow that requires the very research department they do not have.

2. Why legal AI tools fail at citation integrity

Large language models, including the ones embedded in premium legal-research products, generate text by predicting statistically likely continuations of a prompt. A citation to Smith v. Jones, 512 F.3d 233 (9th Cir. 2008), is, to the model, a plausible-looking token sequence — not a database lookup. Retrieval- augmented systems reduce this risk by grounding generation in a real corpus, but they do not eliminate it, because the failure mode shifts rather than disappears: instead of inventing a case outright, the model can retrieve a real case and then generate a quotation, holding, or procedural posture that isn’t actually in it, or cite a real case for a proposition it does not support. Both failure modes produce output that reads as confident and well-formatted, which is precisely what makes them dangerous to a rushed reader.

Independent, peer-reviewed research bears this out. The most-cited study in this space is the Stanford RegLab/Stanford HAI study (Magesh, Surani, Dahl, et al., 2024, “Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools”), which tested commercial legal AI research products against real attorney queries and measured hallucination rates directly. The study found meaningfully high hallucination rates even among the market’s leading, purpose-built legal research products — figures we summarize, with the source cited, on our public scorecard page. AI ethicist Reid Blackman has separately written about these findings as a case study in the gap between AI capability marketing and AI reliability in high-stakes professional contexts. We did not conduct this research; we cite it because a standard that asks the industry to publish error rates has an obligation to publish the one number that already exists in the peer-reviewed literature, not just our own.

The deeper structural problem is that "hallucination rate" is usually reported, when it is reported at all, as a single aggregate number from a vendor’s internal testing, with no disclosed methodology, no held-out negative class, and no distinction between failure types. A tool that never flags anything and a tool that flags everything can both report favorable-sounding statistics depending on what is measured. The Argus Standard exists because "trust us" is not a verification methodology, and because a lawyer facing a malpractice claim or a bar complaint needs a specific, falsifiable answer to "how do you know this citation is real," not a vendor confidence score.

3. The three-layer verification methodology

The Argus Standard defines citation verification as three independent, sequential layers, each answering a distinct question a lawyer actually needs answered before filing. A citation is not "verified" in any meaningful sense until all three layers that apply to it have run.

Layer 1 — Existence. Does the cited case actually exist? This is checked against a primary source of published judicial opinions — in our implementation, CourtListener’s public database, which indexes federal courts (all circuits and the Supreme Court) and state courts to the extent CourtListener itself has ingested them. This is a lookup, not an inference: the reporter, volume, and page are matched against the actual record, with a case-name fuzzy cross-check as a secondary signal. A citation that does not exist fails here, full stop, regardless of how plausible it reads.

Layer 2 — Quote match. If the filing quotes the case, does that exact language appear in the source opinion? This catches the second, subtler failure mode: a real case, correctly cited, with fabricated or materially altered quoted language attributed to it. This is a textual match against the actual opinion text, not a semantic-similarity score, because "close enough" is exactly the standard a hallucinated quote is designed to satisfy.

Layer 3 — Proposition support. Does the cited case actually support the legal proposition the filing cites it for? This is the hardest layer and the one most often skipped entirely by existence-only checkers. A case can be real, and a quotation from it can be accurate, while the case is still cited for a holding it does not support, a posture it was not decided in, or a rule that a later decision overturned. This layer necessarily involves more judgment than Layers 1 and 2, and the Standard requires that any tool implementing it disclose the layer’s comparatively higher error tolerance rather than presenting all three layers as equally certain.

Critically, the Standard requires that a failure at a lower layer can never be overridden by a pass at a higher layer. A citation that fails existence is not "upgraded" to verified because an LLM guesses that a similar-sounding case probably exists and probably supports the proposition. Layer ordering is a integrity guarantee, not a UX convenience.

3.1 A worked illustration

Consider a brief that cites “Doe v. Roe, 512 F.3d 233 (9th Cir. 2008)” for the proposition that a particular pleading standard applies, and quotes the case as holding that “a plaintiff must plead specific facts establishing the defendant’s state of mind.” Layer 1 asks the narrowest possible question first: does a Ninth Circuit opinion exist at 512 F.3d 233, decided in 2008, and does its case name reasonably match “Doe v. Roe”? If no opinion exists at that reporter citation, the citation fails here and the process stops — there is no reason to check whether a nonexistent opinion contains a quotation or supports a proposition. This is deliberate: layered verification is fastest, and safest, when it fails closed at the earliest possible check rather than trying to charitably interpret a fabricated citation.

If the case does exist, Layer 2 asks a narrower question than "is this a fair summary": does the exact quoted language, or something acceptably close to it, actually appear in the opinion’s text? A brief that paraphrases and marks the result as a paraphrase is not a Layer 2 problem; a brief that puts language in quotation marks that does not appear in the source is, regardless of how accurately it captures the case’s general thrust. This distinction matters because several of the sanction cases in the Charlotin database involve exactly this failure mode on a real, correctly-existing case — quotation marks around invented language, which reads to a judge as a much more serious misrepresentation than an unmarked paraphrase would.

Only after a citation clears both of those checks does Layer 3 ask the genuinely hard question: setting aside whether the case exists and whether the quotation is accurate, does the case actually stand for the proposition it is cited for? A real, accurately quoted case can still be cited for a rule from a footnote, a dissent, a since-vacated portion of the opinion, or a fact pattern distinguishable enough that the "holding" as characterized in the brief is not really there. This is the layer where the Standard requires the most candor about error tolerance, because it is the layer closest to legal judgment rather than mechanical lookup.

4. Deterministic vs. LLM-based verification: why the distinction matters

A recurring and reasonable question is: why not just use a better, more careful LLM to check citations, instead of building deterministic lookups? The answer is that verification and generation are different problems with different acceptable error profiles, and using the same class of tool for both re-introduces the exact risk the check exists to catch.

An LLM asked "does this citation exist and say what is claimed" is still, mechanically, predicting a plausible answer conditioned on a prompt. It can be right often. It can also confidently hallucinate a "yes, this citation is accurate" verdict for a citation that does not exist, for exactly the same reason it hallucinated the citation in the first place: pattern plausibility, not ground truth. Using an LLM to grade an LLM’s citations is not independent verification; it is asking the fox to notarize the henhouse inventory. This is not a hypothetical concern — it is the direct implication of the Stanford RegLab findings discussed above, where products with LLM-mediated verification steps still exhibited double-digit hallucination rates.

The Argus Standard therefore requires that Layer 1 and Layer 2 be strictly deterministic: a citation lookup against a primary-source database, and an exact or near-exact text match against the retrieved source document, respectively. Neither step asks a language model whether something is true; both steps look it up. Layer 3 is the one layer where some inference is unavoidable — "does this case support this proposition" is not a string-match problem — and the Standard requires that any tool be explicit about which layer(s) involve inference versus lookup, and report false-verify rates separately per layer rather than blending them into one reassuring aggregate number. A tool that reports a single 99% "accuracy" figure without disclosing that the number blends a near-100%-deterministic existence check with a much harder proposition-support judgment is not meeting this standard, even if every individual number in the blend is honestly measured.

5. Certification process

A tool meets the Argus Standard v1.0 when it satisfies all of the following, publicly and verifiably:

1. Three-layer methodology disclosed. The tool publishes, in plain language, which of existence, quote-match, and proposition-support it checks, and which of those are deterministic lookups versus model-assisted judgments.
2. False-Verify Rate published, per layer, with a dated eval run. Not an aggregate accuracy score — the specific rate at which the tool stamped something VERIFIED when it was in fact fake, wrong, or unsupported. This number must come from a held-out eval set containing both confirmed-fake and confirmed-real citations; a checker tested only against real citations cannot report a meaningful false-verify rate, because it has never been asked to catch anything.
3. Eval-set methodology disclosed. Where the confirmed-fake examples come from (published sanction orders are the strongest source, since a court has already adjudicated the citation as fabricated) and how the confirmed-real negative class was constructed, so the number cannot be inflated by an easy or unrepresentative test set.
4. Coverage gaps disclosed, not hidden. Any citation type or jurisdiction outside the tool’s verification coverage (unpublished decisions, certain state trial courts, statutes and regulations, etc.) must be labeled as out-of-coverage/unconfirmed, never silently passed through as verified or silently dropped from the reported denominator.
5. Refund or remediation contract for a wrong VERIFIED. If the tool stamps a citation VERIFIED and that citation later turns out to be fake, misquoted, or unsupported, the tool commits, in writing, to a specific customer remedy — a refund, at minimum. This is the accountability mechanism that turns a published error rate from a marketing statistic into an actual warranty.

Certification under the Argus Standard is currently self-attested and audited by publication: a tool claims compliance by publishing the artifacts above at a stable public URL, and the claim is falsifiable by anyone who checks that page against the tool’s actual behavior. We expect this to evolve toward independent third-party audit as the standard gains adoption; v1.0 optimizes for getting a real, checkable baseline into the market quickly rather than waiting for a governance body that does not yet exist.

6. Citation Safe’s refund contract

Citation Safe implements the Argus Standard’s certification requirements directly: our live, per-layer False-Verify Rate is published and updated at /scorecard(weekly snapshot) and /quality (live detail, methodology, dispute stats, and coverage map). If we stamp a citation VERIFIED and it is later shown to be fabricated, misquoted, or unsupported by the cited case, the affected customer is entitled to a refund for the verification in question, disputed through the process documented on /quality. We would rather have a public, embarrassing dispute count than a private, comfortable one.

8. Limitations and open problems

An honest standard discloses what it does not solve. First, Layer 3 (proposition support) inherently involves more judgment than a string match, and any tool implementing it — including ours — will have a materially higher error tolerance at that layer than at Layers 1 and 2. The Standard requires this be disclosed separately per layer specifically so a reader cannot mistake a blended number for a uniform guarantee.

Second, no citation-existence database is complete. CourtListener, the primary source we use for Layer 1 and Layer 2, has excellent coverage of federal courts and increasingly good coverage of state appellate and supreme courts, but coverage of state trial-court filings, unpublished immigration board decisions, tribal courts, and statutes/regulations is partial or absent today. A verification tool built on this kind of primary source must label citations in these categories as unconfirmed/outside-coverage rather than either silently passing them or silently failing them — both of those alternatives are worse than an honest "we don’t know," and the Standard treats mislabeling a coverage gap as a verified/not-verified result as a certification failure in its own right.

Third, this Standard does not currently address "good law" / treatment checking — whether a real, correctly quoted, correctly-supporting case has since been overruled, distinguished, or superseded. That is the function Shepard’s and KeyCite serve inside Lexis and Westlaw, and it is a genuinely different check from citation fabrication detection. A tool can be fully Argus Standard-certified on existence, quote, and proposition support, and a lawyer can still cite good-faith, non-hallucinated, but legally dead precedent. We consider this a real gap, not a marketing footnote, and expect a future version of this Standard to address it directly rather than blur the line between "not hallucinated" and "still good law."

9. Adoption path

We expect adoption of this Standard, if it happens at all, to follow the same path most voluntary professional-tooling standards follow: a handful of vendors with something to prove adopt it early because disclosure is a competitive asset when a market has none; a slower middle group adopts once a bar association, malpractice insurer, or court itself starts asking vendors for a false-verify rate as part of due diligence; and a final group never adopts voluntarily and instead has disclosure requirements imposed on them, if at all, by regulation. We are not positioned to force the second or third path. What we can do, and are doing with this document, is make the first path exist: publish our own numbers, in public, on a page anyone can check against our actual behavior, and invite direct comparison. If a competitor publishes a lower false-verify rate than ours under an equivalent methodology, that is a better outcome for lawyers than the status quo, even though it is a worse outcome for us commercially — and that trade is the actual point of calling this a standard instead of a feature.

7. A standard, not a moat

We are publishing this standard, rather than keeping our methodology proprietary, because the actual goal is fewer sanctioned lawyers, not a defensible trade secret. Any competitor is welcome to adopt every requirement in Section 5 verbatim. We believe the honest bet is that most will not, at least not quickly — publishing a real false-verify rate is a commitment with downside (it can look bad) that a vendor only makes if the alternative, staying silent while incidents accumulate in a public court-records database, is worse. We think it is worse. This document is our bet on that being true, made in public, with our own numbers first.