Accuracy — live, not claimed
This page is generated directly from benchmark runs against a held-out set of confirmed-fake citations (from published AI-sanction orders) and confirmed-real citations (from published opinions). Benchmarks run in staging on an ongoing basis, not on a fixed nightly schedule; each layer below shows when it was last measured. We track False-Verify Rate (FVR), the rate at which we stamp VERIFIED on something that is actually fake or wrong, as the one metric that matters. The inverse error (flagging a real citation UNCONFIRMED) is allowed to run far higher; a lawyer double-checking a real citation is a Tuesday, a lawyer sanctioned while holding our green checkmark is not.
Methodology: 159 quote/citation pairs across 8 categories (exact quotes, truncated excerpts, OCR-style noise, ellipsis splicing, paraphrases, single-word meaning flips, quotes attached to the wrong citation, and fabricated quotes) against 20 real published opinions, run live against matchQuote() (deterministic, no LLM, CourtListener search API, $0 cost). Figures above are computed over the 144 pairs the live API resolved definitively; 15 pairs that stayed source-unavailable through retries are excluded rather than counted either way. A known gap found by this run: the fuzzy fallback scores only against CourtListener's own search snippet, not the full opinion, so a noised query can miss a genuinely real quote if the snippet returned isn't centered on the right passage -- flagged for follow-up, not fixed in this pass. Zero false-verifies across every adversarial pair. Full per-pair results and reproduction steps: citeguard-app/data/eval-run-quote-latest.json and citeguard-app/scripts/eval-quote-match-batch.ts.
Reproduction check (2026-07-17): we attempted to re-run this eval live against CourtListener from a build sandbox to refresh this number before publishing it further. CourtListener returned HTTP 429 ("request was throttled") on that sandbox's IP, so no fresh batch was run and the figures above are unchanged from the 2026-07-16 run. Noted here rather than silently retried, per the same rule that governs every other number on this page: report what happened, not what we'd prefer happened.
Layer 3 is eval-published (measured against our held-out set) and rolling out to production in stages.
Disputes
Target: <24h automated resolution, <72h if escalated.
Volume
360 citations in the published eval set, with a measured FVR published for 6 weeks (since 7/13/2026).
228 verification checks run to date. A single document may be checked more than once, so this counts checks run, not distinct documents.
Coverage map
Existence and quote checks run against CourtListener's database of published opinions: federal courts (all circuits + Supreme Court), and state courts to the extent CourtListener has ingested them (coverage varies by state and year, heaviest for appellate/supreme court opinions post-1950). Outside coverage today: statutes and regulations, unpublished BIA/EOIR immigration decisions, tribal courts, and most trial-court-level state filings. Citations in these categories are labeled UNCONFIRMED — OUTSIDE COVERAGE, never silently treated as NOT FOUND.
Methodology
Eval set: confirmed-fake citations drawn from published AI-hallucination sanction orders, paired with an equal count of confirmed-real citations from published opinions (the negative class matters: a checker that flags everything catches all fakes and is useless). FVR per layer = fake/wrong citations stamped VERIFIED at that layer ÷ total citations stamped VERIFIED at that layer. Full methodology and the eval-set generation scripts are open in our repository.