What was hallucinated

False Quotes: Doctrinal Work | Applicant relied on a Copilot-generated quote from the Explanatory Memorandum to the Taxation Laws Amendment (Superannuation) Bill 1992 stating a 15% rebate applies to all disability pensions (taxed and untaxed). The Tribunal checked the EM, found no such paragraph (no para 2.20) and no similar language, concluding it was an AI hallucination.

Details

"The Applicant engaged the Copilot [Microsoft’s Artificial Intelligence product] in a range of probing questions pertaining to superannuation and taxation matters, upon which in part, it returned the following responses:The Explanatory Memorandum to the Taxation Laws Amendment (Superannuation) Bill 1992, which introduced the new regime taxing superannuation benefits, states in paragraph 2.20 that “the Bill will provide a tax rebate of 15 per cent for disability superannuation pensions. This will apply to all disability pensions, irrespective of whether they are paid from a taxed or an untaxed source. The rebate recognises that disability pensions are paid as compensation for the loss of earning capacity and are not merely a form of retirement income.I have examined the Explanatory Memorandum to the Taxation Laws Amendment (Superannuation) Bill 1992. I was unable to locate any paragraph in that document in the same or similar terms to the paragraph generated by Copilot. It did not contain a paragraph 2.20.It has been noted by others that AI bots are prone to hallucinations.[35] That appears to be what has happened here. It is my assessment that submitting unverified material generated by AI, is not consistent with a party’s duty to use their best endeavours to assist the Tribunal to achieve its statutory objectives. To expect the Tribunal to read and consider material which a party does not know is authentic impedes the Tribunal’s attempts to provide a mechanism of review that ensures that applications are resolved as quickly and with as little expense as a proper consideration of the issues permits.Nothing in the remainder of the applicant’s submissions altered my view that the untaxed element of the benefit should be taxed under Subdivision 301-B."

Sanction teardown · Administrative Review Tribunal of Australia, Australia · 2025-02-05

QWYN and Commissioner of Taxation

What happened

In Administrative Review Tribunal of Australia, Australia, a filing relied on Copilot to help draft legal argument. The court identified the following problems with the citations in that filing:

False Quotes (Doctrinal Work)

Applicant relied on a Copilot-generated quote from the Explanatory Memorandum to the Taxation Laws Amendment (Superannuation) Bill 1992 stating a 15% rebate applies to all disability pensions (taxed and untaxed). The Tribunal checked the EM, found no such paragraph (no para 2.20) and no similar language, concluding it was an AI hallucination.

Which AI tool

Copilot. Note: Charlotin's public database records tool attribution only where a court order, brief, or reporting on the matter states it explicitly; "unidentified" or "implied" means the record indicates AI use but does not name a specific product — we do not guess.

Outcome

The Tribunal affirmed the decision under review, rejecting the applicant's submissions based on the AI-generated content.

Additional detail

"The Applicant engaged the Copilot [Microsoft’s Artificial Intelligence product] in a range of probing questions pertaining to superannuation and taxation matters, upon which in part, it returned the following responses:The Explanatory Memorandum to the Taxation Laws Amendment (Superannuation) Bill 1992, which introduced the new regime taxing superannuation benefits, states in paragraph 2.20 that “the Bill will provide a tax rebate of 15 per cent for disability superannuation pensions. This will apply to all disability pensions, irrespective of whether they are paid from a taxed or an untaxed source. The rebate recognises that disability pensions are paid as compensation for the loss of earning capacity and are not merely a form of retirement income.I have examined the Explanatory Memorandum to the Taxation Laws Amendment (Superannuation) Bill 1992. I was unable to locate any paragraph in that document in the same or similar terms to the paragraph generated by Copilot. It did not contain a paragraph 2.20.It has been noted by others that AI bots are prone to hallucinations.[35] That appears to be what has happened here. It is my assessment that submitting unverified material generated by AI, is not consistent with a party’s duty to use their best endeavours to assist the Tribunal to achieve its statutory objectives. To expect the Tribunal to read and consider material which a party does not know is authentic impedes the Tribunal’s attempts to provide a mechanism of review that ensures that applications are resolved as quickly and with as little expense as a proper consideration of the issues permits.Nothing in the remainder of the applicant’s submissions altered my view that the untaxed element of the benefit should be taxed under Subdivision 301-B."

How Citation Safe would have caught this

Citation Safe runs three deterministic layers before a brief is filed: (1) does the citation exist against CourtListener's database of published opinions, (2) if quoted, does that exact language appear in the source, (3) does the cited case actually support the proposition it is cited for. Fabricated case citations fail Layer 1. Fabricated or misattributed quotations fail Layer 2 even when the underlying case is real. Misrepresented holdings — a real case cited for a proposition it does not support — are the target of Layer 3. None of these checks involve asking another language model whether the citation looks right; they are lookups and text-matches against the actual source, which is why a hallucinated citation has to survive a direct lookup against the authoritative source — not another model's opinion — to earn a VERIFIED stamp; our measured false-verify rate is published live at /quality.

Check a brief before you file it → · See our live false-verify rate

Source: https://www.damiencharlotin.com/documents/542/QWYN_and_Commissioner_of_Taxation_Australia_5_February_2025.pdf, via Damien Charlotin's public AI Hallucination Cases Database (CC0).