The research shelf

Legal citation hallucination is not monotonic: ~25% for GPT-3.5, a low of 1.23% for the August 2024 GPT-4o, back up to 6.57% for GPT-5.1 (p=0.001).

Eight generations of default ChatGPT, the same 92 legal-drafting prompts, 8,000+ citations verified against CourtListener and Westlaw. The same paper counts 1,000+ real court filings containing hallucinated citations. Scope: legal drafting in one product line.

3–13% of citation URLs produced by current commercial models and Deep Research agents are hallucinated; 5–18% don't resolve at all.

221,000+ URLs across ten models/agents. Hallucinated = no Wayback Machine record. The agent with the highest hallucination rate (13.3%) also had a top quality score on the same benchmark.

Commercial legal AI tools marketed as hallucination-free still hallucinated in 17–33% of tested queries.

Lexis+ AI, Westlaw AI-Assisted Research, Ask Practical Law AI. Retrieval-augmentation reduced hallucination; it did not eliminate it.

1,769 legal cases involving AI-hallucinated material identified in one hand-maintained database (as of July 17, 2026).

A live counter — cite it with an as-of date. Breakdown at last check: USA 1,219, Canada 194, Australia 96.

55% of GPT-3.5's bibliographic citations were fabricated, 18% for GPT-4; 43% and 24% of the genuine ones carried substantive errors.

636 citations across 42 topics. The models are dated; useful as the historical baseline the 2026 studies extend.

Up to 22% of computer-science papers show signs of LLM-modified text, against roughly 9% for mathematics and Nature-portfolio journals.

1,121,912 papers, 2020–2024. Measures LLM-modified language, not fabrication — don't recruit it into a hallucination argument. Supersedes the widely-quoted 17.5% figure from the earlier preprint.

1 in 277 PubMed-indexed papers in 2026 shows fabricated references; fabricated references were found in 2,800+ biomedical articles.

Reported tier: figures from Retraction Watch's analysis as covered by STAT and Forbes; re-verify against the primary analysis before quoting elsewhere.

Across 2.5 million biomedical papers, the share carrying at least one fabricated citation rose from 1 in 2,828 (2023) to 1 in 458 (2025) to 1 in 277 (first seven weeks of 2026); 4,046 fabrications were confirmed across 2,810 papers.

Published as a letter. 125.6M references extracted, 97.1M with verifiable identifiers, checked against PubMed, Crossref, OpenAlex and Google Scholar. Over 98% of the affected papers had seen no publisher action at the time of the audit. The letter publishes a quarterly-rate figure spanning Jan 2023 to Feb 2026 and names 51.3 per 10,000 for Q4 2025 in its text. The cited letter carries a correction notice dated 2026-07-16. The method is AI-assisted and validated on 500 records; Cochrane’s head of editorial policy called the published methods detail insufficient, and a Northwestern integrity researcher called the analysis simplistic and the tip of the iceberg, arguing the larger problem is citations that are inaccurate rather than wholly invented.

o3 hallucinated on 33% of PersonQA questions; o4-mini on 48%.

PersonQA is an adversarial benchmark of facts about people, not typical usage.

On production-traffic factuality grading, GPT-4o ran a 12.9% hallucination rate and gpt-5-main 9.6%; OpenAI also reports that o3 would sometimes claim to have completed tasks it hadn't.

OpenAI's own LLM-graded methodology (75% human-grader agreement), self-reported.

On grounded summarization, current models range from ~3% to 23.3% hallucination; reasoning-heavy models score worse than lighter ones, the inverse of the open-QA pattern.

Summarization-faithfulness only; not comparable with PersonQA/SimpleQA numbers. Benchmark choice changes the story — check which one a headline figure comes from.

Hallucination is argued to be a statistical consequence of training and evaluation incentives; the paper also notes all reported models below 30% accuracy on Humanity's Last Exam and calibration error rates above 70%.

Frequently mis-cited: this paper does not contain the o3/o4-mini PersonQA figures — those live in the system cards above.

Across 1,600+ annotated multi-agent traces, roughly 23.5% of failures were verification failures, and adding an explicit verification step measurably improved outcomes.

No/incomplete verification 8.2%, incorrect verification 9.1%, premature termination 6.2%.

Agents falsely claimed completion in 45–48% of tau2-bench failures and 75.8% of AppWorld self-assessed coding trajectories; no LLM judge exceeded AUROC 0.65 at catching it.

Denominator is failed runs, not all runs.

29.6% of 'plausible' SWE-bench patches behave differently than the ground-truth fix; reported resolution rates were inflated by 6.2 points.

Passing the test suite is not the same as satisfying the requirement.

In a simulated software company with realistic professional tasks, the best agent completed 30% autonomously.

Frontier models' 50%-reliability horizon for autonomous work was about 50 minutes of human-equivalent task length, doubling roughly every 7 months.

Models become more likely to err when their own context contains prior mistakes — errors compound by self-conditioning, not just context length.

Anthropic's own system card documents Claude Opus 4 hard-coding an expected test value to force a failing case green, then reporting the task as successfully implemented.

The transcript is printed in the card, code and false summary side by side. A vendor documenting its own model faking a pass.

OpenAI's system card names the expected frontier failure class: tests deleted, data invented, tasks pretended solved, with deceptive actions in roughly 4% of gpt-5-thinking trajectories and 8% for o3.

The same card's deception section records o3 making false claims about actions it had taken and saying it had completed tasks it hadn't (section 3.8, wording verified against the PDF).

Judge position bias, quantified at the origin: Claude-v1 kept the same verdict only 23.8% of the time when answer order was swapped; GPT-4 agreed with humans ~85%, about the human-human rate.

2023 models; see the CALM entry below for current judges.

Modern judges, GPT-4o and Claude-3.5 included, fall below 50% robustness to answer-order swaps once three or four candidates are compared.

Twelve bias types quantified per model as robustness rates.

LLM-generated evaluators inherit the problems of the LLMs they evaluate, and still require human validation.

The state-of-the-art LLM judge for summarization quality reached a Spearman correlation of 0.514 with humans, described as outperforming all previous methods by a large margin.

A widely-used judge benchmark needed a post-hoc statistical correction for verbosity bias years after launch; length-controlling raised its arena correlation from 0.94 to 0.98.

ChatGPT and Claude 2 distinguished factual from hallucinated summaries at only 53.8–58.5% accuracy.

Dated models; the point is structural — a judge that can barely detect hallucination is a weak enforcement layer.

Eval frameworks ship deterministic validators for JSON, HTML, SQL and XML; no Markdown-structure equivalent is listed — Markdown output falls through to regex or a model-graded rubric.

A mutable catalogue — this describes it as checked on the date above.

In a randomized trial, experienced open-source developers were 19% slower with AI tools, while estimating they were 20% faster.

16 developers, 246 real tasks in familiar codebases; under 44% of suggestions accepted.

Output quality is the #1 barrier to putting agents in production (32%) and to scaling them (33%), ahead of latency at 20%.

Vendor-run survey of its own ecosystem.

30% of developers trust AI-generated code a little or not at all; the report's follow-up analysis names the 'verification tax': time saved writing is re-spent auditing.

The EU AI Act requires high-risk systems to log events automatically (Art. 12), remain overseeable by humans able to detect anomalies (Art. 14), retain logs at least six months (Art. 19), and ship 'concise, complete, correct and clear' instructions (Art. 13).

Inadequate software-testing infrastructure was estimated to cost the US $22.2–59.5 billion per year.

A 24-year-old software-bug study, often quoted without its year or scope. Cite it as history, not as a current documents figure.

90% of developers regularly use at least one AI tool at work; 74% use a specialized coding tool.

The freshest developer-survey figure available. Stack Overflow's 2026 survey opened in June — 'for human developers only' — with results not yet published, so the 2025 numbers remain the newest SO data.

275 million commits a week on GitHub, against one billion commits in all of 2025.

The verifiable substitute for the viral 4M-to-17M agent-PR claim, which traces only to a paywalled report and stays uncited here.

Reviewer defect-detection drops sharply past 200 to 400 lines per review session.

About 2,500 reviews over 3.2 million lines of code. A per-session attention limit, not an organizational capacity law.

In one study, 15 of 18 writers assisted by ChatGPT could not quote their own essay minutes after writing it.

Small-sample preprint (n=18 per condition); a methodological comment raises sample-size concerns. Formerly on the landing deck; the deck now argues review capacity rather than reader memory.

Anthropic's own developer docs instruct: hallucination-reduction techniques do not eliminate hallucinations — always validate critical information.

The vendor prescribing the practice; the landing page cites it as its quickstart support line.

People find only 40–90% of errors when working alone: readers caught 81% of nonsense-word errors but just 66% of real-word errors, and detection of nonsense-word errors falls to 47–58% in difficult material.

Laboratory experiments on general prose. The 47–58% figure is Riefer (1991) for NONWORD errors in difficult material; Riefer's word-error range is 42–62%. The 87% professional-proofreader figure on the same page rests on three editions of the author's own textbook and is deliberately not used.

Across 22,000 developers, the median time a pull request spends in review rose 441% year over year — against 91% in the prior year's dataset — 31% more pull requests merged with no review at all, and 25% of pull requests are now reviewed by an AI agent.

Vendor telemetry from a company selling engineering analytics. The year-over-year comparison runs on the same pipeline, so it is like-for-like, but the methodology note is not public.

In one year on GitHub, pull requests created rose 20.4% and commits 25%, while comments on issues and pull requests rose 0.35% and comments on commits fell 27%.

GitHub labels these observational signals rather than causal claims. Comments proxy for review attention rather than being it, and Copilot code review shipped in April 2025, so part of the flat line may be review that moved to a machine.

curl ended its six-year bug bounty after AI-fabricated vulnerability reports — some citing functions that don't exist in curl — pushed the valid-report rate to roughly 1 in 20–30.

Django's security documentation now instructs AI tools directly: invented code, fictitious APIs, and made-up citations result in automatic rejection.

English Wikipedia adopted speedy-deletion criterion G15 for pages clearly entirely LLM-generated without human review; WikiProject AI Cleanup has run since 2023.

A tribunal held Air Canada liable for its chatbot's fabricated bereavement-fare policy: CAD $812.02, and a rejection of the argument that the chatbot was 'a separate legal entity'.

The first ChatGPT-citation sanction: $5,000 against the lawyers who filed six fabricated cases.

A $31,100 sanction after an AI-drafted brief outline carried 9 wrong or nonexistent citations out of 27, filed unchecked by two firms.

Deloitte Australia repaid the final instalment of an A$440,000 government contract after its report was found to contain a fabricated court quote and citations to nonexistent papers.

A US federal health report cited studies that don't exist; named researchers denied authorship, and some reference URLs still carried 'oaicite' markers.

KPMG pulled its own report after forensic review found only 5 of 45 citations pointed to real, uncorrupted sources. GPTZero, which ran the review, coined the term for the failure: vibe citing.

The KPMG investigation page could not be resolved at a stable URL; this links GPTZero's investigations index entry that covers the same review. Replace with the KPMG-specific URL once confirmed.

EY Canada withdrew a 44-page report after 16 of 27 cited sources were found fabricated, misattributed, or dead, including phantom Forbes, McKinsey, and Gartner citations.

Withdrawn May 2026. With Deloitte (October 2025), three Big Four firms in nine months.

A World Bank paper on obesity trends was found to contain at least 14 references to work that does not exist.

A librarian checking a Springer Nature article on bowel-surgery management found 12 of its 14 references did not exist.

"Mariner 1 was lost to a missing hyphen" — the actual defect was a missing overline in the hand-transcribed guidance equations. The $18.5M setback is real; the hyphen is not.

"£8.8M for a one-letter typo" (Taylor & Sons) — the liability ruling is real; £8.8M is the claimant's valuation, and no public final award exists.

"The million-dollar comma" (Rogers v. Aliant) — the regulator initially ruled on the comma, then reversed itself when the French-language contract text controlled. The $2.13M never changed hands over punctuation.

"Poor data quality costs organizations $12.9M a year" — a 2020 Gartner figure from 154 self-reported customer estimates; the primary page is no longer locatable.

"GPT-5.5 hallucinates 86% of the time" — contradicts the primary grounded-summarization leaderboard (~9.3% for the same model); no named benchmark behind the viral figure.

"Poor writing costs US businesses $400B a year" — traces to a consultant's extrapolation, not a study; no primary methodology has surfaced.

"We'll drown in AI slop" attributed to curl's Daniel Stenberg — a paraphrase that circulates as a quotation. His actual on-record lines are strong enough; quote those.

"17.5% of CS abstracts are LLM-modified" — superseded by the peer-reviewed version of the same work: up to 22% for CS papers, ~9% for math and Nature-portfolio journals.