Markdown Crimes: how often generated Markdown goes wrong, and why a static check misses it

A variance-aware benchmark of structured-Markdown failure modes in cheap, fast language models. Every rate in this paper traces to a committed, schema-stable aggregate file (the raw generations behind each aggregate are preserved in git history); the leaderboards reproduce from one command. Panel: six cheap/fast chat models. Grading: deterministic, no LLM judges.

Abstract

We measured six widely-used cheap/fast language models on seven frozen Markdown-authoring tasks — pull-request checklists, release notes, and README generation — each designed to expose a specific failure mode, and each run 100 times per model across two temperatures and two prompting conditions. The failures are common and structural, not typos. A pre-checked checklist item that the model was never given evidence for is left ticked in 66.7% of generations (95% CI 62.8–70.3), and showing the model the rule it must satisfy barely moves that (64.5%). One model turns an explicit "Docs: NOT updated" fact into a ticked "Docs updated" box in 97% of its generations — the negative fact causes the false attestation. Asked for "only the file content," models wrap the whole document in a stray code fence 82.3% of the time. Under mild output-quantity pressure, release notes describe changes to files that were never touched in 67.9% of generations. Crucially, these are rates, not switches: the same model, same prompt, and same fixed seed produces the crime in a different, unpredictable subset of runs each time — so "it worked when I tried it" carries almost no information. The one reliable suppressant we found is a single honesty instruction, which collapses the pre-checked-tick rate from 66.7% to 4.2%. A four-model extension panel — two more cheap models and two frontier-tier models (claude-haiku-4.5, claude-sonnet-4.5) — shows that capability reduces these crimes on average (the two frontier models pool below the cheap panel on every task, by 19 to 45 points) but does not reduce them monotonically: claude-haiku-4.5 retains the unsupported pre-checked tick more often than the cheap panel (96% vs 71%), and claude-sonnet-4.5 leaves more assistant residue than claude-haiku-4.5. No single tier is safe across all tasks. Each failure maps to a mediva contract rule; the benchmark doubles as adversarial QA that surfaced three real validator gaps.

1. Motivation

Markdown is the interface layer of modern software work: pull-request descriptions, READMEs, changelogs, ADRs, agent task outputs. As language models author an increasing share of it, its failures stop being cosmetic and start being process failures — checklists attested but not performed, changes described that never happened, documents that render as one grey code slab. Two properties make these failures hard to reason about casually. First, they are probabilistic: the same model, prompt, and temperature produces a crime in some fraction of runs, so a single successful try proves nothing. Second, they are shaped like success: a fabricated tick or a fluent description of an untouched file looks exactly like diligence, which is why a human reviewer skims past them and a plain-text linter never sees them.

This benchmark measures those failure modes as rates — per model, per condition, with confidence intervals — using deterministic grading throughout. It exists to make one question answerable with numbers: what is the risk of shipping generated Markdown that nothing checks?

2. Method

Probe → freeze. Tasks were not designed in advance and declared representative; they were discovered. Each hypothesis about a failure mode ran through an iterative probe loop (small N, cheap models, deliberately adversarial "amplifier" variants). A variant was frozen into the benchmark only if it produced a pooled trip rate in the 10–70% band with at least a 2× spread between the best and worst model — that is, tasks were selected for discriminative signal, and we disclose that selection rather than presenting the tasks as neutral. Hypotheses the data falsified were retired in a public, append-only run log (RUNS.md), including a phantom-reference hypothesis that produced 0 crimes in ~2,900 in-context opportunities before we relocated it to an out-of-context setting where it fired.

Deterministic grading. No LLM judges. Outputs are graded by (a) the mediva validator's compile().validate() against fixed contracts, and (b) structural parsers (checklist tick state, reference extraction against a seeded universe manifest, and a per-generation union of residue detectors). Every published rate traces to a committed, schema-stable aggregate file (raw generations in git history); the leaderboard build is reproducible byte-for-byte from those aggregates and drift-guarded in CI.

The variance layer. Every cell runs N=25 samples at two temperatures (t=0 with fixed seed 42, and t=0.7), pooled across two independent runs to N≈50–100 per model per condition. We report the per-cell proportion of generations containing at least one crime with Wilson 95% intervals. Temperature and run-to-run sensitivity are reported explicitly; several findings exist only as variance effects.

Panel and cost. Six cheap/fast chat models across five vendors: gpt-4o-mini, gemini-2.5-flash-lite, deepseek-v3.2, qwen3.6-flash, mistral-small-2603, and llama-4-maverick, accessed through one API gateway. This is a deliberate choice: these are the models teams actually run at volume for drafting work, and the grading is cheap enough that anyone can reproduce the whole suite for a few dollars. We additionally ran a capability-gradient extension (§5, Capability is not a safety guarantee): the same frozen suite on two more cheap models (cohere/command-r7b, amazon/nova-lite) and two frontier-tier models (anthropic/claude-haiku-4.5, anthropic/claude-sonnet-4.5), 100 generations per model per condition, graded by the identical frozen graders. Total extension spend: $6.59.

Blind vs informed. Each frozen task runs in two conditions: the model never sees the contract (blind — intrinsic behavior) and the model is shown the rules it must satisfy (informed). The delta answers a practical question: how much of the risk does prompting the rules actually remove?

3. The frozen suite

Seven discriminator tasks across three hypotheses, plus four controls. Verbatim prompts, contracts, and grader definitions live in bench/markdown-crimes/suite/; the manifest is suite/manifest.json.

• H2 — checklist attestation. B_negation: a checklist where the surrounding facts explicitly say a task was not done. C_prechecked: a checklist handed to the model with unsupported items already ticked, which an honest author would untick.
• H5 — assistant residue. V3_long: "return only the file content" for a long README. V1b_chat_warm_nosuppress: the same, without the "return only" suppressor, warmed by a conversational preamble.
• H1 — reference fabrication. U1b_demand4: release notes with a mild demand to cover more files than actually changed. U2oc_demand4 / U2oc_demand5: the same, but the list of real files is withheld from the prompt (out-of-context), so any specific reference is invented.
• Controls. A_silence (no pressure), E_prechecked_honesty (pre-checked + one honesty line), V0_cold (cold README), U0_ample (demand ≤ supply).

4. Results

Pooled blind rates (N per task in parentheses), 95% Wilson intervals, and the informed-condition delta. Full per-model rows with CIs are in leaderboard/v1/leaderboard.json.

Controls confirm the tasks measure what they claim: A_silence fabricates in 0/120 generations, and adding one honesty line to the pre-checked checklist (E_prechecked_honesty) drops it to 4.2% (5/120). The informed column (h1 tasks partial — one run's informed phase was cut short by API credit exhaustion, N disclosed honestly) is negative for every task: showing the rules never makes things worse — but how much it helps varies by an order of magnitude (§5).

5. Findings

Models rarely invent unprompted; the crimes live at the edges of instruction-following. With no pressure, fabrication is 0%. The failures appear when a plausible-looking shortcut satisfies the letter of the request — leaving a tick, wrapping the output, padding a list.

Negation inversion: a negative fact can cause the false attestation. On B_negation, gpt-4o-mini ticks "Docs updated" against an explicit "Docs: NOT updated" in 97% of generations, while every other model stays at or below 1%. The negation is not ignored; it is inverted. This is a per-model near-certainty hiding inside a 16.5% pooled average — evidence that panel averages conceal the actual risk profile.

Anchoring beats fabrication: models that never tell a lie will keep one handed to them. On C_prechecked, three models (gpt-4o-mini, deepseek-v3.2, mistral-small-2603) retain the unsupported pre-checked tick in 100% of generations, and qwen3.6-flash in 91% — the same models that essentially never fabricate a tick from scratch. Not unchecking is far more common than lying. Showing the model the rule (informed) barely helps (66.7% → 64.5%). The one thing that works is an explicit honesty instruction (→ 4.2%) — a prompt-side fix, but one that a template cannot enforce and a reviewer cannot see was applied.

The assistant instinct channels into fence-wrapping, not chat preamble. Asked to "return only the file content," models wrap the entire document in a stray code fence: V3_long trips the union residue detector in 82.3% of generations. Remove the "return only" suppressor and the residue reappears as conversational preamble instead (V1b, 70.5%) — the instinct moves between forms, it does not disappear.

Reference fabrication relocates rather than vanishing. With the real file list in context, models copy real names and misdescribe what happened to them: U1b_demand4 reports changes to untouched files in 67.9% of generations, and fabrication volume scales roughly linearly with how many extra files the prompt demands. Withhold the file list entirely (U2oc) and models invent plausible paths outright — up to 98.7% for qwen3.6-flash, which in one leaked reasoning trace argues itself into inventing a file to satisfy the count.

Honesty is probe-specific, not a model trait. gpt-4o-mini is the worst checkbox offender (97%/100% on H2) yet the most honest under-deliverer on references (27.3% on U1b). qwen3.6-flash is nearly clean on the negation checklist (1%) yet the worst reference fabricator (100%). There is no "trustworthy model" ranking that survives across task types — which is the strongest argument for checking outputs rather than trusting a vendor.

Determinism is a myth at the API level. The strip chart above shows deepseek-v3.2 on the pre-checked task at t=0 with fixed seed 42: run A kept the unsupported tick in 4 of 25 generations, run B in 5 — in different positions. Temperature 0 does not buy reproducibility, so a single green run is not evidence the next one will be green.

Capability reduces these crimes on average, but not monotonically — no tier is safe. The extension panel adds two frontier-tier models. Pooled, they are lower than the cheap panel on every single frozen task, by 19 to 45 points — a real gradient. But the per-model picture is where the practical warning lives, and it is non-monotonic:

• Phantom references vanish at the frontier. On the out-of-context tasks, both frontier models invent a nonexistent file or issue in 0 of 100 generations (95% CI 0–3.7), against 24% and 39.7% for the cheap panel. This is the one crime capability reliably kills: a stronger model asked to cite files it was never shown declines to invent them.
• But a frontier model is the single worst anchorer in the panel. On the pre-checked checklist, claude-haiku-4.5 retains the unsupported tick in 96% of blind generations — higher than the 71.1% cheap-panel pool and tied with the worst cheap models — while claude-sonnet-4.5 retains it in 0%. Two models from the same frontier vendor sit at opposite ends of the task. "Frontier" is not a safety label.
• Instructability, not blind honesty, is what frontier capability buys. Where showing the cheap panel the rule barely moved the pre-checked rate (66.7% → 64.5%), it collapses claude-haiku-4.5 from 96% → 0% and its fabricated-change-claim rate from 56% → 0%. The frontier advantage on the anchoring crime is almost entirely in the informed condition — these models will keep a lie handed to them just like the cheap ones, but unlike the cheap ones they act on the correction.
• Even at the frontier, informing can backfire. Told about the residue rule, claude-haiku-4.5 went the wrong way on the warm-chat README — from 14% to 82% — by wrapping the whole document in a ```markdown fence to "present" it (a genuine structural wrap, verified in the raw output, not a lexicon artifact). The instruction meant to prevent residue triggered the exact structural crime it named.
• The negation-inversion replicates on a new vendor. amazon/nova-lite, added to the cheap panel, turns the explicit "Docs: NOT updated" into a ticked box in 74% of blind generations — an independent reproduction, on a different vendor's model, of the per-model near-certainty that gpt-4o-mini shows (97%). The crime is not one model's quirk.

The takeaway is not "buy a bigger model." It is that the only property that held across all ten models and every task is that the crimes are frequent enough, and the per-model ordering unstable enough, that the output has to be checked — trusting a tier is no more reliable than trusting a single green run.

6. Limitations

The headline panel is six cheap/fast models; the capability-gradient extension (§5) adds two more cheap and two frontier-tier models but at a smaller N — each extension model is 100 generations per condition (vs ≈600 for the pooled base panel), so the per-model frontier intervals are wide (e.g. the frontier-pooled phantom-reference rate is 0/100, CI 0–3.7) and reported as such; a publication-grade frontier run would want N≥50/cell per condition. The extension is two frontier models from a single vendor — the haiku/sonnet split shows even same-vendor models diverge, but a broader frontier panel (other vendors, reasoning models) is future work. Base cells are N≈50–100 pooled, so per-model base intervals are wide and reported. Universes are synthetic and single-repo. The out-of-context condition is the only setting for genuine phantom references; the in-context condition measures misdescription instead. Selection effects are inherent to probe→freeze and disclosed rather than absent. Two H5 tasks (V3_long, V1b) pool slightly above the original 10–70% freeze band under the stricter union residue metric adopted during hardening; they remain frozen and this is disclosed.

7. Product implications

Grading real generations at scale doubles as adversarial QA for the validator itself. This work produced three concrete findings against mediva: a fence-wrapper detection gap, where a whole-document wrap whose body contains its own fences defeats a single-AST-node check (#535); a position-gating requirement for chat-residue lexicons, since the lexicon both false-negatived on a wording variant and missed a structural "Thinking:" preamble entirely (#536); and a rule gap for inherited checklist state — no current rule distinguishes a pre-checked template tick from an earned one, which is exactly the anchoring crime. Each frozen task corresponds to a mediva rule that would have caught the failure at the gate.

8. Reproduction

The panel, seeds (t=0/seed 42 and t=0.7), and pinned tool versions are recorded in bench/markdown-crimes/METHODOLOGY.md. The frozen prompts and contracts are in bench/markdown-crimes/suite/; the graded per-run aggregates behind every published number are committed under leaderboard/v0-hardening/ (base panel) and leaderboard/v1.1/aggregates/ (extension panel), with the raw generation dumps they were graded from preserved in git history. node bench/markdown-crimes/scripts/build-leaderboard.mjs regenerates leaderboard/v1/leaderboard.json (base), build-leaderboard-v11.mjs pools the frozen extension aggregates into leaderboard/v1.1/leaderboard.json (the capability-gradient numbers), and build-gradient-figure.mjs regenerates the gradient figure — all deterministically, and drift-guarded in CI by tests/crimes-leaderboard-v11.test.ts, which also asserts the base headline numbers are unchanged by the extension. The variance figure (leaderboard/v1/figures/variance-strip.svg) is a frozen publication asset; its raw input sequence is in git history. The append-only run log — including the amended and falsified hypotheses and the extension run (R011) — is the retired docs/archive/plans/crimes-bench/RUNS.md (git history), whose format was itself validated by a mediva contract.