Validate LLM output before persisting
Let a model repair generated Markdown shape, then require human-owned state before saving or rendering.
TARS: I can reformat the checklist. I cannot truthfully check the box.
Prompt for your coding agentOpen in Claude ↗
In this codebase, validate generated Markdown with mediva before saving it: compile the contract, check it through shape and let the model repair only those findings, then refuse to persist anything whose full check still fails.
<!-- mdv: document content.none=@agent/chatter content.none=@agent/cut-off -->
<!-- mdv: section required content.words.min=25 content.none=@placeholders -->
## Summary
The checkout API returned 502 errors for twelve minutes after the 14:05 deploy, failing about eight percent of card payments; the deploy is rolled back and error rates are back to normal.
<!-- mdv: section required -->
## Evidence
<!-- mdv: list required items.min=2 items.all=@evidence/specific error=vague-evidence -->
- Grafana `checkout_5xx_rate` peaked at 8.1% at 14:09 UTC and fell to 0.2% by 14:17 UTC.
- `kubectl rollout history deploy/checkout` shows revision 42 rolled back to revision 41 at 14:15 UTC.
<!-- mdv: section required -->
## Human Review
<!-- mdv: list required items.all=checked labels.eq=@template -->
- [ ] Source data checked.
- [ ] Reviewer approved persistence.## Summary
Checkout had some problems, TBD.
## Evidence
- It was slow.
- Things looked bad.
## Human Review
- [ ] Source data checked.
- [ ] Reviewer approved persistence.A reply wrapped whole in one code fence needs no rule: the lint gate reports it (fenced-output) on every document. The model repairs lint and shape findings only; thin prose, vague evidence and unticked boxes go to a human.
import { readFile } from "node:fs/promises";
import { compile } from "mediva";
type Generate = (prompt: string) => Promise<string>;
type Route =
| { ok: true; markdown: string }
| { ok: false; reason: "shape"; errors: unknown }
| { ok: false; reason: "state"; errors: unknown };
export async function validateGeneratedMarkdown(input: {
markdown: string;
contractPath: string;
generate: Generate;
persist: (markdown: string) => Promise<void>;
}): Promise<Route> {
const contract = await readFile(input.contractPath, "utf8");
const schema = compile(contract);
let markdown = input.markdown;
let shape = schema.check(markdown, { through: "shape" });
// The model sees only lint and shape findings, so it can fix form but never fake state.
for (let attempt = 0; attempt < 2 && shape.gate !== null; attempt++) {
const findings = shape.failing.map((d) => `line ${d.line ?? 1}: ${d.message} (${d.help})`).join("\n");
markdown = await input.generate(`Fix only these problems. Change nothing else.\n${findings}\n\n${markdown}`);
shape = schema.check(markdown, { through: "shape" });
}
if (shape.gate !== null) {
return { ok: false, reason: "shape", errors: shape.failing };
}
const state = schema.check(markdown);
if (state.gate !== null) {
return { ok: false, reason: "state", errors: state.failing };
}
await input.persist(markdown);
return { ok: true, markdown };
}Persist only after validate passes.
Prompt with the rules attached
import { renderMarkdown } from "mediva";
const prompt = `Fill in this document:\n\n${renderMarkdown(contract, { rules: true })}`;
// ## Summary
// <!-- rules: required · at least 25 words · no placeholders (TODO, TBD…) -->mdv render --schema contracts/ prints the same; validation ignores the comments.
Variations
| Want | Add |
|---|---|
| a linked tracker item that is open | section issue. (with --context) |
| resolvable links and footnotes | document content. |
| a pinned code language | code required lang. (lang. for a set) |
| fences that name their language | markdownlint's MD040 in config lint ({ default: false, MD005: true, MD056: true, MD046: { style: "fenced" }, MD040: true }); the lint gate runs it on every document |