A Changelog Generator Must Not Invent Features

The worst thing KittyLog could ever do is publish a release note announcing a feature that does not exist. Not a crash, not a slow page. A confident bullet that says "You can now export reports to PDF" when no commit in the repository adds PDF export.

Release notes are unusual among AI-generated text because people act on them. A reader who sees that bullet upgrades, goes looking for the export button, and files a support ticket when it is not there. A developer reading "breaking change in the auth API" reworks integration code against a change nobody made. A changelog is a factual record that readers treat as one, and a single invented entry makes every real entry suspect.

This is why "sounds plausible" is the enemy here specifically. A model writing release notes from commits is doing something adjacent to invention by design: it paraphrases, and it compresses dozens of commits into a few readable lines for a chosen audience. The output format practically invites filling in. Sections named Features and Fixes want members, and a fluent model can produce a member that fits the repo's vocabulary perfectly and corresponds to nothing. No reader can tell that bullet from a real one without opening the git history, which is exactly the work the product exists to save them.

In July 2026 we wrote the defense down as a hard engineering rule (issue #1323, ported from a sibling project's honesty engine for resume generation, where the same class of problem is a fabricated job achievement). It is three invariants, and any generation path that lacks them is treated as a regression even if every test passes.

Invariant one: grounding

Every bullet must trace to source material that was actually passed to the model: a fetched commit message, a diff, PR or issue context. Grounding is not "the model believed it." It is "the evidence passed in contains it." Faithful paraphrase is fine. Collapsing ten refactors into "various internal improvements" is fine, and honest. Inventing a feature is fabrication, and when the model is unsure whether the input supports a change, the rule is to omit it.

One corner matters more than it looks: repos can set a custom prompt. "Make it sound exciting, emphasize new features" must never license fabrication, so the grounding rules explicitly override anything a custom prompt says.

Invariant two: a validation pass

After generation, a separate adversarial check audits each bullet against the same evidence the generator saw. Not re-fetched evidence; the same blob, because re-fetching can introduce or hide material and break the audit.

The pass flags suspect bullets and leaves the text alone. Silently rewriting a generated changelog is its own trust hazard, since a false positive would delete a real bullet. A flagged entry is saved as a draft marked "needs review" with the findings attached.

The calibration is the hard part. A validator that cries wolf on "bug fixes and stability improvements" trains users to ignore the flag, and then a real fabrication slips through behind the noise. So honest hedges are on an explicit do-not-flag list; a named feature with no supporting commit is on the flag list. Recall catches fabrications, precision keeps the flag worth reading.

Invariant three: the truncation guard

If generation hits the output-token cap, the result is incomplete. There are two outcomes. Cut mid-JSON, the parse fails and the failure is loud, though only by accident. The nastier case is valid JSON that simply stopped early: it parses cleanly, looks complete, and silently drops the last commits' bullets. The sibling project shipped exactly this bug in June 2026, a truncated document saved as if finished. The rule: output at or near the cap is treated as a failure and is never saved as an entry.

One chokepoint, and honesty about status

KittyLog generates changelogs from five different triggers, but all five converge on a single model-call function, and in July 2026 we consolidated the five pipelines above it into one shared runner (issue #1299). Everything funnels through one chokepoint before anything is persisted, and that is what makes one guard sufficient.

So where does this stand? Writing the invariants down was itself an audit, and the audit's findings were uncomfortable: the system prompt carried no grounding block, nothing compared output tokens to the cap, and the validation pass existed only on paper. As of this writing, that is still the state of the code. The document exists precisely to gate the work of closing those gaps, and to stop new generation paths from reopening them. And even finished, validation reduces the risk rather than eliminating it. The validator is a model too, and a bullet grounded in a diff that was never fetched is one it cannot falsify.

Which is why the last line of defense is not a model at all. Review-before-publish is available on every tier, including Free: turn it on and the generated entry arrives as a draft so you can read it against your own memory of what you shipped. Disable the separate 24-hour auto-publish option and nothing goes public until you say so. For a document people act on, that step stays.

Source records

• Issue #1323, the changelog-grounding discipline (ported from the sibling honesty engine).
• Commit 8ac94fe0, adding .claude/skills/changelog-grounding/SKILL.md.
• Issue #1299 / commit 923f08d1, consolidating the five generation pipelines into one shared runner over the existing chokepoint.
• api/services/changelog/ai.py and api/services/changelog/prompt/system.py, the generation path the invariants govern.