The Changelog That Logged Zero Tokens

The last row landed in llm_usage_logs at 21:55 UTC on May 5, 2026. For most of the next day, KittyLog kept generating changelogs normally. Webhooks fired, scheduled runs ran, entries were written, users saw their release notes. Ten changelogs went out in a single 24-hour window while the table that records every AI call and its token cost accepted exactly four rows, all from before the break.

Nothing paged. Nothing errored, as far as any dashboard could tell. I found it by accident, while wiring up a daily status email that puts a few production numbers in my inbox each morning. The first draft of that query came back with tokens_today = 0 and changelogs_today = 2, which is a combination that cannot happen. A changelog without tokens is a changelog nobody generated.

What actually broke

The day before, a commit had added two columns to llm_usage_logs: raw model output and reasoning text, for an admin debugging view. The commit included the Alembic migration, the INSERT statement was updated to write the new columns, tests passed, and the code shipped to production via fly deploy.

The migration was never run.

On this stack, deploying code and migrating the database are two separate steps: deploy first so the migration file exists on the server, then run alembic upgrade head over SSH. The second step was skipped. So production had the new INSERT and the old table, and every single insert from that moment raised UndefinedColumnError.

That error never surfaced, because of two decisions that were each individually reasonable. First, usage logging is deliberately non-blocking: if the metering insert fails, we swallow the error and the changelog generation succeeds anyway. I still think that is right. A customer's release notes should not fail because our bookkeeping did. Second, the swallowed error was logged at warning level, and our Sentry integration only captures ERROR-level records that carry a traceback. Warnings go to the log stream, which nobody reads when everything appears fine.

Put those together and you get a failure mode with no symptoms. The feature worked, so nobody looked. Every surface a human actually watches was green.

The meter is a product feature

It is tempting to file this under "observability hygiene" and move on, but for a SaaS the usage table is not internal plumbing. Cost monitoring reads it, the admin billing reports read it, and so do the per-account usage display and the AI provider failure tracking. When that table went dark, all of those went quietly stale at once, and each of them looked like a slow day rather than a broken one.

That is the trap with metering specifically: its absence is indistinguishable from low traffic. A missing changelog gets noticed by the customer who expected it. A missing usage row gets noticed by nobody, because zero is a perfectly plausible number.

The cost: roughly a day of usage data is gone. We looked at reconstructing the missing rows from the changelog entries themselves and decided against it. The reconstruction would have been lossy, roughly right at best, and a table that mixes measurements with guesses is worse than one with an honest gap.

The fix was four lines, mostly

The repair itself took a morning. Run the migration, and the bleeding stops. The real change in PR #1218 was smaller and duller: four catch sites in the logging module were elevated from logger.warning to logger.exception, so any future insert failure lands in Sentry as an ERROR with a full traceback. The behavior for callers is unchanged. Usage logging still never raises into a generation; it just can no longer fail invisibly.

The PR also added regression tests, including one that asserts the failure path emits an ERROR record with traceback attached. The happy path already had tests. So did the failure path, after a fashion: an existing test asserted that a failed insert quietly returns None. Nothing checked that the failure would be visible anywhere, which in hindsight was the tell.

Two more guards accumulated afterwards. The daily status email now carries the token total next to the changelog count every single day, so the impossible zero that exposed this bug is checked automatically each morning. And since July 2026, CI replays the entire migration chain from scratch on every backend change, which catches chains that cannot be applied cleanly. That last guard has a limit: CI can prove a migration is runnable, but it cannot prove someone ran it against production. That step is still a human typing a command after a deploy, and the Sentry alarm exists precisely because humans skip steps.

So the check that exists today is layered and unglamorous: a failed usage insert now shows up in Sentry within minutes, with a traceback, and a tokens-to-changelogs mismatch shows up in the next morning's email. Neither required new infrastructure. If you run a metered product, I'd suggest wiring the equivalent of both before you need them.

Source records

• Issue #1217, the zero-tokens report with the production evidence.
• PR #1218, the logger.exception elevation and regression tests.
• Commit ad91c4c2, the column addition that outran its migration.
• Commit 3865af12, the merged fix.
• api/core/llm_logging.py, the current catch sites.