When the Database Stops Matching the Migrations

Every team with a migrations directory believes the same quiet thing: replay the chain against an empty database and you get the schema production is running. It feels less like a belief than a definition. The migrations are the schema; that is what they are for.

KittyLog's were not. On July 31, 2026, a read-only audit compared production's column types against a fresh alembic upgrade head replay and found twelve timestamp columns where the two disagreed. Production stored them as naive timestamp columns holding UTC values. The migration chain said they should be timestamptz. Same data, same application code, two different schemas, and the test suite validating the one production did not have.

How a database drifts

The usual suspects are hotfixes applied straight to production, manual surgery during an incident, and migration files edited after they have already run somewhere. Ours was a fourth, more respectable-looking mechanism: stamping.

KittyLog's production database predates its Alembic baseline. When the chain was introduced, production was stamped to a revision rather than replayed, which is the standard move; you cannot replay a creation migration against tables that already exist and hold data. But a stamp is a statement, not an action. It records "treat this revision as applied" without running anything. Somewhere in the stamped range sat convert_timestamps_to_timestamptz, a December 2025 revision that converts those twelve columns. Alembic's version table said it had happened. The columns said otherwise.

The audit confirmed it cleanly: production's alembic_version was several revisions past the conversion, and the conversion had never executed there. Nothing in Alembic could have flagged this earlier; the version table holds only the current revision, so there is no run log for a stamp to contradict.

We had a warning shot ten days earlier. The work in PR #1297 switched the integration tests from a hand-written schema to one built by replaying the chain, and immediately found that the chain could not replay from scratch at all, plus a live production bug where one naive-timestamp column made a two-column UPDATE ambiguous for the driver. Issue #1308 was filed on the suspicion that those two drifts were not the last. They were not.

Why the guard fails closed

The reconciliation migration that came out of the audit converts each of the twelve columns with AT TIME ZONE 'UTC' semantics, preserving the instant each value represents. The interesting design decision is what it does when a column is not in a state it recognizes.

The first version quietly handled the two expected cases: naive timestamp gets converted, timestamptz only gets its nullability aligned, which leaves a fresh replay unchanged. A follow-up commit, "fail closed on timestamp schema drift", added the third case: anything else, including a missing column, raises an error and stops the migration.

That choice is worth defending, because the polite alternative (log a warning, keep going) is tempting. But think about what an unexpected type means in this specific migration. This is a migration that exists because the database's actual state diverged from its recorded state. If, mid-run, a column turns out to be something the audit never saw, the one thing you know is that your model of the database is wrong again. Proceeding on a wrong model is how the drift accumulated in the first place. A hard stop leaves the operator with an exact table and column name and an untouched database; a warning leaves them with a half-applied migration and a log line nobody reads.

The migration is also deliberately forward-only. Its downgrade is a documented no-op, because Alembic has no way to know whether a given column was naive before the upgrade ran, and guessing would corrupt a freshly built database.

What is still not fixed

"We fixed the schema drift" would be overclaiming. The audit and the timestamp reconciliation merged in PR #1366. Running the migration against production is a separate, operator-gated step with its own review.

And the audit found more than timestamps. Issue #1369 tracks the rest, and it is open as I write this: a text column in production that the chain thinks is varchar(100), a server default that exists on one side only, five columns the baseline creates that production has never had, and three legacy team tables that live in production with no matching migration anywhere in the chain. Each needs a decision about which side is the contract, and a couple need data profiled before anything gets narrowed or dropped.

So the honest status is: the belief was false, the worst class of drift now has a merged fix waiting on an operator, and the remainder is at least a numbered list instead of an unknown. #1369 stays open until each of those items has a decision and evidence behind it. I would rather report that than a tidy ending.

Source records

• Issue #1308, the schema-drift audit task.
• Issue #1369, the remaining non-timestamp divergence (open).
• PR #1366 and commits cc97b9ca / f5abd016, the reconciliation migration and its fail-closed guard.
• PR #1297, the chain-replay hardening that surfaced the first drifts.
• docs/audits/issue-1308-schema-audit-2026-07-31.md and alembic/versions/2026_07_31_reconcile_production_timestamp_schema.py.