CI You Can Afford to Run on Every Push

Every end-to-end gate you might add to CI has a price in runner minutes, and the price recurs on every push. KittyLog plainly needed several gates anyway: the billing path had no end-to-end test, and neither did the auth callback, connecting a repository, or generating a changelog. When we audited the CI setup in July 2026 (issue #1306), the whole Playwright directory held exactly one spec, a visual theme check, and no workflow ran it at all. The config also carried forbidOnly: false, so whenever e2e did land in CI, a committed .only could have quietly shrunk the suite and reported green.

That kind of finding produces an obvious reflex: gate everything. The reflex is wrong, or at least incomplete, because that tax is collected for the rest of the project's life. If the tax is high enough, people start routing around the toll booth. They batch changes to push less often, they stop pushing work-in-progress branches, or they learn which failures are "probably just CI" and merge anyway. A gate nobody trusts is worse than no gate.

So the repo keeps two disciplines side by side, and they pull in opposite directions on purpose. One says close the coverage gaps. The other, written down as an internal cost-control skill for anyone touching .github/workflows/, says treat runner minutes as a budget: path-filter every workflow so backend-only diffs never build the frontend, cancel superseded runs when a new push lands on the same branch, and never weaken an affected-code gate to save minutes. The interesting work is where the two meet.

What a gate has to be before it earns its slot

The e2e workflow that came out of #1306 (merged in July 2026, commit c70d57a9) runs Playwright against a stub API server that the test runner itself starts. A small Node script serves a fixture changelog on localhost; the production Next.js build points at it; no backend, no database, no external service is involved. The specs tagged @hermetic check that fixture entries render, that a ?theme= override actually applies the theme CSS variables, and that an unknown slug gets the 404 page.

Hermetic is the load-bearing word. An e2e suite that talks to a live backend fails when the backend hiccups or a seed script drifts. Every one of those failures teaches developers that red does not mean broken. The stub-API suite can only fail when the frontend is actually wrong, which is the whole reason it is allowed to block merges.

The same commit set forbidOnly: true, so a committed .only now fails CI instead of shrinking it. And the visual theme spec stayed local-only, deliberately: it needs live data and per-OS screenshot baselines, which is exactly the flavor of flake the blocking suite exists to exclude.

Cost-wise, the workflow triggers only on frontend/** changes, runs two shards, and cancels in-progress runs when a newer push supersedes them. A backend fix pays none of this tax.

The gate we refused to add

There is also a changelog-quality eval harness in the repo, and the audit asked whether it should run in CI. The answer was no: live AI providers make a lousy merge gate, a story that belongs with the harness itself. It is documented as manual-only instead, while its plumbing (the runner, fixtures, and CLI) still runs in CI with the AI manager mocked, so a refactor that breaks the harness gets caught without a single live model call.

Coverage thresholds got the same treatment: raised from measured reality, not aspiration. The backend suite measured 80.4% statements while the gate sat at 55, so the gate moved to 75 on main. Frontend measured 53.4% against a gate of 32; it moved to 50. A threshold a few points under what you actually have catches regressions without ever blocking an honest change.

One more fix from the same audit is my favorite, because it is the inverse failure. The build had typescript.ignoreBuildErrors: true, historically safe because a separate typecheck job would catch errors, assuming branch protection made that job required. On a free-plan private repo, GitHub's branch-protection API returns 403, so required checks could not even be verified. The flag came out; the build itself now fails on type errors. A gate that depends on configuration you cannot inspect is a gate you do not have.

A related one-liner (commit d52e79e1) closed a hole in the backend path filter itself, which had been letting eval-harness changes skip the backend suite.

Before adding a CI gate now, the questions are roughly these. Can it fail only when the code is actually wrong, with no live dependency that flakes on someone else's schedule? Does it run only when the paths it protects have changed, and does a newer push cancel the stale run? Is its threshold set from a measured baseline rather than a wish? Does it enforce itself in the artifact rather than in settings nobody can audit? And if the answer to the first question is no, as it was for the eval harness, does it stay out of the merge path entirely? A gate that clears all of that can stay. The rest we run by hand.

Source records

• Issue #1306, the CI-gap audit (Playwright e2e, eval harness, coverage gates, branch protection).
• Commit c70d57a9 / PR #1329, hermetic Playwright e2e, strict builds, raised coverage gates.
• Commit 6e4c4cf5, the earlier CI hardening pass (alembic check, frontend build, bandit, Codecov).
• Commit d52e79e1, adding eval/** to the backend path filter.
• .github/workflows/ci.yml and .github/workflows/e2e.yml.
• .claude/skills/github-actions-cost-control/SKILL.md, the runner-minute discipline.