MotirBuilding in public
MOTIR · moooon
onMotir
You’re viewing a public project. Anyone can view it — no account needed. Sign in to submit, upvote, or comment on requests.View-only — you can’t edit work items
MOTIR-3716

(motir-core) NOTHING reconciles the census against the live secret — `MIGRATED_TO_ENGINE` and `MOTIR_POSTGRES_JOB_IDS` drift silently, and each newly-routed job costs one more operator card

Done
Description

Type · code (a shipped reconciliation + its guard) Parent · MOTIR-3413 (the discovery epic — the finding card is a bug directly under it, so the edge test has no finding STORY to name; log-bug.md step 2 rung 1) Discovered in · MOTIR-3709, which is the FOURTH instance of the class and says so in its own explanation: "it will keep producing one of these cards per newly-routed job until something reconciles the declaration against the live value. That reconciliation is not this card's scope." This card is that scope.

The defect

tests/jobs/every-job-declares-its-lane.test.ts holds MIGRATED_TO_ENGINE — a checked-in DECLARATION of which lane each job runs on, asserted TOTAL over the registry so that adding a job fails the build until its author names a lane. The lane a job ACTUALLY runs on is decided by the Fly secret MOTIR_POSTGRES_JOB_IDS, read live by routedToEngine. docs/jobs.md states the split outright and then says the two are kept equal by hand — which is a way of saying nothing keeps them equal.

Nothing reads production and nothing ever compares the two. The test cannot (its own header explains why, correctly: CI would fail for an operator action taken minutes earlier and go green when somebody changed production rather than the code). No other check does either, so a divergence is detectable only by a person running comm against a value read from inside a machine — which is exactly what each instance below cost.

Why it is silent in the worst direction

The consequence of the drift is not "no lane" — it is the wrong lane, and every code-side signal reads green:

  • lib/jobs/engine/scheduler.ts:186if (!routedToEngine(def.id)) continue; ⇒ the engine gives the job no timer.
  • lib/jobs/defineJob.ts:483if (routedToEngine(id)) return { skipped: 'routed-to-postgres-engine' } ⇒ the Inngest handler DOES run it, and the deploy registers the function.

So the job runs, daily, apparently fine, while a reviewed file states it runs on the engine. The disagreement surfaces only when MOTIR-3418 deletes the SDK and the job silently loses its subscriber.

The measured record — four instances in ~34 hours

cardwhat driftedfound by
MOTIR-3682system.public-follow-digest-ticka person reading a log line
MOTIR-3688plan-drift/transitioned, public-follow/digesta hand reconciliation
MOTIR-3709system.job-run-reapa hand comm while discharging MOTIR-3688's AC 4

MOTIR-3709 is the one that proves a census alone cannot close this: its author DID declare the lane in the same pull request that added the job, because the build made them. The authoring half now always happens. The deployment half is exactly as unowned as it was before the guard existed.

Fix direction — and the ONE decision it opens first

The declaration currently lives in a test file, so nothing shipped can read it. That is the first thing to settle, and it is the card's only real design choice:

  • (a) move the two lists into a shipped module (lib/jobs/engine/ beside cutover.ts) that the test then asserts against, or
  • (b) put the lane on defineJob itself, as a required option beside catchUp — which is where catchUp already proved this shape works.

Whichever wins, the reconciliation is then a runtime read that COMPARES the declared set against routedJobIds() and reports the two-way difference. Route it where an operator already looks — system.daily-health-check and the health surface are the obvious homes, and a startup WARN from the worker is the cheap complement (the worker is the process that would silently stop running the job).

A reconciliation must not be a hard failure at boot. A deploy legitimately runs for minutes with the code ahead of the secret — that ORDERING is required (docs/jobs.md's image trap, and MOTIR-3709's first criterion). It reports, loudly and somewhere durable; it does not refuse to start.

Acceptance criteria

  • The lane DECLARATION is readable from shipped code, not only from a test file, and the existing census test asserts against that same source rather than holding its own copy — one list, not two.
  • A runtime reconciliation computes the two-way difference between the declared engine set and routedJobIds(), and reports BOTH directions separately: declared-but-not-routed (the job runs on the wrong lane) and routed-but-not-declared (production is ahead of review). A one-way check reproduces this defect in the other direction.
  • The difference is surfaced where an operator already looks, and the surfacing is covered by a test that asserts the reported content for a seeded mismatch in each direction — not merely that the function was called.
  • A non-empty difference does NOT fail process start-up, and a test asserts that: the ordinary deploy window has the code ahead of the secret, and refusing to boot there would turn a routine release into an outage.
  • The empty case is asserted too — a set-equal deployment reports clean, so the signal is readable as an assertion rather than as the absence of noise.
  • docs/jobs.md's cutover section is amended where it says the two are kept equal by hand: it names what now reports the drift, and what an operator does when it fires.

Context refs

  • tests/jobs/every-job-declares-its-lane.test.ts — the census (MIGRATED_TO_ENGINE / DELIBERATELY_ON_INNGEST), and its header on why a TEST cannot read production
  • lib/jobs/engine/cutover.tsroutedToEngine / routedJobIds, the live read of the secret
  • lib/jobs/engine/scheduler.ts:186 · lib/jobs/defineJob.ts:483 — the two reads that decide the lane
  • lib/jobs/definitions/dailyHealthCheck.ts — an existing operator-facing reporting surface
  • docs/jobs.md — the cutover operator section: the declaration-vs-deployment split, the image trap, the read-modify-write race