MotirBuilding in public
MOTIR · moooon
onMotir
You’re viewing a public project. Anyone can view it — no account needed. Sign in to submit, upvote, or comment on requests.View-only — you can’t edit work items
MOTIR-3463

(motir-core) Cut the 20 event-triggered jobs over IN PRODUCTION — `MOTIR_POSTGRES_JOB_IDS` in waves, read back from the ledger

Done
Description

Move the 20 event-triggered jobs onto the Postgres engine in production, in waves, reading each wave back from the ledger before starting the next. No PR — this is an operator action, marked done on Yue's confirmation.

Why this is not a child of the story

Its first step needs the story's merge. A manual post-deploy card INSIDE the container it verifies deadlocks that container — no PR until the child is done, no child until there is a deploy — and completing the container anyway CASCADES done onto it, unverified, from any status. That is planning bug MOTIR-3429, filed one story ago against exactly this shape, and MOTIR-3153 the story before that. So it sits under the epic as a sibling MOTIR-3415 blocks, the same position MOTIR-3425 was re-parented into.

Hypotheses to verify rather than assume

This card describes a system outside the agent's reach, so its claims about that system's current state are unverifiable at authoring time and are written as hypotheses with a verified-no-change exit:

  • The worker process group is running at count ≥ 1. MOTIR-3425 owns bringing it up. Confirm with fly status -a motir-core before anything else — a queue with no worker accumulates rows silently and the dashboard shows nothing, which is indistinguishable from a job that never fired.
  • MOTIR_POSTGRES_JOB_IDS is currently unset or empty, so every job is on Inngest today. Confirm with fly secrets list -a motir-core. If it already holds ids, stop and reconcile — someone has cut over ahead of this card.
  • The deploy carrying MOTIR-3415's merge is live. A merge is not a deploy: confirm the running release, not the PR.

⚠️ The mechanism, and the one thing about it that is easy to get wrong

lib/jobs/engine/cutover.ts reads process.env[MOTIR_POSTGRES_JOB_IDS] on every routing call rather than caching it — deliberately, so no machine can hold a stale view. But an env var is still fixed at PROCESS boot, so a new value takes effect when the machines restart, not when the secret is written. fly secrets set triggers a rolling machine update, which is what applies it; --stage defers it to the next deploy and would leave the change written and not live. Do not use --stage here.

The same applies to rollback, which is the property this whole design was bought for: remove an id and set the secret again. No deploy, no code change, one rolling restart.

The waves

Each wave: set the secret, wait for the roll, drive the trigger, read the ledger. Do not start a wave until the previous one shows succeeded runs.

  1. Single-subscriber, low blast radiuswork-item/embedding.requested, filter-subscription/deliver, system.billing-seat-sync.
  2. email.send — the job carrying the only idempotency key. Confirm one email per key, and that a DLQ replay still succeeds.
  3. The fast lane, one consumer at a timestatus-derivation/transitioned, then watcher-notify/transitioned, then notification-fan-in/transitioned, then automation-engine/transitioned. Moving them singly is what the split-subscriber design exists for; leave all four routed once they are through, because the measurement in the next card needs the lane whole.
  4. The remainder — the consumers of work-item/comment.created, work-item/created, work-item/mentioned, work-item/field.changed, work-item/child-set.changed and work-item/derivation.requested.

Leave the three container-supervisor ids OUTsystem.code-graph-index, system.code-graph-refresh, system.ci-runner-boot. They are MOTIR-3417's, and this story only made them routable.

The latency reading is NOT this card's — it was split out

It was scope here, and it is not any more. Producing the six figures needs HOURS of whole-lane tail after wave 3 completes, and holding this card open for it kept MOTIR-3475 and MOTIR-3418 blocked on a cutover that was already finished. The reading now has its own operator card, which this one blocks; transcribing the figures into the codebase remains MOTIR-3464's. This card ends when the 20 ids are routed and read back.

Acceptance criteria

  • fly secrets list -a motir-core shows MOTIR_POSTGRES_JOB_IDS, and the running machines were rolled after it was set — verified from the platform, never from a config file.
  • All 20 job ids are in the set, and the three supervisor ids are not. The final value is recorded on this card verbatim.
  • For each wave, succeeded job_run rows appear for that wave's ids on the engine — quoted on the card per wave, not asserted in aggregate at the end. An id with no production traffic cannot satisfy this and must be recorded with an explicit exit rather than left silent.
  • A migrated job's Inngest runs show { skipped: 'routed-to-postgres-engine' } rather than executing — confirming the double-run guard holds in production. ⚠️ That marker is returned BEFORE the ledger row is written (lib/jobs/defineJob.ts:398), so it is visible only in Inngest's own API / dashboard and never in job_run. In the ledger the guard shows as the ABSENCE of a second row.
  • One rollback is exercised deliberately on one job: remove its id, confirm it runs on Inngest again, re-add it. The reversibility this design was bought for is asserted once rather than assumed.
  • The app process group is unaffected and still at its previous count.

Context refs

  • lib/jobs/engine/cutover.ts — the env var, the live read, and the --stage hazard above
  • lib/jobs/defineJob.ts:398 — the decline path, and why the skip marker never reaches the ledger
  • /settings/workspace/jobs — the operator surface the waves are read back from
  • lib/jobs/latencyBudget.tsFAST_LANE_CONSUMER_IDS, wave 3's list
  • MOTIR-3425 — the worker provisioning this depends on, and the precedent for reading a platform back from its own API