MotirBuilding in public
MOTIR · moooon
onMotir
You’re viewing a public project. Anyone can view it — no account needed. Sign in to submit, upvote, or comment on requests.View-only — you can’t edit work items
MOTIR-3475

(motir-core) Cut the 14 SCHEDULED jobs over IN PRODUCTION — ADD the ids to `MOTIR_POSTGRES_JOB_IDS`, read each cadence back from the ledger

Done
Description

Move the 14 scheduled jobs onto the Postgres engine in production, in waves, reading each job's cadence back from the ledger before starting the next. No PR — this is an operator action, marked done on Yue's confirmation.

Why this is not a child of the story

Its first step needs the story's merge. A manual post-deploy card INSIDE the container it verifies deadlocks that container — no PR until the child is done, no child until there is a deploy — and completing the container anyway CASCADES done onto it, unverified, from any status. That is planning bug MOTIR-3429, and MOTIR-3153 before it. So it sits under the epic as a sibling the story blocks, exactly where MOTIR-3463 sits.

⚠️ ADD to the variable. Do NOT replace it.

MOTIR_POSTGRES_JOB_IDS is one comma-separated list for the whole migration, and by the time this card runs THREE cards will have written it: MOTIR-3467 puts the pilot email.send in it, MOTIR-3463 adds the other 19 event-triggered ids, and this card adds these 14. fly secrets set writes the whole value; there is no append.

So the first step is to READ the current value's ids back and set the union — and that is why this card is sequenced after MOTIR-3463 rather than beside it. Setting only the 14 would silently move every already-migrated job back to Inngest in one rolling restart, which is a regression nobody would attribute to a scheduled-jobs card.

Record the value you read and the value you set, both verbatim, on this card.

Hypotheses to verify rather than assume

This card describes a system outside the agent's reach, so its claims about that system's current state are written as hypotheses with a verified-no-change exit:

  • The worker process group is running at count ≥ 1MOTIR-3425 owns bringing it up. fly status -a motir-core before anything else. A queue with no worker accumulates rows silently, and for scheduled jobs that is indistinguishable from a job that never fired.
  • The deploy carrying MOTIR-3416's merge is live — a merge is not a deploy; confirm the running release, and confirm the migration adding scheduled_for actually ran.
  • The variable already holds the pilot and MOTIR-3463's ids, and their waves read clean. If MOTIR-3463 has not run, stop: interleaving two cutovers on one variable is how a rollback becomes ambiguous. If the value holds ids nobody's card accounts for, stop and reconcile.

The mechanism, and the one thing easy to get wrong

lib/jobs/engine/cutover.ts reads process.env[MOTIR_POSTGRES_JOB_IDS] on every routing call rather than caching it — but an env var is still fixed at PROCESS boot, so a new value takes effect when the machines restart, not when the secret is written. fly secrets set triggers a rolling update, which is what applies it; --stage defers it to the next deploy and would leave the change written and not live. Do not use --stage. Rollback is the same move in reverse: remove an id, set the secret again, one rolling restart, no deploy.

The waves

Each wave: set the secret (union, per above), wait for the roll, wait out one full cadence of the wave's slowest job, then read the ledger. Do not start a wave until the previous one shows succeeded runs at the expected cadence.

  1. Fast and low-stakes, so a mistake is visible within minutessystem.ci-runner-provision-sweep (* * * * *), system.ci-runner-reap, system.plan-target-lock-sweep (*/10 * * * *).
  2. Hourlysystem.abandoned-plan-sweep, system.auto-plan-cadence-tick, system.ci-actions-gate-sweep, system.filter-subscription-tick, system.migrate-onboarding-sweep.
  3. Dailysystem.attachment-gc, system.automation-retention-sweep, system.code-graph-offboard-sweep, system.rate-limit-sweep, and system.daily-health-check. The health check goes in this wave deliberately: once it is on the engine, the schedule-health probe is itself running on the substrate it reports about, and confirming it still reports the others correctly is the point.
  4. Monthlysystem.ci-minutes-reconcile (0 4 3 * *). It will not fire inside this card's window. Do not wait for it and do not mark the card done on a hope: confirm the row is routed and that the tick's next fire time is computed, and NAME the date its first engine run is due so somebody can check it.

Leave the three container-supervisor ids OUTsystem.code-graph-index, system.code-graph-refresh, system.ci-runner-boot. They are MOTIR-3417's.

Acceptance criteria

  • fly secrets list -a motir-core shows MOTIR_POSTGRES_JOB_IDS, and the machines were rolled after it was set — verified from the platform, never from a config file.
  • The final value contains the union of every previously-routed id and these 14, is recorded verbatim on this card alongside the value that was there before, and does NOT contain the three supervisor ids.
  • For each wave, /settings/workspace/jobs shows succeeded scheduled runs for that wave's ids at the declared cadence — quoted per wave with the observed run times, not asserted in aggregate at the end. A single run proves it fired; two consecutive runs one interval apart prove it is scheduled.
  • Exactly one run per tick, checked with more than one worker machine running if the group is at count > 1 — this is the production reading of the story's headline property, and it is the only place it is observed on real hardware.
  • A migrated cron job's Inngest runs show { skipped: 'routed-to-postgres-engine' } rather than executing.
  • One rollback is exercised deliberately on one scheduled job: remove its id, confirm it fires on Inngest again at its next tick, re-add it.
  • system.ci-minutes-reconcile's first engine fire date is named on this card, with whoever will confirm it.
  • The app process group is unaffected and still at its previous count.

Context refs

  • lib/jobs/engine/cutover.ts — the env var, the live read, and the --stage hazard
  • /settings/workspace/jobs — the operator surface every wave is read back from
  • lib/services/jobScheduleHealthService.ts — the probe that will start reporting on the engine's behalf
  • MOTIR-3467 · MOTIR-3463 — the two cutovers that write this variable before this one does
  • MOTIR-3425 — the worker provisioning this depends on, and the precedent for reading a platform back from its own API