MotirBuilding in public
MOTIR · moooon
onMotir
You’re viewing a public project. Anyone can view it — no account needed. Sign in to submit, upvote, or comment on requests.View-only — you can’t edit work items
MOTIR-3489

(motir-core) Cut the THREE container supervisors over IN PRODUCTION — ADD the ids to `MOTIR_POSTGRES_JOB_IDS`, watch a real push coalesce, boot and index

Done
Description

Move the three container supervisors onto the Postgres engine in production, and read the result back from the ledger and the platform — the operator half of MOTIR-3417, which its merge cannot perform.

Why this is a task under the EPIC rather than a child of the story

Its first step needs the story's merge to be deployed, and a post-merge card parented inside the container it verifies either deadlocks the parent run or is cascaded done unverified. MOTIR-3463 and MOTIR-3475 sit here for exactly this reason and this card is their third sibling — the one the epic's partition was missing. 14 cron + 20 event + 3 supervisors is the whole fleet; two of those three families had an operator card and the supervisors did not.

The steps

  1. Ship the DECLARATION first. ⚠️ Added 2026-08-27 — this card acquired a code half it did not have when it was written. MOTIR-3716 moved the lane census into shipped code, and lib/jobs/engine/census.ts lists all three supervisors under DELIBERATELY_ON_INNGEST. Adding the ids to the secret and nothing else produces the drift that file exists to catch, in the routedNotDeclared direction: dailyHealthCheck.ts throws JobLaneDriftError and dead-letters daily. So the three ids move to MIGRATED_TO_ENGINE in a pull request, and that pull request deploys before step 2 — census.ts states the order itself, deploy-then-route.

  2. Confirm the deploy that carries the story is LIVE — not that the pull requests merged. Read the running release, not main.

  3. ADD the three ids to MOTIR_POSTGRES_JOB_IDS, preserving everything already in it:

    • system.code-graph-index
    • system.code-graph-refresh
    • system.ci-runner-boot

    ⚠️ ADD, never SET. By the time this runs the variable already carries the 20 event ids and the 14 scheduled ones. Read the current value first and append; a fly secrets set with a fresh list silently moves 34 jobs back to Inngest, and nothing would report it — every one of them would simply keep working, on the wrong engine, until somebody looked. This is why this card is sequenced after MOTIR-3475: both write the same single variable, and the two must not race.

  4. Make it LIVE, and verify that it is. fly secrets set triggers a rolling machine update; --stage defers it to the next deploy. Both the app and the worker process groups must be running with the new value — app reads the switch when it emits, worker when it decides whether to run. Read it back from the platform (fly secrets list -a motir-core, and the machines' state), never from a local file.

  5. Watch a real push. Push to a connected repo's default branch — twice within the debounce window, which is what makes the coalescing observable — and confirm ONE system.code-graph-refresh run appears, boots its containers, and settles succeeded with its output.repoRef.

  6. Watch a real CI job. ⚠️ DOWNGRADED 2026-08-27 to routed and switch-verified, never exercised. system.ci-runner-boot has 0 job_run rows all time — it has never executed, so there is no window in which watching one becomes possible; system.code-graph-index has 15 all time and none since 2026-08-19. This is the same disposition MOTIR-3463 made for its five never-firing ids, and the lesson is already recorded (motir-ai#309). Prove the ids are ROUTED and that the switch reads them, record the counts, and do not wait for a trigger that has never fired.

  7. Check the fleet's spend did not move. The admission cap is untouched by this story, so the number of concurrent containers should look exactly as it did the week before. A coalescing bug shows up here first.

Rollback is removing the three ids and letting the machines roll. The switch defaults to Inngest for any id it does not name, and both lanes are still deployed — that is what makes this reversible and why it is worth doing as its own act rather than inside a deploy.

Scope boundary

ENDS at: the three ids live in production, with a real push proven from the ledger and the two dormant supervisors proven ROUTED.

Changes only the DECLARATION. ⚠️ Amended 2026-08-27: this card originally said "Changes NO code", and that stopped being true when MOTIR-3716 made the lane census shipped code. The one permitted edit is moving the three ids between the two lists in lib/jobs/engine/census.ts and the tests and docs that read them. Anything else that is wrong still STOPS this card and is filed as a bug; it does not patch.

Does NOT remove anything from Inngest. The SDK, the serve route and the dependency are MOTIR-3418's, and it is blocked_by this card precisely so the retirement cannot begin while a third of the fleet is still on the old lane.

Does NOT touch the fleet's spend controls — MOTIR_INDEX_MAX_IN_FLIGHT, codeGraphIndexAdmissionService, lib/ciFleet/limits.ts. Step 6 OBSERVES them; it does not adjust them.

Acceptance criteria

  • The three ids are named in MIGRATED_TO_ENGINE in lib/jobs/engine/census.ts on main, DELIBERATELY_ON_INNGEST is EMPTY, and that commit is in the RUNNING release — checked in the image, not in main.
  • fly secrets list -a motir-core shows MOTIR_POSTGRES_JOB_IDS updated, and the value read back from inside a machine contains every id it carried before plus these three, by name — a comm against the recorded baseline, with nothing dropped. ⚠️ No literal count. The card originally said "all 37 job ids — the 20, the 14 and these 3"; the baseline was 38 on 2026-08-27 (MOTIR-3682, MOTIR-3688 and MOTIR-3709 landed after this card was written), making the target 41, and any figure written here will be stale again the next time a job lands. The set is the criterion; the number is not.
  • reconcileLanes() reports in_sync — the declaration and the live secret agree — read from a system.daily-health-check job_run row rather than computed by hand.
  • Both the app and worker process groups are running a release that carries the new value, read from the platform's own API rather than from fly.toml or a dashboard screenshot.
  • A real default-branch push produces exactly ONE system.code-graph-refresh job_run, succeeded, on lane = engine, carrying one output.repoRef, and TWO pushes inside the debounce window still produce one.
  • system.ci-runner-boot and system.code-graph-index are proven ROUTED — present in the live secret and reported so by the switch — with their execution counts recorded on this card as measured, NOT observed running. Neither can be exercised by waiting.
  • Concurrent index containers over the following day are within the range they occupied before the flip, checked rather than assumed.
  • The Inngest dashboard shows no new runs for these three function ids after the flip — the half that proves nothing is running on BOTH engines.
  • Anything unexpected is a filed bug with its evidence, and the ids are rolled back rather than left half-flipped.

Context refs

  • lib/jobs/engine/cutover.ts — MOTIR_POSTGRES_JOB_IDS, and the default-to-Inngest safety property that makes rollback trivial
  • lib/jobs/engine/census.ts — the DECLARATION half (step 0) and reconcileLanes(); docs/jobs.md carries the two-edits-plus-a-read-back procedure
  • fly.toml — app = "motir-core", and the app / worker process groups this value must reach
  • /settings/workspace/jobs — the ledger surface these runs appear on
  • lib/services/codeGraphIndexAdmissionService.ts · lib/ciFleet/limits.ts — the spend controls step 6 observes and must not adjust
  • MOTIR-3463 · MOTIR-3475 — the two sibling flips that write the same variable, and the pattern this follows
  • MOTIR-3425 — the worker machines these runs execute on, already provisioned