MotirBuilding in public
MOTIR · moooon
onMotir
You’re viewing a public project. Anyone can view it — no account needed. Sign in to submit, upvote, or comment on requests.View-only — you can’t edit work items
MOTIR-3415

Cut the EVENT-triggered jobs over — fan-out, the fast lane's latency budget, and the notification / automation / email paths

Done
Description

Moves every EVENT-triggered job onto the engine, one job id at a time through the cutover switch. This is the story that actually fixes the defect MOTIR-3245 was filed about.

⚠️ AMENDED by motir plan 3415 (2026-08-25) — the cutover is NOT just configuration

This card was written believing its work was to add job ids to MOTIR_POSTGRES_JOB_IDS one at a time. Read on origin/main@b944dab5, the switch cannot currently move anything from a web request, and two of the criteria below had no implementation behind them. The three gaps are now this story's actual subtasks; the 20 job definitions still change by zero characters.

Two criteria that read on PRODUCTION state have also moved OUT of this story, to siblings under the epic — see What this story hands off below. Nothing here is descoped: it is re-homed where it can be discharged, per planning bug MOTIR-3429.

The three clauses in the title, and what each covers

  1. Fan-out — one emitted event reaching N subscribing jobs. The fast lane is the reason this matters: work-item/transitioned has FOUR consumers (status-derivation/transitioned, watcher-notify/transitioned, notification-fan-in/transitioned, automation-engine/transitioned), and their simultaneous dispatch is what exhausted a 5-slot account. The dispatcher itself shipped with MOTIR-3423 — including per-subscriber failure isolation and (event, job) enqueue idempotency. What did NOT ship is the dispatcher being able to SEE its subscribers from the process that emits.
  2. The fast lane's latency budgetlib/jobs/latencyBudget.ts states p95 <= 5 s and records a measured Inngest baseline of 29.4 s that the budget deliberately does NOT meet. This story builds the INSTRUMENT that can measure the new substrate; the reading itself, and the recording of it, are the epic-level sibling below.
  3. The notification / automation / email paths — the remaining event-triggered jobs, enumerated rather than gestured at (below).

The job set, COUNTED

lib/jobs/registry.ts registers 37 functions. 14 declare a cron and belong to MOTIR-3416. Of the 23 event-triggered remainder, three are the container supervisors (MOTIR-3417's: system.code-graph-index, system.code-graph-refresh, system.ci-runner-boot). The 20 that are this story's:

trigger eventconsumers
work-item/transitionedstatus-derivation/transitioned · watcher-notify/transitioned · notification-fan-in/transitioned · automation-engine/transitioned
work-item/comment.createdwork-item/comment.created (mention) · watcher-notify/comment.created · notification-fan-in/comment.created · automation-engine/commented
work-item/createdstatus-derivation/created · automation-engine/created · outward-bug-telemetry/created
work-item/mentionedwork-item/mentioned (mention) · notification-fan-in/mentioned
work-item/field.changedautomation-engine/field.changed
work-item/child-set.changedstatus-derivation/child-set-changed
work-item/derivation.requestedstatus-derivation/requested
work-item/embedding.requestedwork-item/embedding.requested
email.sendemail.send
filter-subscription/deliverfilter-subscription/deliver
system.billing-seat-syncsystem.billing-seat-sync

The original prose named roughly half of these. automation-engine/field.changed, both of status-derivation's child-set-changed / requested arms, outward-bug-telemetry/created, filter-subscription/deliver and system.billing-seat-sync were unlisted and are in scope — the first acceptance criterion is total over the set, so they always were.

What this story actually BUILDS — the three gaps

  1. The subscriber set is invisible on the emit path. dispatchEventToEngine derives subscribers from a Map that defineJob fills as each definition MODULE is evaluated. Exactly two files import lib/jobs/registry.ts: app/api/inngest/route.ts and scripts/worker.ts. A server action calling sendEvent imports neither, so engineSubscribers(name) returns [], nothing is enqueued, hasInngestSubscribers returns its safe-default true, and the migrated job's Inngest handler returns { skipped: 'routed-to-postgres-engine' }the event runs on neither lane.
  2. email.send's idempotency has no engine implementation. defineJob's idempotency option is forwarded to Inngest and dropped before registerEngineJob; EngineJobDefinition has no such field, and nothing reads the job_event.idempotency_key the dispatcher writes.
  3. Four emitters bypass sendEvent, so the switch is never consulted for them: lib/billing/seatSync.ts:25, lib/ciFleet/bootDispatch.ts:99, lib/jobs/definitions/ciRunnerFleet.ts:152, lib/github/indexEnqueue.ts:37,56.

Scope boundary

ENDS at: the cutover switch being able to move any of the 20 jobs correctly, proven by the story's vitest gate and its E2E against a real worker — with the merge deploying that capability.

HANDS OFF:

  • The production cutover itself — flipping MOTIR_POSTGRES_JOB_IDS and reading the ledger back — to the production cutover task, a sibling under the epic. It cannot be a child here: its steps only begin after this story's PR merges, and a child in that position deadlocks the container or is cascaded done unverified (MOTIR-3429, MOTIR-3153).
  • The re-measurement and the recorded figure to the re-measurement task, for the same reason — the number is a property of production.
  • Scheduled jobs to MOTIR-3416, the container supervisors to MOTIR-3417, all deletion to MOTIR-3418.

The emit-seam fix covers all four raw-send sites; only system.billing-seat-sync MOVES here. Routing system.code-graph-index / -refresh / ci-runner-boot through sendEvent is what makes them switchable at all — but their ids stay OUT of MOTIR_POSTGRES_JOB_IDS, so they keep running on Inngest until MOTIR-3417 moves them. Making a switch reachable is not throwing it.

Does NOT change the fan-out itself. Reducing four consumers to one would cut peak demand fourfold and is a genuinely good idea — but it is a change to our own design, not to the substrate, and folding it in here would make it impossible to tell which change moved the number.

No user-visible surface changes. /settings/workspace/jobs keeps the shape MOTIR-3424 gave it, reading the same ledger through the same DTOs. So this story plans no design subtask and no acceptance video — a non-UI story accepts on its tests.

Verification recipe

Close a parent work item with several children — the cascade that produced the original 18-20 s lag. Watch the board update; the derivation, the watcher notification, the bell entry and any automation rule all land promptly. Then send the same workspace invite twice and confirm one email.

Acceptance criteria

  • Every one of the 20 event-triggered jobs above is REACHABLE by the cutover switch: with its id in the routing set, an emit from an ordinary request path enqueues a job_queue row, and with the id absent it reaches Inngest exactly as today.
  • One emitted event reaches all its subscribers; a subscriber that throws does not prevent its siblings from running.
  • emailSend's idempotency key still dedupes a repeated send on the ENGINE — asserted by a test, not by inspection.
  • No consumer of the fast lane sits behind a per-function concurrency limit or a debounce — tests/jobs/fast-lane-latency-budget.test.ts still passes unchanged.
  • No emitter in the tree calls inngest.send() outside lib/jobs/, and a lint rule fails the build if one is added.

Context refs

  • lib/jobs/engine/dispatcher.ts · lib/jobs/engine/registry.ts · lib/jobs/engine/cutover.ts — the switch and the fan-out, and where they cannot see each other
  • lib/jobs/sendEvent.ts · lib/jobs/defineJob.ts — the two halves of the switch
  • lib/jobs/latencyBudget.ts · tests/jobs/fast-lane-latency-budget.test.ts — the budget and its guard
  • lib/jobs/definitions/emailSend.ts — the only idempotency user
  • docs/decisions/job-lane-occupancy.md §6 — the burstiness finding this story is expected to move