MotirBuilding in public
MOTIR · moooon
onMotir
You’re viewing a public project. Anyone can view it — no account needed. Sign in to submit, upvote, or comment on requests.View-only — you can’t edit work items
MOTIR-3416

Cut the 14 SCHEDULED jobs over — crontab, catch-up after downtime, and the sweeps that keep the platform tidy

Done
Description

Moves the 14 SCHEDULED jobs onto the engine. These are the sweeps that keep the platform tidy — nobody watches them, and that is exactly why their failure mode is quiet.

⚠️ AMENDED by motir plan 3416 (2026-08-25) — the count was wrong, and the scheduler does not exist

Read on origin/main@a26d500d, this card was wrong in three ways. The second changes what the story IS.

1 · It is 14, not 13, and the enumeration was wrong in both directions. system.billing-seat-sync declares NO cron (lib/jobs/definitions/billingSeatSync.ts:20) — it is event-triggered from lib/billing/seatSync.ts:25, and MOTIR-3415 already counts it among its 20. Meanwhile system.ci-runner-provision-sweep and system.ci-runner-reap DO declare one (lib/jobs/definitions/ciRunnerFleet.ts) and were unlisted. The corrected set is the table below, and it closes the epic's partition exactly: 14 cron + 20 event + 3 supervisors = the 37 functions lib/jobs/registry.ts registers.

2 · The engine has no scheduler at all — this story BUILDS one. lib/jobs/engine/registry.ts exports engineScheduledJobs() with the comment "The scheduled story (MOTIR-3416) consumes this", and git grep engineScheduledJobs returns its definition plus one test (tests/jobs/engine-units.test.ts:86) and nothing else. dispatchEventToEngine enqueues for EVENTS only. Nothing on the engine turns a cron expression into a job_queue row, so this is not a configuration change with an unanswered policy question attached — it is a component that does not exist.

3 · "Two workers do not double-fire a tick" needs a MIGRATION. The enqueue idempotency the engine has is job_queue's @@unique([eventId, jobId]), and the schema's own comment says a scheduled run carries a NULL event_id and "is therefore not constrained" — in Postgres a NULL never equals a NULL. Two workers ticking the same minute would each insert a row and the job would run twice. The criterion is right; it needs a per-tick key to stand on.

What is unchanged: the 14 job definitions still change by zero characters of behaviour, and the cutover switch's cron half already works — defineJob's handler returns { skipped: 'routed-to-postgres-engine' } for a routed id, so a migrated cron job cannot run on both engines.

What moved OUT: the production flip, to an epic-level sibling — see What this story hands off.

The 14, counted

Every job in lib/jobs/definitions/ declaring a cron: option. Each declares its schedule as a named constant rather than a literal, which is what makes them enumerable.

job idcrondeclared in
system.abandoned-plan-sweep10 * * * *abandonedPlanSweep.ts
system.attachment-gc30 3 * * *attachmentGc.ts
system.auto-plan-cadence-tick20 * * * *autoPlanCadenceTick.ts
system.automation-retention-sweep15 4 * * *automationEngine.ts
system.ci-actions-gate-sweep30 * * * *ciActionsGateSweep.ts
system.ci-minutes-reconcile0 4 3 * *ciMinutesReconcile.ts
system.ci-runner-provision-sweep* * * * *ciRunnerFleet.ts
system.ci-runner-reap7,17,27,37,47,57 * * * *ciRunnerFleet.ts
system.code-graph-offboard-sweep45 4 * * *codeGraphOffboardSweep.ts
system.daily-health-check0 9 * * *dailyHealthCheck.ts
system.filter-subscription-tick0 * * * *filterSubscriptionTick.ts
system.migrate-onboarding-sweep7,22,37,52 * * * *migrateOnboardingSweep.ts
system.plan-target-lock-sweep*/10 * * * *planTargetLockSweep.ts
system.rate-limit-sweep10 4 * * *rateLimitSweep.ts

The cadences span once a minute to once a month, and that range is the reason the catch-up question below has no single answer.

The three clauses in the title

  1. Crontab — schedule parsing and the tick that enqueues due runs. The parsing half already exists and is not to be rewritten: lib/jobs/cron.ts is a UTC evaluator with parseCron / previousFireAtOrBefore, built for the schedule-health probe and covered by tests/jobs/cron.test.ts. What is missing is a TICK that calls it, and a key that stops two workers acting on the same answer.
  2. Catch-up after downtime — the question Inngest answered for us and nobody had to think about: when the worker was down across a scheduled fire, does the job run late, or is that tick skipped? This is a DECISION this story must make explicitly and record, per job — a missed system.attachment-gc should probably catch up, a system.ci-runner-provision-sweep that missed 600 fires must not run 600 times, and silently choosing either is how a sweep stops running for a week unnoticed.
  3. The sweeps themselves — routed through the cutover switch, whose cron half is already built.

Scope boundary

ENDS at: all 14 jobs firing on the Postgres engine on their declared cadence, proven by the story's vitest gate and its E2E against a real worker, with the catch-up policy recorded — and the merge deploying that capability.

HANDS OFF:

  • The production flip itself — adding the 14 ids to MOTIR_POSTGRES_JOB_IDS and reading each cadence back — to the epic-level operator task Cut the 14 SCHEDULED jobs over IN PRODUCTION. It cannot be a child here: its first step needs this story's merge, and a post-deploy child in that position either deadlocks the container or is cascaded done unverified (MOTIR-3429, MOTIR-3153). That task is sequenced AFTER MOTIR-3463 because both write the same single environment variable.
  • Deletion of the Inngest SDK to MOTIR-3418; the three container supervisors to MOTIR-3417; the 20 event-triggered jobs to MOTIR-3415.

Does NOT change any sweep's own logic, thresholds or schedule — only which engine fires it. The schedule CONSTANTS are unchanged, and a test asserts it.

A named seam, not a dependency. system.ci-runner-provision-sweep emits system.ci-runner-boot (lib/jobs/definitions/ciRunnerFleet.ts:152), and that consumer is MOTIR-3417's and stays on Inngest until then. The seam is already safe in both directions: hasInngestSubscribers defaults to the old lane for an id that is not routed, so a scheduled job running on the engine can emit to a consumer that is not. This story adds no edge to MOTIR-3417 and must not introduce one.

No user-visible surface changes. /settings/workspace/jobs keeps the shape MOTIR-3424 gave it, reading the same ledger through the same DTOs. So this story plans no design subtask and no acceptance video — a non-UI story accepts on its tests.

Verification recipe

Open /settings/workspace/jobs and filter to a scheduled job. Confirm run rows appear on the expected cadence. Stop the worker process for longer than one interval, restart it, and confirm the recorded catch-up policy is what actually happens.

Acceptance criteria

  • All 14 scheduled jobs are REACHABLE by the cutover switch: with an id routed, a worker tick enqueues a job_queue row at that job's declared fire time; with the id absent, Inngest fires it exactly as today.
  • Their schedule constants are unchanged — asserted against the shipped values, not by inspection.
  • Two workers do not double-fire a tick — asserted by a test against a real Postgres, since this is a locking property a single-worker test cannot see.
  • The catch-up-after-downtime policy is decided and recorded as an amendment to docs/decisions/job-queue-foundation.md, is TOTAL over all 14 jobs, and is DECLARED in code beside each job's cron rather than in a second hand-maintained list.
  • docs/jobs.md § Scheduled jobs describes the substrate that actually fires the jobs, including what a missed tick does.
  • A scheduled run that throws still reaches the DLQ and is replayable.
  • The ledger row a scheduled engine run writes carries event_name = scheduled.<job_id>, so jobScheduleHealthService keeps judging these jobs with no change to it.

Context refs

  • lib/jobs/cron.ts — the UTC evaluator this story consumes rather than rewrites
  • lib/jobs/schedules.ts — the self-registering schedule table, and its import-completeness caveat
  • lib/jobs/engine/registry.tsengineScheduledJobs(), defined and uncalled
  • lib/jobs/engine/dispatcher.ts · lib/jobs/engine/worker.ts · scripts/worker.ts — the enqueue path the scheduler mirrors, and the process it runs in
  • lib/jobs/engine/cutover.ts — the switch; defineJob.ts's handler guard is its cron half
  • prisma/schema.prismaJobQueueRun, the (eventId, jobId) unique and its NULL-event note
  • lib/services/jobScheduleHealthService.ts — the probe that reads these jobs' ledger rows
  • docs/jobs.md § Scheduled jobs · docs/decisions/job-queue-foundation.md — the two documents this story amends