MotirBuilding in public
MOTIR · moooon
onMotir
You’re viewing a public project. Anyone can view it — no account needed. Sign in to submit, upvote, or comment on requests.View-only — you can’t edit work items
MOTIR-3468

(motir-core) Decide the CATCH-UP POLICY for a missed tick — per job, and where the disposition is DECLARED

Done
Description

Decide what happens to a scheduled tick the worker was down for — per job, and totally — and record it as an amendment to docs/decisions/job-queue-foundation.md.

The question, and why nothing answers it today

Inngest answered it for us and it never had to be written down. Owning the scheduler means owning the answer.

Verified on origin/main@a26d500d: lib/jobs/engine/registry.ts exports engineScheduledJobs() under the comment "The scheduled story (MOTIR-3416) consumes this", and git grep engineScheduledJobs returns its own definition plus tests/jobs/engine-units.test.ts:86 and nothing else. dispatchEventToEngine enqueues for EVENTS only. There is no scheduler, so there is no catch-up behaviour to inherit, discover or measure — this is a policy being authored, not one being read back.

Why one global rule is the wrong shape

The 14 jobs' cadences span once a minute to once a month:

  • system.ci-runner-provision-sweep is * * * * *. A worker down for six hours has missed 360 fires. Running them is absurd; running the newest one is exactly right.
  • system.ci-minutes-reconcile is 0 4 3 * * — monthly. A worker down across the 3rd has missed the only fire that month, and skipping it means the reconciliation does not happen at all until September.
  • system.attachment-gc is 30 3 * * * and cursor-bounded per run. A missed night is a night of orphan blobs; catching up costs one extra sweep.

So the deliverable is a table, not a sentence.

Two things the decision must NOT conflate

  1. retryPolicy: 'idempotent' is not a catch-up licence. attachmentGc, rateLimitSweep and ciMinutesReconcile declare it, and it says a handler may safely run the SAME tick twice. It says nothing about whether a tick that is now six hours stale is still worth running. A sweep can be perfectly idempotent and still be pointless (or harmful) when replayed late — system.ci-runner-provision-sweep provisions against a fleet state that has since moved.
  2. "Catch up" is under-specified until you say HOW MANY. Three distinct answers: run every missed fire in order, run only the most recent missed fire, or run nothing. Name which one each job gets; "catches up" alone is the ambiguity this card exists to remove.

The second question, and it is the one that decays: WHERE the disposition is declared

A policy recorded only in an ADR is a policy a new cron job silently opts out of. This codebase has already solved that shape twice and recorded the argument both times — lib/jobs/schedules.ts ("a hand-maintained array is a second source of truth that a new job forgets to join") and lib/jobs/engine/registry.ts ("Two lists drift; one list cannot").

Recommendation, to be confirmed or overturned in the amendment: a catchUp field on DefineJobOptions, carried onto EngineJobDefinition by registerEngineJob, and typed so that a definition supplying cron cannot omit it. A default would let a new cron job inherit a disposition nobody chose for it, which is the failure mode in miniature. The rejected alternative is a Map<jobId, policy> beside the scheduler — cheaper to write and a second list by construction.

Scope boundary

ENDS at: the amendment merged. Writes no lib/, prisma/ or tests/ code — the option and the 14 declarations are the declaration card, and the tick that reads them is the scheduler card.

Also settles the lib/jobs/schedules.ts import caveat for this story. MOTIR-3455 names that file as "MOTIR-3416's problem in the same shape". It is NOT the same shape here and the amendment should say so rather than leave a reader to re-derive it: the emit path is a Next.js request that imports no definition module, whereas the scheduler runs in scripts/worker.ts, which imports @/lib/jobs/registry for its side effect. Only two non-test files import that module — app/api/inngest/route.ts and scripts/worker.ts (git grep -l '@/lib/jobs/registry' -- '*.ts' '*.tsx') — and the scheduler lives in one of them.

Acceptance criteria

  • The amendment is a NEW numbered section in docs/decisions/job-queue-foundation.md. Run grep -nE '^## §' docs/decisions/job-queue-foundation.md first and take the next number — a sibling PR may have added one. At authoring time the last was ## §10 — The risk this decision accepts, named.
  • It carries a table with one row per scheduled job id, and the row set is derived from engineScheduledJobs() / jobSchedules() rather than transcribed — a job present in the registry and absent from the table is the defect this card exists to prevent.
  • Each row names one of exactly three dispositions: run every missed fire, run only the most recent missed fire, or skip. Not "catches up".
  • It states, in one sentence per job class, the reason — and that reason is about STALENESS, never about retryPolicy.
  • It names where the disposition is DECLARED, and why not a second hand-maintained map, citing the completeness-by-construction argument lib/jobs/schedules.ts and lib/jobs/engine/registry.ts already record.
  • It states what a caught-up run carries: which fire time is written as the run's scheduled instant, and whether a handler can tell it is late.
  • It treats system.daily-health-check explicitly. That job runs jobScheduleHealthService, whose own tolerance already forgives exactly one missed tick — so a skip disposition on it interacts with the tripwire that would report the skip, and the amendment must say which way that goes.
  • It states what the schedule-health probe means once a job is on the engine: it currently detects a stale INNGEST registry, and for a migrated job the same silence means a dead worker or a stalled scheduler instead. The probe is unchanged; its meaning is not.
  • No file under lib/, prisma/ or tests/ is modified by this card.

Context refs

  • docs/decisions/job-queue-foundation.md — §4 / §8, which name the dispatcher and the registry without deciding this
  • lib/jobs/engine/registry.tsengineScheduledJobs(), and the two-lists-drift argument
  • lib/jobs/schedules.ts — the self-registering schedule table and the same argument
  • lib/jobs/cron.tspreviousFireAtOrBefore, which is how "which fires were missed" is computed
  • lib/jobs/retries.ts — the three named retry policies, and what they actually promise
  • lib/services/jobScheduleHealthService.ts — the one-missed-tick tolerance, and the judge() function that encodes it
  • lib/jobs/definitions/ — the 14 definitions carrying cron:, listed with their cadences on the parent story