MotirBuilding in public
MOTIR · moooon
onMotir
You’re viewing a public project. Anyone can view it — no account needed. Sign in to submit, upvote, or comment on requests.View-only — you can’t edit work items
MOTIR-3731

(motir-core) The engine has NO per-job concurrency, and a job that wanted one lost it silently — `account/data-export.requested` declared `{ limit: 1, key: userId }` and the engine never read the field

Done
Description

defineJob accepted a concurrency option and forwarded it to the vendor, which implemented it. The Postgres engine never read the field: JobWorker claims CLAIM_BATCH due runs per tick with SELECT … FOR UPDATE SKIP LOCKED and consults no per-job limit, and registerEngineJob does not even carry the option. So from the moment a job was routed to the engine, any concurrency it declared was an accepted-and-ignored field.

MOTIR-3418 removed the option rather than porting it, on the evidence that no job declared one — which was true when that work started and false by the time it merged: MOTIR-3701 landed account/data-export.requested with

concurrency: { limit: 1, key: 'event.data.userId' },

and the comment beside it — "an archive is a whole-account read, so two of them for one person running at once is the one shape worth serialising" — is a real requirement, stated deliberately.

What is NOT broken, and it matters to the priority

The serialisation that job wanted is still enforced, by the stronger of its two guards. dataExportService.requestDataExport takes findLatestByUserIdForUpdate and returns the existing request WITHOUT emitting when one is already preparing, so at most one build event per user is ever in flight. That is a row lock on the request, not an admission decision on the run — it cannot be defeated by a worker restart or a replay, which a scheduler-side limit can. The concurrency option was belt to that braces.

So this is not an incident. It is a capability the substrate advertised, quietly stopped honouring at the cutover, and has now been removed from the type — and the next job to want one will have nowhere to say so.

Acceptance criteria

  • The engine enforces a per-job concurrency limit at CLAIM time, keyed by an event expression, or docs/decisions/job-queue-foundation.md records the decision NOT to and says what a job should reach for instead (a request-time lock, an admission cap, a debounce).
  • If it is implemented: account/data-export.requested gets its declaration back, and a test drives two same-key runs against a real Postgres and asserts the second is not claimed while the first holds.
  • If it is declined: lib/jobs/definitions/dataExportBuild.ts's comment naming this bug is replaced by a pointer to the decision record.
  • Either way, defineJob does not accept an option nothing reads — the failure mode this bug is about.

Context refs

  • lib/jobs/definitions/dataExportBuild.ts — the declaration removed by MOTIR-3418, and the comment recording why
  • lib/services/dataExportService.tsrequestDataExport's FOR UPDATE, the guard that actually holds
  • lib/jobs/engine/worker.tsclaimDueRuns / CLAIM_BATCH, where a limit would have to live
  • docs/decisions/job-lane-occupancy.md §3 — the argument the removed option was measured against