(motir-core) The WORKER — its own Fly process group, claim with FOR UPDATE SKIP LOCKED, backoff, graceful shutdown

The process that actually runs jobs: claim due rows, execute them, record the outcome, repeat.

Where it runs

Its own Fly process group, declared in fly.toml beside the existing app group — not inside the web process. Job load and request serving then cannot contend for the same event loop, and the two scale independently. The [processes] entry is this card's deliverable; scaling it to a running count is an operator action and belongs to its own manual sibling.

The claim, and why it is the whole card

Claiming is a read-derived write: the loop reads which rows are due, then writes a claim based on what it read. Two workers doing that with a plain read-then-write both see the same row and both run the job. So the claim is:

SELECT … FROM job_queue
 WHERE state = 'pending' AND run_at <= now()
 ORDER BY run_at
 FOR UPDATE SKIP LOCKED
 LIMIT n

inside one transaction, with the state write in the same transaction. SKIP LOCKED is what makes a second worker take the NEXT row rather than block on the first.

A serial test cannot see this defect. Two sequential claims never collide; the race needs genuine concurrency against a warm pool, and that is what the acceptance criteria demand.

The rest

• Backoff between empty polls, with LISTEN/NOTIFY (or the chosen library's equivalent) so a freshly emitted event does not wait out a poll interval.
• A lease, so a run claimed by a worker that dies is reclaimable rather than stuck in running forever. The lease duration and its renewal are this card's decisions and belong in a comment.
• Graceful shutdown — on SIGTERM, stop claiming, let in-flight runs finish or checkpoint, release claims. A deploy is a routine event and must not orphan work.
• Side effects stay outside the claiming transaction. The transaction holds the claim write and nothing else; the job's own handler runs after it commits.

Acceptance criteria

• Two workers running concurrently against a real Postgres never execute the same job_queue row twice — asserted by a test that drives genuine concurrency and accepts every legitimate interleaving, not a serial one.
• A worker killed mid-run releases or expires its lease, and another worker picks the run up.
• SIGTERM drains rather than aborting: no run is left running with no live claimant after a shutdown.
• The [processes] worker group is declared in fly.toml; the card does NOT scale it.
• A newly emitted event is picked up promptly rather than at the next poll boundary.
• Unit and real-Postgres integration tests ship with the change.

Context refs

• fly.toml — the [processes] block, and min_machines_running as the precedent for how process groups are declared here
• prisma/schema.prisma — job_queue from the schema card
• motir-core/CLAUDE.md § concurrency — lock-before-a-contended-update and the warm-pool race
• lib/services/codeGraphIndexAdmissionService.ts — an existing FOR UPDATE claim in this codebase, for shape