MotirBuilding in public
MOTIR · moooon
onMotir
You’re viewing a public project. Anyone can view it — no account needed. Sign in to submit, upvote, or comment on requests.View-only — you can’t edit work items
MOTIR-3683

(motir-core) One failed run writes TWO `job_run` rows and neither is legible — the `running` row never resolves and the failure lands on a second row with an unattributable `event_id`

Done
Description

Found while reading the ledger during MOTIR-3475. job_run is the surface every wave of this migration was verified against and the table /settings/workspace/jobs renders, so its integrity is load-bearing.

Two defects, one shape

1. Runs enter running and never leave. Measured 2026-08-27, status='running' and older than one hour:

system.daily-health-check         INNGEST  n=24   2026-08-02 .. 2026-08-26
system.code-graph-refresh         INNGEST  n=3    2026-08-20 .. 2026-08-27
system.ci-runner-reap             INNGEST  n=1    2026-08-02
system.ci-runner-provision-sweep  INNGEST  n=1    2026-08-04

Twenty-nine rows. The oldest has been running for 25 days. A daily job has left one behind every single day for three weeks.

2. The failure that ends those runs is written as a SEPARATE row, with an event_id that belongs to no lane.

system.daily-health-check         n=27   last 2026-08-26T09:01:52Z
system.ci-runner-provision-sweep  n=2
system.ci-runner-reap             n=1

These rows' event_id matches neither the engine's cuid (^c[a-z0-9]{24}$) nor Inngest's ULID (^[0-9A-HJKMNP-TV-Z]{26}$), so the lane discriminator that MOTIR-3467 established — and that every read-back in this epic depends on — cannot attribute them at all.

The pairing is exact and daily:

2026-08-26T09:01:16.943Z  INNGEST  system.daily-health-check  running    <- never resolves
2026-08-26T09:01:52.776Z  ?        system.daily-health-check  failed     <- 36s later, different event_id

One logical run; two rows; neither one tells the truth on its own.

Why it matters beyond tidiness

  • The operator surface lies in the safe-looking direction. /settings/workspace/jobs shows a job that failed 25 days ago as still running. "Running" reads as healthy-and-working, which is the worst available misreading of a dead job.
  • It defeated a check in this very migration. Every wave was read back by grouping job_run on the cuid/ULID lane discriminator. A row matching neither is invisible to that query. It happened not to matter here because the affected runs were pre-existing Inngest ones — but a ?-lane row is a run that no lane audit can see, and lane audits are how this whole epic is being verified.
  • It hid MOTIR-3606 in plain sight. A three-week-old daily failure presented as 24 perpetually-running jobs plus 27 unattributable failures. Neither shape says "this has been broken since 2026-08-04".

Not diagnosed — deliberately

I did not establish WHY the second row is written or why the first is never closed. Both plausible stories (a step-level retry record vs. a start row whose completion write is lost on a particular failure path) predict this data, and guessing between them would put a theory on the card instead of evidence. jobRunsService.recordStart / its completion counterpart, and how ledgerIdentity() derives event_id on the failure path, are where to start.

Note the affected runs are all INNGEST/?, none ENGINE — so this may be specific to the Inngest wrapper and may retire itself with MOTIR-3418. Establish that before investing in a fix; if it is Inngest-only, the correct outcome may be to backfill the stale rows and let the retirement remove the cause.

Acceptance criteria

  • The mechanism is named with evidence: which code path writes the second row, and which one fails to close the first.
  • Whether it can occur on the ENGINE lane is stated either way, with the reasoning — this decides whether the fix is worth building or whether MOTIR-3418 removes it.
  • No run can remain running indefinitely: either a completion write that cannot be lost, or a reaper that marks an abandoned run terminal with a distinguishable status. A run stuck in running must not render as healthy.
  • Every job_run row's event_id is attributable to a lane, or the ledger exposes the lane explicitly instead of leaving it to be inferred from the id format.
  • The 29 existing stale rows are dispositioned — backfilled or explained.

Context refs

  • lib/jobs/engine/ledger.ts — ledgerIdentity(), run.eventId ?? run.id
  • lib/services/jobRunsService.ts — recordStart and the completion path
  • lib/jobs/defineJob.ts — the Inngest wrapper's job-run:start step
  • app/(authed)/settings/workspace/jobs — the surface that renders these rows
  • MOTIR-3606 — the 23-day-old failure this shape concealed