MotirBuilding in public
MOTIR · moooon
onMotir
You’re viewing a public project. Anyone can view it — no account needed. Sign in to submit, upvote, or comment on requests.View-only — you can’t edit work items
MOTIR-1545

E2E flake: an in-flight inngest job outlives its test's DB teardown → `recordSuccess` throws + FK-violation cascade crashes the WebServer mid-shard

Done
Description

Symptom (CI, intermittent). A Playwright E2E (bulk-*) shard goes red partway through: from some point on, every remaining test times out with page.goto: net::ERR_ABORTED; maybe frame was detached? (90s each), inflating the job to ~2–3× its normal wall-clock. First observed on PR #1495 bulk-1 (board-scrum.spec.ts:160/183/218 all failed twice), while the SAME specs passed on sibling PR #1492 and on main — so it is not a defect in those specs; it's a shared-server crash they happen to run after.

Root cause (from the WebServer logs). A background inngest job spawned by an earlier test is still in flight when that test's teardown truncates the DB between tests. When the orphaned job then completes:

  • lib/services/jobRunsService.ts:89 recordSuccess() reads the job_run row inside the tx and throws job_run <id> not found when recording success because teardown already truncated it (see also the sibling status-flip paths).
  • Concurrent job writes that reference now-truncated parents raise a cascade of DriverAdapterError: ForeignKeyConstraintViolation.

These unhandled rejections degrade the dev WebServer to where subsequent page.gotos abort — so every later test in the shard times out. Because it depends on job/teardown timing, it is non-deterministic and shard-position-dependent; under merge-with-main CI it can red-light unrelated PRs (per CLAUDE.md's flaky-test-is-a-release-blocker rule).

Repro signature (grep the failing job log)

  • [WebServer] Error: job_run <id> not found when recording success
  • driverAdapterError: Error [DriverAdapterError]: ForeignKeyConstraintViolation (repeated)
  • Test timeout of 90000ms exceededpage.goto: net::ERR_ABORTED; maybe frame was detached?

Example: run 28625407163, job Playwright E2E (bulk-1) (84890943046).

Fix direction (pick after a repro)

Two complementary angles; probably want both:

  1. Harden the job-completion path so a vanished job_run is a benign no-op, not a throw/unhandled-rejection — e.g. recordSuccess/recordFailure treat "row gone" as already-terminal and return quietly (an orphaned run from a torn-down test, or a genuinely deleted run, should never crash the process). Guards the server against ANY late-arriving job, not just tests.
  2. Order E2E teardown after in-flight jobs drain (or scope truncation so it can't run under an active job) so tests don't strand jobs mid-flight in the first place — the cleaner long-term fix for the test harness.

Acceptance criteria

  • The job-completion path never throws/emits an unhandled rejection when its job_run row is absent (unit-tested: call recordSuccess/recordFailure for a deleted run → no throw, no server-level rejection).
  • E2E teardown no longer strands in-flight inngest jobs across a truncation (or the stranded job is provably harmless) — the FK-violation cascade + ERR_ABORTED signature does not reproduce over N repeated bulk-1 runs.
  • One repo = motir-core.

Notes

  • Not caused by PR #1495 (a CI-config-only change: OIDC publish workflow + package.json repository field). That PR's bulk-1 was re-run and is expected green; this bug is logged as a pre-existing latent flake, not a blocker for it.
  • Related prior E2E-flake bug (different cause, resolved): MOTIR-989.