MotirBuilding in public
MOTIR · moooon
onMotir
You’re viewing a public project. Anyone can view it — no account needed. Sign in to submit, upvote, or comment on requests.View-only — you can’t edit work items
MOTIR-3457

(motir-core) The engine's fast-lane latency probe — event to run p95, read from the ledger instead of a vendor API

Done
Description

Build the instrument that can measure the fast lane's event-to-run latency on the Postgres engine, so the figure in lib/jobs/latencyBudget.ts can be replaced by a reading rather than by an estimate.

Why a new script rather than an edit

scripts/experiments/inngest-fastlane-lag.mjs produced the baseline recorded on FAST_LANE_LATENCY_BUDGET.baseline (2026-08-23, 72 h, n=556, median ≈1.3 s, p95 ≈29.4 s, max 93.3 s). It reads Inngest's production REST API, which the engine does not have and will not grow. The successor reads our own ledger, so it is a different program with the same output contract — and both stay in the tree until MOTIR-3418 retires the old lane, because during the migration the two lanes are measured side by side and that comparison is the whole point.

What it measures

Event arrival → run start, for the consumers of FAST_LANE_LATENCY_BUDGET.events, over a window.

⚠️ Determine the join from the code, not from the column names. job_queue carries event_id and job_run carries event_id, but the engine's ledger writes job_run through jobRunsService.recordStart and what it puts in that column on the ENGINE arm is a fact to read off lib/jobs/engine/ledger.ts, not to infer from the Inngest arm in defineJob. Read it, and say in the script's header which columns the latency is a difference of. If the correlation turns out not to be expressible, that is a finding worth reporting rather than working around — stop and report instead of substituting a proxy metric, because a number measured against the wrong pair is worse than no number.

Report the same six fields the constant records, so the result drops into baseline without reinterpretation: measuredOn, windowHours, samples, medianMs, p95Ms, maxMs. Use the same percentile method as the Inngest script — if the two disagree on how p95 is computed, the comparison the budget rests on is not a comparison. Read that script and match it, or state the difference in the header.

How it will be run

Against production, from inside the deployment, since the ledger is the production database: fly ssh console -a motir-core -C '…', or an equivalent that keeps the credential inside the machine. The script therefore takes its connection from the environment and prints to stdout — no dashboard, no file, no network egress. It is a read-only script and must contain no write of any kind.

It also runs locally against a seeded database, which is how this card is accepted.

Scope boundary

ENDS at: a script that runs and reports, plus a unit test for its percentile arithmetic.

Does NOT take the production reading and does NOT edit lib/jobs/latencyBudget.ts. Both belong to the re-measurement task, which runs after the cutover — a figure read before any job has moved would measure the old lane through the new instrument and mean nothing.

Does NOT delete scripts/experiments/inngest-fastlane-lag.mjs.

Acceptance criteria

  • scripts/experiments/engine-fastlane-lag.mjs runs against a database and prints the six fields above, plus the consumer ids it covered and the window it used.
  • Its header states, in one sentence, the exact pair of columns whose difference it reports, and cites where in lib/jobs/engine/ledger.ts that was read.
  • Run against a seeded database with known timestamps, it returns the arithmetically correct median, p95 and max — asserted by a test over the percentile function with a fixture whose expected values are computed by hand, not by the function under test.
  • It reports samples: 0 and exits cleanly on an empty window rather than dividing by zero or printing NaN. An empty read must be legible as "no data", never as a latency of zero.
  • It performs no write: no INSERT, UPDATE, DELETE or TRUNCATE, asserted by reading the file in the test.
  • The percentile method matches inngest-fastlane-lag.mjs, or the header says how it differs and why.

Context refs

  • scripts/experiments/inngest-fastlane-lag.mjs — the predecessor, its output shape and its percentile method
  • lib/jobs/latencyBudget.tsFAST_LANE_LATENCY_BUDGET.baseline, the six fields to fill, and FAST_LANE_CONSUMER_IDS
  • lib/jobs/engine/ledger.ts — what the engine writes to job_run, and with which event id
  • prisma/schema.prismaJobEvent.receivedAt, JobQueueRun, JobRun.startedAt and their indexes
  • docs/decisions/job-lane-occupancy.md §6 — the burstiness finding the new reading is measured against