MotirBuilding in public
MOTIR · moooon
onMotir
You’re viewing a public project. Anyone can view it — no account needed. Sign in to submit, upvote, or comment on requests.View-only — you can’t edit work items
MOTIR-3593

(motir-core) The fast-lane latency probe reads the POOLED url and reports `samples: 0` — RLS hides every row, so it says "no fast-lane engine runs" while the engine is running the whole lane

Done
Description

scripts/experiments/engine-fastlane-lag.mjs — the probe that produces the six figures MOTIR-3463 owes and MOTIR-3464 transcribes into lib/jobs/latencyBudget.tscannot see the ledger when run the way its own header says to run it.

Reproduced, 2026-08-26T16:5xZ, on machine 7817663f103648

Same script, same 6-hour window, same machine, one env var apart:

### DATABASE_URL  (exactly as the script's HOW TO RUN block prescribes)
window:    2026-08-26T10:57:24.973Z → now  (6h)
consumers: automation-engine/transitioned, notification-fan-in/transitioned,
           status-derivation/transitioned, watcher-notify/transitioned
ledger rows matched: 0
samples: 0 — no fast-lane engine runs in this window. Nothing to report.
{ "samples": 0, "medianMs": null, "p95Ms": null, "maxMs": null }

### DATABASE_URL_UNPOOLED
ledger rows matched: 166
{ "samples": 79, "medianMs": 254, "p95Ms": 1646, "maxMs": 2077 }
median 0.3s · p95 1.6s · max 2.1s

During that window 56+ fast-lane runs succeeded on the engine, individually quoted on MOTIR-3463.

Root cause — scripts/experiments/engine-fastlane-lag.mjs:161

const connectionString = process.env.DATABASE_URL;

DATABASE_URL is the pooled Neon url. Inside the machine it connects as motir_app, which has rolbypassrls = false; job_run, job_event and job_queue are all relrowsecurity = true and relforcerowsecurity = true, and a plain pg client sets no tenant GUC. So every policy matches nothing and the join returns the empty set — a successful query, no error, no warning.

pooled   (motir_app)     select count(*) from job_run  →       0
unpooled (neondb_owner)  select count(*) from job_run  → 198,540

DATABASE_URL_UNPOOLED is present in the machine's environment and connects as neondb_owner.

⚠️ Why this is worse than an empty result

The script is well built and that is precisely what makes this dangerous. It deliberately refuses to report a latency of zero:

samples: 0 — no fast-lane engine runs in this window. Nothing to report.

That line is correct behaviour and a false statement about the world. An operator running the documented command mid-cutover reads "no fast-lane engine runs" and concludes the cutover failed — the opposite of the truth, arrived at through the script's own good-faith guard. The 0-vs-null discipline protects against a wrong NUMBER; nothing protects against a wrong VERDICT.

This is the same trap as MOTIR-3227, where a sweep script run on DATABASE_URL per its own USAGE header printed "Scanned 0 design_evidence row(s) … ✓" and exited 0 with 42 rows in the table. Second instance, different script, identical mechanism.

A second, independent defect in the same header

The HOW TO RUN block also prescribes:

fly ssh console -a motir-core -C 'node scripts/experiments/engine-fastlane-lag.mjs --hours 72'

scripts/ is not in the deployed image — the runtime is a Next standalone output:

$ ls /app/scripts
ls: cannot access '/app/scripts': No such file or directory

So the documented command fails outright before the connection string ever matters. The file has to be uploaded into the machine (base64 -d > /app/engine-fastlane-lag.mjs) — and it must keep that basename, because the script self-guards on it:

if (process.argv[1] && process.argv[1].endsWith('engine-fastlane-lag.mjs')) { await main(); }

Saved under any other name it exits 0, silently, printing nothing.

Fix direction

  • Read process.env.DATABASE_URL_UNPOOLED ?? process.env.DATABASE_URL, matching the recipe every other ad-hoc production read in this project already uses. One line.
  • Better: make the blind case impossible to mistake for an empty one. Before the measuring query, assert what the connection can see — SELECT current_user, (SELECT rolbypassrls FROM pg_roles WHERE rolname = current_user) — and refuse to report when the role cannot bypass RLS on a force-RLS table, naming the reason. samples: 0 should be reachable only from a genuinely empty window.
  • Correct the HOW TO RUN block: the upload step, the required basename, and the unpooled url.
  • The same DATABASE_URL read exists in scripts/experiments/inngest-fastlane-lag.mjs — check it; it queries the Inngest API rather than the ledger, so it may be unaffected. Say which, on this card.

Acceptance criteria

  • Run from inside a machine with no env override, the script reports a non-zero ledger rows matched over a window containing known fast-lane engine runs — quoted in the PR body against a window whose runs are independently listed.
  • A connection that cannot see the ledger produces a NAMED refusal, not samples: 0 — proven by a deliberate negative (force the pooled url and show the refusal).
  • The HOW TO RUN block is executable verbatim against the deployed image, basename requirement included.
  • tests/jobs/engine-fastlane-lag.test.ts stays green, and its assertion that the file contains no write is untouched.
  • The status of inngest-fastlane-lag.mjs is stated either way.

Context refs

  • scripts/experiments/engine-fastlane-lag.mjs:161 — the read; the HOW TO RUN block is at the top of the same file
  • tests/jobs/engine-fastlane-lag.test.ts — the read-only guard that must stay green
  • scripts/detect-stray-design-results.mjs — the prior instance of this exact failure
  • MOTIR-3457 — the subtask that authored the probe
  • MOTIR-3464 — the card that transcribes the figures, and the one most exposed to a wrong reading