MotirBuilding in public
MOTIR · moooon
onMotir
You’re viewing a public project. Anyone can view it — no account needed. Sign in to submit, upvote, or comment on requests.View-only — you can’t edit work items
MOTIR-3913

(motir-core) With Vitest balanced, E2E becomes CI's ceiling at ~15.5 min — five bulk legs carry 1705 s of test work behind a 178 s per-leg boot nobody has costed

In Review
Description

Type · chore (CI configuration) · Parent · MOTIR-1464 · Repo · motir-core · Measured · run 33251966134 (PR #2453) · Pairs with · MOTIR-3912 — neither card alone takes CI below ~15.5 min

Why this is worth doing NOW and was not worth doing last week

Until MOTIR-3902 the Vitest lane ran 24.8 min and E2E's 11.3 was invisible behind it. On run 33251966134 the two lanes are 43 seconds apart:

12:28:54  Playwright E2E (bulk-3) ends
12:29:37  Vitest (6/8) ends
12:31:07  CI complete                → 17.7 min

MOTIR-3912 balances Vitest to ~10 min per leg, which lands Vitest + coverage around 12:26 and makes E2E the binding constraint at ~15.6 min (changes 0.5 + build 4.4 + bulk-3 10.9). This card moves that constraint. Doing it WITHOUT MOTIR-3912 buys nothing at all — Vitest's leg 6 would still end last.

Where an E2E leg's 10.9 minutes actually goes

Read off bulk-3's job log, which is the only place the split is visible:

phasetime
checkout + Postgres + e2e-setup (job steps)83 s
inside the Run E2E step, before the first test178 s
test execution (Running 99 tests using 1 worker 12:22:21 → 99 passed 12:28:50)389 s
job total650 s (10.9 min)

The 178 s is the dual-next start webServer boot plus seeding, and it is re-paid by every leg. Together with the job steps that is 261 s of fixed cost per leg against 389 s of work — the reason ci.yml's own comment stopped at five legs and said "more legs past this give diminishing returns for the cost". That comment is right about the SHAPE and has never been given the number; this card supplies it.

Total test work across the five legs, netting out the boot: (459−178) + (522−178) + (567−178) + (509−178) + (538−178) = 1705 s.

legstest/legstepjobfixed cost total
5 (today)341 s519 s602 s avg (650 s worst)1305 s
8213 s391 s474 s (7.9 min)2088 s
10170 s348 s431 s (7.2 min)2610 s

8 is the knee. Going 5 → 8 takes ~3 min off the slowest leg for ~13 extra runner-minutes; 8 → 10 buys 0.7 min more for another 9. Below ~8 legs the boot dominates and the curve flattens — which is what "diminishing returns" meant, now with a number attached.

E2E path after this card: changes 0.5 + build 4.4 + leg 7.9 ≈ 12.3 min, against a balanced Vitest path of ~11.9. The two lanes land level, which is the right place to stop.

The recorded costs have DRIFTED and must be re-measured in the same change

SPEC_COST_SECONDS in tests/e2e/shard-plan.ts is sourced from two green runs of 2026-08-10 plus a handful of locally-measured additions. tests/e2e-shard-plan.test.ts asserts the legs balance to within 15% of those recorded numbers — which stays green while reality drifts, because the guard measures the table against itself.

It has drifted. Net of the 178 s boot, the five legs ran 281 / 344 / 389 / 331 / 360 s — a 38% spread, not 15. Re-packing stale costs across 8 legs would carry that error into a finer split, where it hurts more. So: re-measure from the playwright-report-bulk-* artifacts of run 33251966134 (all five present, unexpired) by summing result.duration per spec file, and update the table in the same PR. Keep the file's provenance discipline — say which runs each number came from, and keep the local-measurement warnings intact for entries this run does not cover.

Acceptance criteria

  1. BULK_LEG_IDS is eight legs; ci.yml's e2e matrix lists the same eight, and tests/e2e-shard-plan.test.ts's ci.yml cross-check passes against them.
  2. SPEC_COST_SECONDS is re-measured from run 33251966134's five playwright-report-bulk-* artifacts, with the provenance note updated to name that run and date. Entries that run does not cover keep their existing value AND their existing provenance warning — do not silently re-stamp a number this measurement did not take.
  3. On this PR's own run the eight legs' test-execution time (step time minus each leg's own pre-test window, read from its Running N tests line) spreads under 1.25x, and no bulk leg's job exceeds ~9 min. Quote the eight readings.
  4. Every spec still runs exactly once: summed N passed across the eight legs equals what the five legs reported on run 33251966134, adjusted for specs the diff adds or removes. The existing "assigns every spec to exactly one leg" guard covers the mechanism; the count is the receipt.
  5. tests/ci-job-timeouts.test.ts stays green, and e2e's timeout-minutes: 45 is revisited against the new per-leg readings — same rule as MOTIR-3569: a ceiling bounds a hang, never a slow-but-healthy run, so it may stay high, but its comment must not keep quoting a five-leg membership.

Out of scope — and one of them is the bigger prize

  • ⚠️ The 178 s pre-test boot itself. It is now 45% of a leg and, at eight legs, 2088 s of runner time per run — more than the test work it protects. Halving it would beat this card. It is out of scope because it is a different investigation (dual next start vs. seeding vs. migrate deploy, none of them separated yet by measurement) and because sizing the legs first is what makes the boot's share legible. File it as its own card once this lands — do not fold it in.
  • The build job (4.4 min), which every leg waits on.
  • e2e-at-scale and the @a11y legs — the a11y pair already runs 5.6–5.9 min and is not near the ceiling.

Context refs

  • tests/e2e/shard-plan.tsBULK_LEG_IDS, SPEC_COST_SECONDS, assignBulkLegs; the packer is already total and needs no change beyond the leg list.
  • tests/e2e-shard-plan.test.ts — the guard, including its ci.yml matrix cross-check and the 15%-of-recorded-cost balance assertion.
  • playwright.config.ts — consumes E2E_SHARD via legTestMatch.
  • .github/workflows/ci.yml — the e2e matrix (five bulk-* entries sharing one --grep-invert).
  • Run 33251966134 — the measurement, and its playwright-report-bulk-{1..5} artifacts.

Resolution: open.