MotirBuilding in public
MOTIR · moooon
onMotir
You’re viewing a public project. Anyone can view it — no account needed. Sign in to submit, upvote, or comment on requests.View-only — you can’t edit work items
MOTIR-3684

(motir-core) `system.code-graph-refresh` fails ~32% of the time — 62 of 194 runs in 7 days die on `admission_deferred: index admission was refused for 60 attempts`

Done
Description

Found while reading the ledger during MOTIR-3475. Unrelated to the cutoversystem.code-graph-refresh is one of the three container-supervisor ids deliberately left on Inngest (MOTIR-3489's), so nothing in this epic has touched it.

The measurement

job_run, 7 days to 2026-08-27, read on DATABASE_URL_UNPOOLED:

succeeded  n=132   last 2026-08-26T23:04:40Z
failed     n=62    last 2026-08-27T01:21:01Z
running    n=3     last 2026-08-27T07:20:38Z   (see MOTIR-3683 — these never resolve)

62 of 194 completed runs failed — 32%. Every sampled failure carries the same error:

IndexDispatchFailedError: Indexing moooon-B-V/motir-core into project cmqfb4d8q000e2d0i6n62otyc
failed (admission_deferred): index admission was refused for 60 attempts

Sampled failures on 2026-08-26 at 15:49, 17:20 and 17:50 — all moooon-B-V/motir-core or motir-meta, all admission_deferred, all after 60 attempts. The count being exactly 60 every time says this is a retry ceiling being hit, not a transient.

What it means, stated carefully

admission_deferred means the index admission gate refused the work repeatedly rather than erroring — the job asked 60 times and was told "not now" 60 times. That is a saturation or stuck-lease shape, not a crash. Two candidate readings, both consistent with the data and NOT distinguished here:

  • Genuine capacity: concurrent index admissions are capped, and the refresh cadence exceeds what the cap admits, so a third of attempts age out.
  • A stuck claim: something holds an admission slot it never releases, so the gate is permanently near its ceiling and callers starve.

The 32% is the point, whichever it is. A third of code-graph refreshes are not happening, which means the index silently lags the repository — and the code graph is what codegraph_explore reads. A stale graph does not error; it answers questions about code that has changed.

⚠️ How this relates to the OTHER indexing card — do not merge them on the noun

MOTIR-3606 reports the daily health check failing with IndexFleetImageUnpullableError on the INDEXER image. Both mention indexing, and it is tempting to treat them as one fault. The evidence says otherwise, and merging them would produce one card wrong about both:

  • Different errors, at different stages: Unpullable at the boot preflight vs admission_deferred at dispatch.
  • code-graph-refresh succeeds 132 times. An unpullable image does not obviously permit that.
  • dailyHealthCheck.ts:147 states outright that the probe "pulls a different image" from other paths.

They may share a root cause. Establish it with evidence rather than by assuming.

Acceptance criteria

  • The admission gate's configured ceiling and its current occupancy are read from the running system and stated — the number, not a description.
  • Whether the 60 refusals are capacity or a stuck/leaked admission slot is settled, with the query or log that settles it.
  • The failure rate over a 7-day window after the fix is below a stated threshold, measured the same way (job_run, unpooled, succeeded vs failed).
  • The staleness this caused is quantified: for at least one affected repo, how far behind the graph fell while a third of refreshes were failing.
  • The relationship to MOTIR-3606 is stated as shared-cause or independent, with evidence either way.

Context refs

  • lib/services/codeGraphIndexService.ts / lib/github/indexEnqueue.ts — the dispatch path that reports admission_deferred
  • lib/jobs/definitions/codeGraphRefresh.ts — the job, its debounce (2m / 15m timeout) and its idempotent retry policy
  • MOTIR-3606 — the other indexing failure; probably independent
  • MOTIR-3683 — why the 3 running rows above never resolve
  • MOTIR-3489 — the card that will eventually route this job to the engine; it should not inherit a 32% failure rate