MotirBuilding in public
MOTIR · moooon
onMotir
You’re viewing a public project. Anyone can view it — no account needed. Sign in to submit, upvote, or comment on requests.View-only — you can’t edit work items
MOTIR-3606

(motir-core) The daily health check has been RED for 23 days and nobody noticed — `IndexFleetImageUnpullableError` every night since 2026-08-04, last success 2026-08-03

Done
Description

Found while running MOTIR-3475. Not caused by the cutover — the engine reproduced a pre-existing failure faithfully, which is how it surfaced.

The verified fact

system.daily-health-check (0 9 * * *) has failed every single day since 2026-08-04. Its last success was 2026-08-03. Read from job_run, all-time, on DATABASE_URL_UNPOOLED:

INNGEST  succeeded  n=27   2026-07-04 .. 2026-08-03      <- last success 23 days ago
INNGEST  failed     n=1    2026-08-04
?        failed     n=27   2026-07-08 .. 2026-08-26
ENGINE   failed     n=1    2026-08-26T19:01:30Z          <- the cutover, same error

Every recent failure carries the identical error and the identical digest:

IndexFleetImageUnpullableError: The fleet's INDEXER image cannot be pulled:
registry.fly.io/motir-index-runners@sha256:0b4d27470b7d83769febcea954dd4ac2af8dc0bd37861e123a57e4ddf4f6b9f0

Confirmed on 2026-08-21, 08-23, 08-24, 08-25, 08-26 (Inngest) and 08-26 (engine). The engine's run failed with byte-identical output to Inngest's, which is positive evidence for the cutover and is why this is filed separately from it.

⚠️ Why this is the bad one

The check that exists to tell you production is broken has itself been broken for 23 days, and its own failure is the thing nobody was watching. It is the alarm, and the alarm has been ringing into an empty room since the beginning of August.

That it works is not in doubt — on 2026-08-22 it caught something real and different:

ScheduledJobsOverdueError: 1 scheduled job(s) have not run since their previous tick:
system.ci-runner-provision-sweep (cron * * * * *; last run 2026-08-22T08:59:00.518Z). Suspect a stale In[ngest registry]

So the liveness probe genuinely detects overdue crons — exactly the failure mode a job-substrate cutover would produce. For 23 days that detector has been firing on a different fault and its verdict has been going unread. During MOTIR-3463 and MOTIR-3475 the safety net was down and nobody knew.

An adjacent observation — NOT diagnosed, do not assume a shared root cause

system.code-graph-refresh over the last 7 days: 159 succeeded, 55 failed — a 26% failure rate, with a different error:

IndexDispatchFailedError: Indexing moooon-B-V/motir-core into project cmqfb4d8q… failed
(admission_deferred): index admission was refused for 60 attempts

Two reasons not to merge these into one story without investigating: the errors differ (Unpullable at the boot preflight vs admission_deferred at dispatch), and refresh succeeds 159 times, which an unpullable image alone does not obviously permit. dailyHealthCheck.ts:147 says the probe "pulls a different image" from other paths, so the two may be genuinely independent. Establish which before treating this as one fault.

Fix direction

  1. Determine whether MOTIR_INDEXER_IMAGE's digest is actually unpullable today, from the platform — not from the error text, which is 23 days old and may describe a condition that has since changed. MOTIR-2006 made an unpullable digest fail loudly at admission, so this alarm is that mechanism working as designed; the defect is that nothing consumed the alarm.
  2. Either repair the digest/credential, or — if the fleet is intentionally unconfigured in this deployment — make that a green state. dailyHealthCheck.ts:204 already contemplates a deployment "that runs CI but has not wired MOTIR_INDEXER_IMAGE" as green; check why this one does not take that branch.
  3. The real remedy is that a red health check must reach a human. A daily probe whose only output is a job_run row nobody reads cannot do its job. MOTIR-1933 is open and adjacent ("real probes on the daily health check … so a silent lapse fails loudly") — decide whether this belongs there or stands alone.
  4. Diagnose the code-graph-refresh 26% failure rate separately and say whether it is related.

Acceptance criteria

  • The current pullability of MOTIR_INDEXER_IMAGE's digest is established from the registry/platform, and stated on this card either way.
  • system.daily-health-check either succeeds, or fails for a reason that is a real fault with a card against it — no third state where it is red and nobody is accountable.
  • A red daily health check produces a signal outside job_run that a person actually receives; the mechanism is named and demonstrated once.
  • The relationship (or independence) of the code-graph-refresh admission_deferred failures is stated with evidence.
  • The 23-day gap is explained: what would have had to exist for this to be noticed on 2026-08-05 rather than 2026-08-26.

Context refs

  • lib/jobs/definitions/dailyHealthCheck.ts:147, :204 — the indexer probe and the deliberately-green unconfigured branch
  • lib/orchestrator/imagePull.ts · lib/orchestrator/adapters/fly/indexImage.ts:32 — MOTIR_INDEXER_IMAGE and the pull path
  • lib/orchestrator/errors.ts:39 — verifyFleetBootable()'s fail-once-loudly contract
  • MOTIR-2006 — made an unpullable digest fail loudly at admission
  • MOTIR-1933 — open card for credential-expiry + scheduled-job liveness probes