MotirBuilding in public
MOTIR · moooon
onMotir
You’re viewing a public project. Anyone can view it — no account needed. Sign in to submit, upvote, or comment on requests.View-only — you can’t edit work items
MOTIR-1742

CI infra flake (3 PRs in a row): E2E legs die at `Initialize containers` when `docker pull postgres:16-alpine` can't reach Docker Hub — pre-pull/mirror the service image instead of re-running

Done
Description

The failure

A Playwright E2E leg fails at the Initialize containers step — the GitHub Actions service container bringup, which runs before actions/checkout. Consequences that make it unmistakable:

  • Every later step (checkout, pnpm install, prisma migrate deploy, playwright install, Run E2E) is reported skipped. Zero tests run, and the PR's diff is never even on disk — so the failure cannot possibly be caused by the change under test.
  • Job runtime is ~1 minute (vs. the usual 5–15m), and the artifact step warns No files were found with the provided path: out/playwright-report*.

The log signature is stable across all occurrences:

##[command]/usr/bin/docker pull postgres:16-alpine
Error response from daemon: Get "https://registry-1.docker.io/v2/":
  net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
##[warning]Docker pull failed with exit code 1, back off 5.205 seconds before retry.
   … (2 more attempts, one failing with `context deadline exceeded`) …
##[error]Docker pull failed with exit code 1

The runner's built-in 3-attempt retry with backoff is already exhausted inside the job, so Docker Hub (or the runner's egress to it) was unreachable for that ~45-second window. gh run rerun <id> --failed gets a fresh runner and clears it every time.

Why this is a bug now, not another re-run

Logged as a known flake after the 1st occurrence with an explicit escalation threshold ("if a 3rd lands, file it as a bug rather than re-running"). That threshold is now met — 3 occurrences on 3 consecutive PRs, all within ~24h, each on a diff that touches nothing the failing leg exercises:

#DatePRLegDiff under test
12026-07-27#1615reporting-at-scaleMOTIR-1728 plan-change conversation (new table + new route tree)
22026-07-27#1618bulk-4MOTIR-1730 conversational rail
32026-07-27#1628reporting-at-scaleMOTIR-909 contextual-planning API

In every case every other check was green — all Vitest legs, coverage, TypeScript, Lint, build, CLA, and every other E2E shard.

The cost is not the re-run itself: it is that a red check indistinguishable-at-a-glance from a real regression lands on unrelated PRs, and each one burns a human's root-cause pass. This is the same class as the packages.microsoft.com apt-source flake fixed in PR #1565 (MOTIR-1679) — an external dependency in the job's critical path with no in-repo resilience.

Fix options (in preference order)

  1. Pull the service image from the GitHub Container Registry mirrorghcr.io/…/postgres:16-alpine (or a Docker Hub authenticated pull, which lifts the anonymous rate/availability posture). Same-host egress as the rest of the Actions infrastructure, so it does not depend on Docker Hub being reachable at all. Preferred: it removes the dependency rather than tolerating it.
  2. Job-level retry around container bringup. The services: block gives no retry hook of its own, so this means either a composite step that docker pulls with a retry loop before the services start, or moving Postgres out of services: and into an explicit docker run step that can be retried (the retry pattern PR #1565 used for playwright install-deps).
  3. continue-on-error + an automatic single re-dispatch — a last resort. It hides the signal rather than fixing it, and would mask a real container misconfiguration.

Whichever lands, apply it to every leg that declares the Postgres service (all E2E shards + the at-scale + a11y + acceptance lanes), not just the two that have flaked — the leg that hits it is random.

Acceptance criteria

  • No E2E leg can fail at Initialize containers because Docker Hub is transiently unreachable: either the image comes from a mirror the runner reaches over Actions-internal egress, or the pull is retried beyond the runner's built-in 3 attempts.
  • The change is applied uniformly to every job declaring the Postgres service container in ci.yml.
  • Verified by a green CI run on the PR itself (the pull path actually exercised — confirm in the job log that the new source/retry ran, not just that CI was green on a lucky day).
  • The flake log entry is updated to "fixed" with the PR reference.

Where this came from

Surfaced during motir run [MOTIR-909](motir:cmqgmjqq7000004jo4ap0vwdp) (PR #1628) per the notes.html #27 out-of-scope-finding protocol — logged rather than absorbed into that PR, whose scope is the contextual-planning API.