MotirBuilding in public
MOTIR · moooon
onMotir
You’re viewing a public project. Anyone can view it — no account needed. Sign in to submit, upvote, or comment on requests.View-only — you can’t edit work items
MOTIR-3726

`app.motir.co/robots.txt` 404s — motir-core ships a sitemap, JSON-LD and canonicals on its public surface but no `app/robots.ts`

To Do
Description

Type: code · Executor: coding_agent · Repo: motir-core

Found while working MOTIR-1155 (8.3.8, Search Console setup), 2026-08-27.

https://app.motir.co/robots.txt returns 404 (text/html, Next.js' not-found page — measured with a Googlebot UA). motir-core has no app/robots.ts, while the rest of the public-surface SEO scaffolding did ship under 6.13/8.9.4: app/sitemap.ts (10 URLs), per-project + explore JSON-LD, canonicals, section OG images.

Severity is low, and the honest reason matters: an absent robots.txt reads as allow everything, so nothing is blocked or de-indexed today, and sitemap discovery is already solved a better way — the sitemap is submitted directly to the verified Search Console property (MOTIR-1155). This is crawl hygiene ahead of going public, not a live defect.

What is actually missing is the ability to say don't crawl this: /api/*, the auth routes, and the ?rank= facet URLs (/explore?rank=popular, /explore?rank=recent) which are duplicate views of /explore and currently sit in the sitemap as separate <loc>s.

Scope

  • Add app/robots.ts (Next.js MetadataRoute.Robots) — allow crawl, sitemap: → https://app.motir.co/sitemap.xml, host set.
  • Disallow: /api/, /sign-in, /sign-up, and the other authed-only route groups that currently return a redirect to sign-in.
  • Decide the ?rank= facets deliberately — either drop them from app/sitemap.ts in favour of the canonical /explore, or leave them and let the existing canonical tags do the de-duplication. Don't do both half-way.
  • Build the URLs from the same base the sitemap uses; no second hardcoded host. (metadataBase was the subject of MOTIR-2505 — reuse it, don't reintroduce a literal.)

⚠️ This card creates the file, ONE host at a time. It ships the app host's directives against today's single origin; making the same route answer by host is MOTIR-3881's work, not this card's. Do not anticipate the second host here — a robots.ts that branches on a hostname nothing yet serves is untestable and pre-empts the ADR (MOTIR-3879) that decides which host answers for what.

Acceptance criteria

  • https://app.motir.co/robots.txt returns 200 text/plain with a Sitemap: line pointing at the live sitemap.
  • The disallow set covers /api/ and the auth routes; /explore and /p/* remain crawlable.
  • A test asserts the route renders and that /explore is NOT disallowed — the failure mode of a robots file is silently de-indexing the surface it was meant to help, so guard the ALLOW, not just the deny.
  • The ?rank= facet decision is recorded on this card, not left implicit.

Context refs

  • app/sitemap.ts (motir-core) — the existing, shipped sitemap this must reference
  • MOTIR-1154 (8.3.7) plans a robots.ts for motir-marketing — a different repo, a different file; that card is not widened to cover this one
  • MOTIR-2505 — metadataBase; the base-URL source of truth
  • blocks MOTIR-1130 (8.10.3, smoke the public surfaces) — going public is where this bites
  • blocks MOTIR-3881 — the card that makes this file host-aware once motir.co is served by motir-core

⚠️ Where this card lives, and why it moved (2026-08-29)

Authored under 8.10, re-parented to MOTIR-3876 — beside MOTIR-3881, the card it gates. A prerequisite takes the SAME parent as the gated card and the blocked_by edge is the whole relationship; left under 8.10 the edge crossed a story boundary and made MOTIR-3876 un-finishable (validate_work_item returned valid: false naming exactly this edge). The MOTIR-1130 edge is unchanged and now points backward into an earlier story, which is the legal direction.