MotirBuilding in public
MOTIR · moooon
onMotir
You’re viewing a public project. Anyone can view it — no account needed. Sign in to submit, upvote, or comment on requests.View-only — you can’t edit work items
MOTIR-3881

One robots.txt and one sitemap.xml for the whole host — and publicSiteOrigin() stops delegating to MOTIR_BASE_URL

Blocked
Description

Type: code · Executor: coding_agent · Repo: motir-core · ONE PR.

Two origins now exist where one variable answers for both, and the crawl directives for each host are a different file. This card separates them, and it is the card the rest of the story's URLs depend on.

1 — Split the PUBLIC SITE origin from the APP origin

lib/baseUrl.ts is deliberately the one place MOTIR_BASE_URL is read, and its own comment is emphatic: "Nothing else may read the variable directly — a second reader is a second answer to the question this module exists to answer once." Today it answers two questions with one value — where is the application? (email links, Better-Auth baseURL and trustedOrigins, OAuth callbacks) and what origin do public canonicals, OpenGraph URLs and sitemap entries use? Those are about to be different hosts.

The change belongs inside that module's contract, not beside it: lib/publicProjects/urls.ts's publicSiteOrigin() stops delegating to resolveBaseUrlTrimmed() and resolves its own configured origin, with the same two-rung precedence, the same empty-is-unset rule, and the same single-reader discipline. lib/baseUrl.ts's comment is updated to say which consumers it keeps and which moved — a module whose comment lists a consumer it no longer serves is the drift this rule exists to stop.

⚠️ /p/* is NOT re-origined by this card. publicProjectUrl() keeps naming the host /p/* is actually served from, which until the next story is still the application host. Getting this wrong publishes canonicals at an address that 404s.

2 — One robots.txt per host, host-aware

MOTIR-3726 — the sibling card directly above this one — creates app/robots.ts (it does not exist at all today; app.motir.co/robots.txt 404s). This card makes that file answer by host:

  • on the public site — allow, and name that host's sitemap;
  • on the application host — disallow the authenticated segments and the auth routes, and name nothing indexable.

Next's robots.ts is a route handler, so the host is read from the request the framework hands it; the resolution belongs in a small pure function with the host as an argument, unit-tested per host, with the route as a thin caller (the 4-layer discipline applied to a framework boundary).

3 — Two sitemaps, each honest about its own host

app/sitemap.ts today emits the square, its rank variants, every topic page and every public project against ONE origin, and it is force-dynamic for a recorded reason (MOTIR-2490: it reads the database, and the Fly image builds with placeholder credentials). Both properties stay. What changes is that it becomes host-aware in the same shape as robots.ts: the public site's sitemap lists /, /explore + rank variants, every /explore/topic/<slug>, /docs and its surfaces, and /legal + each document; the application host's lists the public-project URLs only while /p/* is still served there, and the file says so with a pointer to the story that moves them.

Conventions this respects

Framework boundaries call services (publicProjectsService, projectTagsService, legalDocumentSlugs, lib/apiDocs/surfaces) rather than Prisma; no new direct database read is introduced; the dynamic sitemap stays dynamic; metadataBase continues to resolve absolutely (the defect MOTIR-2505 fixed must not regress under a second origin).

Acceptance criteria

  • publicSiteOrigin() resolves from its own configured variable; lib/baseUrl.ts no longer serves the public-canonical question, and its precedence comment names the split. Exactly ONE module reads each variable — asserted by a test that greps the tree, in the shape tests/hosting/appUrlSeam.test.ts already uses.
  • An unset public-site origin falls back to the local dev origin exactly as today, and an empty-or-whitespace value counts as unset. Both arms unit-tested.
  • app/robots.ts returns allow + sitemap for the public host, and disallows /home, /settings, /items, /sign-in, /sign-up and the rest of the authed segment set for the application host. The disallow list is derived from the app/(authed)/ filesystem, not hand-listed — a hand-listed set is the defect proxy-matcher.test.ts was written against.
  • app/sitemap.ts emits per host as described, stays force-dynamic, and its /p/* entries name the host those pages are actually served from today.
  • No canonical, og:url, JSON-LD @id or sitemap entry anywhere in app/(public)/p/ changes in this PR — asserted, because this is the card most able to break it by accident.
  • Unit tests cover: origin resolution per variable and per fallback, robots per host, sitemap composition per host, and the authed-segment derivation against a fixture tree.
  • ≥90% branch/function/line coverage on every file this PR touches.

Context refs

  • motir-core/lib/baseUrl.ts — the precedence contract and its single-reader rule
  • motir-core/lib/publicProjects/urls.tspublicSiteOrigin(), publicProjectUrl(), publicProjectPath()
  • motir-core/app/sitemap.ts — the force-dynamic reasoning (MOTIR-2490) that must survive
  • motir-core/tests/hosting/appUrlSeam.test.ts — the existing seam guard this extends
  • motir-core/tests/navigation/proxy-matcher.test.ts — the filesystem-enumeration idiom the disallow list follows
  • MOTIR-3726 — the SIBLING card that creates the file this card edits; wired blocked_by
  • MOTIR-2505 — the metadataBase defect that must not regress
  • motir-marketing/app/robots.ts · app/sitemap.ts — what the public host's versions must subsume

⚠️ The blocked_by into MOTIR-3726, and why that card was MOVED (2026-08-29)

This card is blocked_by MOTIR-3726, and the dependency is genuine: 3726 CREATES app/robots.ts and this card makes it host-aware; building this first would mean writing that file twice.

3726 was authored under 8.10 (MOTIR-1104) and has since been re-parented to this story, so the edge is now an ordinary sibling edge inside one parent. The earlier version of this section argued the opposite — that the edge was cross-parent with no legal lift, and should stay at the leaf. That argument was wrong on both halves. The two cards were never cross-EPIC: 3726 and this card have always shared epic MOTIR-3875, so the nearest common level was the epic, not the project root. And a leaf edge across a story boundary is not a cheaper lift — it makes this story un-finishable, which validate_work_item(MOTIR-3876) reported as valid: false with this edge as the only blocker. A prerequisite takes the SAME parent as the card it gates; that is the placement rule, and moving the card is what discharges it.

8.10's MOTIR-1130 keeps its own blocked_by into 3726. That edge now points backward, into an earlier story, which is the legal direction and needs no lift.