serve robots.txt and sitemap.xml

Two exact paths a crawler asks for by name, so they are mux entries rather than
resolver cases — no bundle can collide, since a key always sits under a section.

robots.txt at the site root is served verbatim, because a site that ships one has
said something deliberate; otherwise the engine emits the minimum that is true and
points at the sitemap. The sitemap lists every bundle in every language it exists
in, since each variant is separately reachable, with lastmod only where a bundle
has a date. Every URL comes from content.URL like every other path the engine
emits, so a sitemap cannot disagree with what is actually served.

Both need a declared base. Without one the sitemap answers 404 rather than listing
paths no crawler can resolve, and robots omits the Sitemap line rather than
writing a relative one.

write() was setting text/html for every caller, and headers only go out with the
first byte — so a handler setting its own type would have had it silently replaced,
which is how a sitemap gets served as a web page. It now splits into write and
writeAs, and the tests assert the content types rather than only the bodies.

docs/state.md

@@ -10,23 +10,27 @@ If this file disagrees with the code, the code is right and this file is a bug.
 | `go.mod` | module `khosra`; `goldmark`, `x/text`, `yaml.v3` direct | 10 |
 | `internal/content/doc.go` | package comment | 5 |
 | `internal/content/content.go` | bundles: `os.Root` open, walk, frontmatter split, key/lang derivation, NFC, tag slugs, partial files, permalink building | 368 |
-| `internal/content/site.go` | the indexed site: lookup with language fallback, aliases, `Query` and `Run`, sections, `Sequence` | 286 |
-| `internal/render/render.go` | goldmark with the typographer, per-kind template sets with site override, the `Partial`/`Origin` seams features render and resolve through, `Page`/`List`/`Sequence`/`head` | 386 |
+| `internal/content/settings.go` | `site.yaml`: the site's own declarations (`base`, `title`) and absolute-URL building (ADR-0039) | 59 |
+| `internal/content/site.go` | the indexed site: lookup with language fallback, aliases, `Query` and `Run`, sections, `Sequence`, `Everything` | 300 |
+| `internal/render/render.go` | goldmark with the typographer, per-kind template sets with site override, the `Partial`/`Origin` seams features render and resolve through, `Page`/`List`/`Sequence`/`head` | 409 |
 | `internal/render/chrome.go` | the engine's own words: phrase table, month names, digits, and the `t`/`num`/`day` template funcs (ADR-0034) | 105 |
 | `internal/render/templates/` | reference theme: `base.html`, `page.html`, `list.html`, `shortcodes.html`, `theme.css` (ADR-0026) | — |
 | `internal/ext/shortcodes/` | first feature: `{{< name key="value" >}}` block parser and node renderer, rendering through a theme fragment (ADR-0036). `figure`, `gallery`, `include` | 315 |
 | `internal/ext/widows/` | second feature: joins the last two words of a paragraph or heading with a non-breaking space, over the tree so code spans are safe | 108 |
 | `cmd/khosra/wire.go` | the only list of enabled features (`extensions.md`) | 20 |
 | `internal/web/resolve.go` | URL → (key, lang, page, tag) or a canonical redirect: language prefix, `/en/…` fork guard, pagination, tags, trailing slash | 112 |
+| `internal/web/discover.go` | `/robots.txt` and `/sitemap.xml`, absolute and only with a declared base (ADR-0039) | 74 |
 | `internal/web/web.go` | handler: resolve, look up with fallback, section and tag listings, sequence, `/static/` (misses and refusals alike answer 404), degrade on failure | 152 |
-| `cmd/khosra/main.go` | flags, wiring, startup — the only place things are assembled | 53 |
-| `*_test.go` | table-driven, one file per source file; symlink escape (content and static), canonical paths, language fallback, aliases, pagination, tags, sequences, chrome, typography, shortcode escaping, galleries, includes, partials, widows, 404 | 1562 |
+| `cmd/khosra/main.go` | flags (`-site`, `-addr`, `-base`), wiring, startup — the only place things are assembled | 60 |
+| `*_test.go` | table-driven, one file per source file; symlink escape (content and static), canonical paths, language fallback, aliases, pagination, tags, sequences, chrome, typography, shortcode escaping, galleries, includes, partials, widows, site settings, absolute URLs, robots, sitemap, 404 | 1755 |
 
 Serves a bundle at `/{section}/{slug}/`, a paginated listing per section, tag listings global and
-section-narrowed, sequence navigation and a series archive on any nested bundle, and `static/` verbatim.
+section-narrowed, sequence navigation and a series archive on any nested bundle, `static/` verbatim, plus `/robots.txt` and
+`/sitemap.xml`.
 Chrome text, dates and digits render in English or Bengali; authored text is untouched but for typographic
 smoothing and widow prevention (ADR-0034). This repo holds engine source only — the site root is external and passed with
-`-site` (ADR-0011).
+`-site` (ADR-0011). `site.yaml` declares `base` and `title`; with a base, canonical, hreflang and OpenGraph
+URLs go absolute (ADR-0039).
 
 Frontmatter the parser lifts today: `title`, `date`, `tags`, `aliases`, `order`. Every other key in
 `content-model.md`'s table — including `slug`, `draft` and `type` — lands in `Extra` unread, so that table
@@ -42,7 +46,7 @@ this change*.
 | Counter | Now | Extraction due at | What it buys |
 |---|---|---|---|
 | Render transforms — **page-level only** | 0 | **3** | Stage pipeline (ordered `func(ctx,*Page) error`). Parse-phase work does *not* count and must not: goldmark's extender list is already an ordered pipeline for it, so typography, shortcodes and widows compose there (`cmd/khosra/wire.go`) and a second pipeline beside it would be pure duplication. This counts transforms over the assembled page, which nothing hosts yet — OpenGraph and JSON-LD (queue 15) are the first candidates |
-| Routing cases | 5 | **2** — done | Resolver at `internal/web/resolve.go`: bundle, language prefix, pagination, tag, section-narrowed tag |
+| Routing cases | 7 | **2** — done | Resolver at `internal/web/resolve.go`: bundle, language prefix, pagination, tag, section-narrowed tag |
 | Collection pages | 4 | **1** — done | Query primitive: `content.Query{Section, Tag, Lang}` + `Site.Run`. The fourth — a series archive — resolves through `Site.Sequence` instead: membership is structural and the sort ascends, so it shares the index but not the Query |
 | Views / output formats | 2 | **2** — due | Two template sets exist (bundle, listing); the View layer is Arc 2's third item |
 | Effects | 0 | **2** | Effect runner + trigger wiring (change / schedule / demand) |