ADR-0059: Public documentation rendering pipeline — self-hosted rustdoc and mdBook onto the Cloudflare site

Reference

Builds on ADR-0052 (documentation site on Cloudflare Pages at docs.thalweg.dev, apex reserved, build command none) and inherits its constraints wholesale. Interacts with ADR-0047 (independent per-crate publishing) — the crates that most need reference docs are the ones published standalone. Governed by the hard rule in AGENTS.md (nothing company-sensitive on the world-readable surface), enforced by scripts/check-public-surface.sh.

Context

The repository is private; only the Cloudflare-served site under docs/site/ is public (ADR-0052). The prose documentation — ADRs, blueprints, operations notes, the testing standard — lives as markdown under docs/, and API reference for the crates does not exist anywhere yet.

Two gaps follow from that:

• No published API reference. The crates are publish = false today (they are scaffolds), so nothing appears on docs.rs, and even once they publish, the four internal crates (twg-cli, the two *-arrow compositions, twg-e2e) never will. There is no rendered cargo doc for anyone — contributor or reader — without checking out the repository and building it.
• The prose docs are only readable as raw markdown. They render on GitHub, but the repository is private, so a reader arriving from the public site cannot reach them. Linking to GitHub blobs from docs.thalweg.dev would send public readers to a 404-behind-auth wall.

The binding constraint from ADR-0052 shapes every option: the Cloudflare build command is NONE. Cloudflare serves docs/site/ verbatim on push; it does not run mdbook, cargo, or any generator. So anything rendered has to be rendered before the push and committed as static output, or it does not exist on the site. A second, build-running deploy path was explicitly rejected in ADR-0052 because it would race Cloudflare’s app.

A further constraint: the site is world-readable and the repository is not, so whatever is rendered onto it is subject to the same secret-scanning gate as the hand-written pages. Generated output must pass check-public-surface.sh.

Options considered + consequences

API reference: how to publish cargo doc

Dimensions: works for unpublished crates, no company-sensitive leakage, no second deploy path, cost to keep current.

Option A1: Self-host rustdoc — build cargo doc locally, commit into docs/site/api/

• Pros: works regardless of publish status, so it covers the four internal crates and today’s all-unpublished workspace alike. No dependency on crates.io or docs.rs. Served from the same origin as the rest of the site, behind the same (optional) Access policy, scanned by the same gate. Cloudflare stays build-none.
• Cons: the rendered tree is committed, so it is large and it can drift from the source if not regenerated. rustdoc’s search index is a minified base64 blob that trips the AWS-key heuristic in the public-surface scan (a false positive to be handled narrowly, not by blinding the gate).

Option A2: Rely on docs.rs

• Rejected. docs.rs only builds published crates, and only builds them from crates.io. The workspace is entirely unpublished today, and the four internal crates are publish = false forever, so docs.rs can never carry them. It also puts the reference on a different origin from the rest of the docs, and outside the private-repo boundary and the public-surface gate.

Option A3: A CI job builds docs to a separate api. subdomain

• Rejected. This reintroduces exactly the second deploy path ADR-0052 refused: a GitHub Actions job publishing to its own target, racing Cloudflare and splitting the origin. It also runs the secret-bearing build in CI rather than behind the same gate as everything else. The whole point of build-none is that there is one publish mechanism.

Prose rendering: how to turn the markdown into a browsable site

Dimensions: static output (build-none compatible), auto-nav from the tree, handles the ADR/blueprint YAML frontmatter, Rust-toolchain-native.

Option B1: mdBook, rendered locally into docs/site/reference/

• Pros: produces a static tree that Cloudflare serves as-is. It is the Rust-native documentation renderer, so it is one cargo install for a Rust contributor and shares the toolchain already required. Navigation is a generated SUMMARY.md, which a committed generator can rebuild from the docs tree so a dropped-in file appears without hand-editing nav. Intra-doc .md links are rewritten to .html automatically.
• Cons: the rendered tree is committed (same drift risk as rustdoc, same mitigation — regenerate on commit). YAML frontmatter in the ADR/blueprint files is not markdown and renders as raw text unless stripped or preprocessed. mdBook’s own searchindex.js is a minified blob, the same false-positive class as rustdoc’s.

Option B2: MkDocs (or another Python/JS static-site generator)

• Rejected. It adds a non-Rust toolchain (Python + pip) to a Rust workspace purely for docs, which every contributor would then need. mdBook gives the same static-output property while staying inside the toolchain the project already mandates.

Option B3: Link to the raw markdown on GitHub

• Rejected. The repository is private, so public readers arriving from the site or a future crates.io listing hit an auth wall. This is the exact coupling ADR-0052 and DOCS-DEPLOY.md warn against.

Freshness: when the rendered output is produced

Option C1: Cloudflare runs the build

• Rejected by ADR-0052. The build command is NONE and a second deploy path races the first. Not available.

Option C2: A pre-commit hook regenerates incrementally and stages the output

• Pros: the rendered output lands in the same commit as the change that caused it — the discipline ADR-0052 already established for the graphify map and AGENTS.md already requires for docs. Incremental (only affected crates rebuild their rustdoc; the book rebuilds only when docs/** markdown changes) keeps commit latency bounded. Fail-soft, so a contributor without mdbook or the toolchain is warned, not blocked.
• Cons: contributors need mdbook and the Rust toolchain installed to regenerate locally; without them the hook no-ops and the output can go stale until someone with the tooling commits. Accepted: the tooling is already needed to work on the crates, and the fail-soft design matches the existing graphify hook.

Option C3: A periodic/manual regeneration

• Rejected. Exactly the “I’ll update the docs after” failure mode AGENTS.md calls out — it is how the site once carried a decision count of 44 against an actual 57. Same-commit regeneration is the established discipline here.

Decision

Self-host both reference surfaces, rendered locally and committed, served by the existing build-none Cloudflare project.

• API reference (Option A1). cargo doc --workspace --no-deps is rendered into docs/site/api/, served at docs.thalweg.dev/api/ and per-crate at /api/<crate>/. scripts/gen-api-docs.sh regenerates the docs for a given set of affected crates and syncs them into the tree.
  • The rustdoc SOURCE VIEW is stripped, never published. rustdoc’s default output includes a full syntax-highlighted source view — every crate rendered into api/src/**/*.rs.html, indexed by api/src-files.js, and reachable from a per-item Source link on every page. The repository is private, so publishing that would publish the entire private codebase onto the world-readable site — forbidden by the hard rule in AGENTS.md. rustdoc has no stable flag to suppress source generation, so the pipeline generates then strips: scripts/strip-rustdoc-src.sh deletes api/src/ and api/src-files.js and removes the relative Source anchors from every rendered page (upstream https://doc.rust-lang.org/…/src/… links, which reference Rust’s own source rather than ours, are preserved). gen-api-docs.sh invokes it after every render — whole-workspace and per-crate alike — so a future docs change can never re-publish the source view. Only the signatures and doc comments (the per-crate api/<crate>/ pages) reach the site.
• Prose reference (Option B1). An mdBook under docs/book/ renders the ADRs, blueprints, operations notes, testing docs and PUBLISHING.md into docs/site/reference/, served at docs.thalweg.dev/reference/. Its build-dir targets ../site/reference so mdbook build docs/book writes straight into the publish root. The SUMMARY.md navigation is generated from the docs tree by scripts/gen-reference-book.sh, which also strips the YAML frontmatter as a pre-render step so it does not appear as raw text.
• Freshness (Option C2). .githooks/pre-commit is extended, alongside the existing graphify step, to regenerate incrementally on commit: changed crates/<name>/** trigger gen-api-docs.sh for just those crates; changed docs/** markdown triggers a SUMMARY regeneration and mdbook build. Every step is fail-soft — a missing cargo/mdbook or any error warns to stderr and the commit proceeds; the hook always exits 0.
• The bespoke landing page stays the apex. docs/site/index.html remains hand-authored; it gains plain links into /reference/ and per-crate /api/ pages, without a new numbered section (which would renumber the gated nav).
• One theme and one way back across all three surfaces. The reference and the crate docs shipped with their own generator defaults (mdBook’s rust/ayu, rustdoc’s light/dark), and neither offered a route back to the landing page — a reader arriving at a crate page was stranded. Both are now re-skinned to the landing palette (paper / iron / verdigris) and carry the same persistent brand bar linking back to the main docs:
  • Reference (mdBook). default-theme/preferred-dark-theme are pinned to light, re-skinned by docs/book/custom/thalweg.css, and a brand bar is injected by docs/book/custom/thalweg.js — both loaded via additional-css/ additional-js and kept outside src/ so the SUMMARY/src/ regeneration never clobbers them.
  • Crate docs (rustdoc). docs/rustdoc/head.html (palette + layout) and docs/rustdoc/before.html (the brand bar) are injected on every page via RUSTDOCFLAGS=--html-in-header/--html-before-content, set in scripts/gen-api-docs.sh, so the theme is re-applied on every render rather than patched in after the fact.
• /api/ has a crate overview instead of a 404. cargo doc --workspace renders one tree per crate but no root index, so docs.thalweg.dev/api/ 404’d and per-crate docs were reachable only by guessing a URL. scripts/gen-api-index.sh writes docs/site/api/index.html — a landing page in the main theme listing and linking every rendered crate. The list is derived from the filesystem (each rustdoc crate root under api/), so it cannot drift from what was rendered, and it runs after the source-view strip, so it only ever links crate index pages, never source. gen-api-docs.sh invokes it after every render.
• The public-surface gate is rescoped minimally. check-public-surface.sh excludes only the minified search-index artifacts (rustdoc’s search-index*.js / search.index/*.js, mdBook’s searchindex.js / searchindex.json) — not the /api or /reference trees — so the rendered HTML (real ADR and rustdoc prose) is still scanned for secrets.

The crates stay publish = false. Self-hosting is precisely what lets the reference exist without publishing, so nothing here flips a crate to published.

Consequences

• Generated output is committed, and must be kept in step. The pre-commit hook is the mechanism; the fallback is scripts/gen-api-docs.sh and scripts/gen-reference-book.sh run by hand. Regeneration is deterministic, so a stale tree is a diff, not a mystery.
• Contributors need mdbook and the Rust toolchain to regenerate locally. This is documented in PUBLISHING.md, operations/DOCS-DEPLOY.md and CONTRIBUTING.md. Absent them the hook fail-soft no-ops, matching the graphify hook’s contract.
• The file count under docs/site/ grows substantially — rustdoc emits a few thousand files for a 44-crate workspace. This stays well under Cloudflare’s 20,000-files-per-deploy limit, but it is now a number to watch as crates gain real surface; the file count is reported in the pull request that introduces the pipeline.
• The public-surface exclusion is narrow by construction. It names the search-index files specifically; the rendered prose is still scanned. If a future generator emits a new minified blob that trips the gate, the fix is to name that file too, never to exclude a whole tree.
• rustdoc --no-deps documents the workspace crates only, not dependencies, so the tree does not balloon with third-party docs and does not leak dependency internals onto the public surface.
• The API surface is signatures and doc comments, not source. Stripping the source view (above) means the public reference shows what each crate exposes and how it is documented, but never the implementation — the appropriate boundary for a private codebase’s public reference. The strip is a script step, not a manual deletion, so it holds across every regeneration.
• Cloudflare stays build-none. No second deploy path is introduced; the site is still served verbatim, so ADR-0052’s ordering argument (gates protect the merge, branch protection enforces sequencing) carries over unchanged.

Advice Received

Document version history