Support explicit product-specific sitemap selection across discovery and coverage

[dacharyc]

Motivation

Follow-up to #120 and PR #140, separate from the bounded-walk fix. On a shared documentation host, a correct product prefix can still find no pages before the global sitemap-index budget is exhausted. Broadening the prefix changes the assessment scope; it does not make the sample representative of the product.

The CLI/config can select a published llms.txt using llmsTxtUrl, but cannot select a sitemap. PR #140 adds an explicit sitemap-root list to the helper for internal coverage fallback; that does not settle the public configuration contract.

Design Questions

• Decide whether users select one sitemap root, several roots, or another existing source abstraction. Settle the contract before naming flags or fields; reuse existing config resolution and validation.
• Define precedence between explicit sources, robots.txt directives, origin/subpath fallback, and the coverage-specific fallback. If an explicit source fails, decide whether fallback is allowed and how it is disclosed; do not silently replace an explicitly selected product source with the whole host.
• Apply selection consistently to page discovery and sitemap coverage, preserving their different URL caps and raw/refined inputs.
• Specify canonical-origin rewriting, redirects, cross-origin source hosting, path/locale/version scope, and root-index traversal. Selecting a source must not implicitly bypass scope rules or become a network allowlist.

Acceptance Criteria

• A product sitemap can be selected through supported CLI, config, and programmatic entry points without walking unrelated global shards first.
• Tests cover conflicting inputs, invalid URLs, unavailable sources, roots containing direct URLs, one-level indexes, staging/canonical rewriting, and declared scope.
• Coverage and discovery identify the source actually used and preserve partial-result warnings/metadata.
• Default behavior remains compatible when no explicit selector is supplied. Existing sampling, verification, and traversal budgets still apply.
• Request assertions demonstrate no per-discovered-page probes or deeper traversal; no product-name heuristics are added.
• Document a complete product-team example and the chosen fallback semantics. Record rejected alternatives.

References

#120, PR #140, #25, #31, #32, #53. The remaining problem is selecting an acquisition source before filtering, not reimplementing the subsection scoping resolved by those earlier issues. Related considerations are recorded in working-notes/page-discovery-notes.md in the local Documentation at Scale draft.