Add docs-site robots.txt allowlist and generate llms.txt from docs content

[Copilot]

The docs site was missing effective GEO signals for AI crawlers: robots.txt did not match the expected allowlist, and llms.txt described raw agent prompt files rather than the documentation surface at github.github.com/gh-aw. This updates both outputs so AI indexers can discover and traverse the docs site directly.

• Robots policy

  • Reduced docs/public/robots.txt to the explicit crawler allowlist called out by the issue:
    • GPTBot
    • OAI-SearchBot
    • ClaudeBot
    • Claude-SearchBot
    • PerplexityBot
    • Google-Extended
  • Pointed the sitemap directive at the site-level sitemap.xml.

  User-agent: GPTBot
  Allow: /
  
  User-agent: ClaudeBot
  Allow: /
  
  Sitemap: https://github.github.com/gh-aw/sitemap.xml

• llms.txt now indexes the docs site

  • Reworked docs/src/pages/llms.txt.ts to generate a docs-oriented llms.txt from docs/src/content/docs/**.
  • The output now includes:
    • a site-level H1 and description
    • homepage reference
    • grouped sections such as Setup, Guides, Reference, Specifications
    • page descriptions pulled from frontmatter where available
    • an Optional section for blog content
  • This replaces the previous prompt-file listing backed by .github/aw/*.

  # GitHub Agentic Workflows
  
  > Official documentation for GitHub Agentic Workflows (gh-aw)...
  
  ## Setup
  - [Quick Start](https://github.github.com/gh-aw/setup/quick-start/): Get your first agentic workflow running in minutes.
  
  ## Reference
  - [Tools](https://github.github.com/gh-aw/reference/tools/): Configure GitHub API tools, browser automation, and AI capabilities...

• Docs-site coverage

  • Updated the existing robots.txt expectation to match the new policy.
  • Added focused coverage for llms.txt so the docs build continues to serve:
    • site metadata
    • docs-section headings
    • representative docs links
    • docs-derived content instead of raw GitHub prompt URLs

• Docs README

  • Documented that the docs site serves /gh-aw/llms.txt and /gh-aw/llms-full.txt.

[github-actions]

Great work on the docs GEO signals improvement! The PR cleanly separates the robots.txt crawler allowlist from the reworked llms.txt generator — and both come with updated tests (llms-txt.spec.ts newly added, robots-txt.spec.ts updated) and a thorough description. This looks ready for review.

One minor note: docs/src/content/docs/agent-factory-status.mdx and docs/src/content/docs/reference/model-tables.md are changed but not mentioned in the PR body. If these are frontmatter/description updates feeding the new llms.txt generator, a brief note in the description would help reviewers confirm they are in scope.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• patchdiff.githubusercontent.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "patchdiff.githubusercontent.com"

See Network Configuration for more information.

Generated by ✅ Contribution Check · 296.9 AIC · ⌖ 13.7 AIC · ⊞ 6.2K · ◷

[github-actions]

🤖 PR Triage

Field	Value
Category	docs
Risk	🟢 low
Score	27/100 (impact:15 urgency:5 quality:7)
Action	⏸ defer

Status: DRAFT — needs undraft + CI before review.

Adds GEO/AI crawler signals: updates docs/public/robots.txt to the expected allowlist and regenerates llms.txt to describe the docs site surface rather than raw agent prompt files. Low criticality.

Run §28966928999

Generated by 🔧 PR Triage Agent · 106.9 AIC · ⌖ 10.9 AIC · ⊞ 5.4K · ◷