[jat255]

Summary

The public demo became intermittently unavailable while crawlers sent hundreds of requests per minute to Django admin pages. Add crawler guidance and enforce admin request limits at Caddy so excess traffic is rejected before it reaches Gunicorn. Cloudflare now proxies the demo hostname and challenges /staff-admin/; this PR also closes the direct-origin bypass for that route.

Changes

• Serve a demo robots.txt that disallows /staff-admin for cooperative crawlers.
• Build a demo-only Caddy 2.11.2 image with a pinned caddy-ratelimit revision. Production and development continue using stock Caddy.
• Import common application and file routes from Caddyfile.shared in the production and demo entry files, so demo crawler rules stay separate without duplicating those routes.
• Limit admin requests to 30 per minute per visitor IP, then 120 per minute across all visitors. Use Cloudflare's CF-Connecting-IP header for the visitor limit, after rejecting admin requests whose immediate peer is outside Cloudflare's published IP ranges.
• Keep the files hostname and non-admin demo routes directly accessible because the files hostname remains DNS-only on the same origin IP.
• Add a CI workflow that compares the Caddy allowlist with Cloudflare's published IPv4 and IPv6 ranges on PRs, relevant pushes, and weekly. A divergence fails the check and opens one deduplicated GitHub issue on main-branch or manual runs.
• Build and validate the custom image before the existing deploy workflow stops the stack. Update manual build commands and deployment documentation.
• Allow local .localhost testing to select Caddyfile.dev through DEMO_CADDYFILE; the public demo defaults to Caddyfile.demo even if its existing .env names Caddyfile.prod.

Testing

• docker compose -f docker-compose.base.yml -f docker-compose.demo.yml build caddy
• docker compose --env-file .env.demo.example -f docker-compose.base.yml -f docker-compose.demo.yml config --quiet
• actionlint .github/workflows/deploy-demo.yml
• Validated the production and demo Caddyfiles with the built Caddy image and compared their adapted routes against the pre-refactor versions.
• Verified the resolved demo Compose mounts include Caddyfile.demo and Caddyfile.shared.
• In isolated local Caddy containers, robots.txt returned the disallow rule; repeated admin requests returned HTTP 429 at the per-IP cap; direct admin requests returned HTTP 403 even with a forged visitor-IP header; and two simulated Cloudflare visitors received separate rate-limit buckets.
• Validated the exact production demo Caddyfile with the pinned image (Valid configuration).
• The live Cloudflare comparison passed. A temporary stale allowlist produced the expected missing-range report and exit code 1.
• python -m unittest tests.test_cloudflare_ip_check -v: 3 passed.
• actionlint .github/workflows/check-cloudflare-ips.yml: passed.
• python runtests.py: 227 passed.

Risks and deployment

• The first deploy builds a new Caddy image and restarts the demo stack through the existing workflow. Production Caddy now needs the shared file mounted by the base Compose file. No database migration or .env edit is required.
• Cloudflare's demo A record must stay proxied and its /staff-admin/ Managed Challenge rule enabled. The direct-origin restriction takes effect only after this PR is merged and deployed.
• Keep Caddy's Cloudflare IP ranges current with Cloudflare's published list. The new weekly check alerts via a failed workflow and a deduplicated GitHub issue. An outdated list could block legitimate admin requests. Cloudflare IP ranges are shared across customer zones and are not cryptographic origin authentication.
• Visitors can receive HTTP 429 on admin pages when the limits are reached. Review access logs after rollout and adjust the limits if normal demo use is affected.