01
Access
Can the answer engines fetch this page at all?
- robots.txt behaviour: a 5xx or 429 means “disallow everything” to crawlers; an HTML body means a single-page app is answering for it
- Per-crawler verdicts for 14 crawlers and 2 control tokens, read with an RFC 9309 parser that keeps the line number that decided each one
- The accidental block: training crawlers blocked by name while the answer crawlers fall through to a wildcard that blocks them too
- Cloudflare Content-Signal directives, and hints when an edge provider or a bot challenge sits in front of the site
- Indexing directives in headers and meta tags: noindex, nosnippet, max-snippet:0, data-nosnippet
- HTTP health: status, redirect chain, HTTPS, time to first byte, size, content type
