Everyone is optimizing to get quoted by ChatGPT and Perplexity. Almost nobody checks whether those crawlers are allowed in the door — and blocking them is usually an accident nobody noticed. This checks in about five seconds. Free, no signup.
Because those crawlers decide whether your brand can be quoted in an AI answer. Blocking GPTBot or PerplexityBot removes you from the surface where a growing share of research now happens — and most sites that block them did it by accident, inheriting a rule from a template or a plugin.
Yes. If your content is your product — paid research, a members-only library — refusing training crawlers is a legitimate licensing decision. The distinction that matters is training crawlers versus retrieval crawlers: you can block the first and allow the second.
A plain-markdown file at the root of your site that points AI clients at your most useful pages, similar in spirit to robots.txt or a sitemap. It is an emerging convention rather than a standard, it costs almost nothing to publish, and hardly anyone has one yet.
Most AI crawlers do not execute JavaScript. If your text only appears after a client-side render, they receive a nearly empty page — no matter how good the content is for human visitors.
// Ready to scope it?
Tell me where you are, where you want to go, and which lane fits — I’ll come back with a plan within 48 hours.
