Home  /  Tools /  AI Visibility

Can the AI engines actually read your site?

Everyone is optimizing to get quoted by ChatGPT and Perplexity. Almost nobody checks whether those crawlers are allowed in the door — and blocking them is usually an accident nobody noticed. This checks in about five seconds. Free, no signup.

// What it checks

Five questions, answered from your live site.

  1. robots.txt, agent by agent. GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot and more — allowed, blocked, or unmentioned.
  2. llms.txt. Whether you publish the machine-readable index AI clients are starting to look for.
  3. Common Crawl. Whether your domain appears in the corpus most models train on.
  4. Entity markup. Whether anything tells a model which real-world organization or person this site is, and whether it links out to confirm it.
  5. JavaScript dependency. How much text exists in the HTML before any script runs.
// Straight answers

Before you ask.

Why does blocking an AI crawler matter?

Because those crawlers decide whether your brand can be quoted in an AI answer. Blocking GPTBot or PerplexityBot removes you from the surface where a growing share of research now happens — and most sites that block them did it by accident, inheriting a rule from a template or a plugin.

Is blocking them ever the right call?

Yes. If your content is your product — paid research, a members-only library — refusing training crawlers is a legitimate licensing decision. The distinction that matters is training crawlers versus retrieval crawlers: you can block the first and allow the second.

What is llms.txt?

A plain-markdown file at the root of your site that points AI clients at your most useful pages, similar in spirit to robots.txt or a sitemap. It is an emerging convention rather than a standard, it costs almost nothing to publish, and hardly anyone has one yet.

Why check for JavaScript rendering?

Most AI crawlers do not execute JavaScript. If your text only appears after a client-side render, they receive a nearly empty page — no matter how good the content is for human visitors.

// Ready to scope it?

Most engagements start with a free 30-minute call.

Tell me where you are, where you want to go, and which lane fits — I’ll come back with a plan within 48 hours.

Yerain Abreu