What the test does
Four things, in about ten seconds:
- Reads your robots.txt and works out, for 49 AI crawlers plus Googlebot, whether they're allowed, partly blocked or shut out.
- Opens your home page as a regular browser and counts the words that show up with JavaScript switched off.
- Knocks again as six crawlers (ClaudeBot, PerplexityBot, Googlebot, GPTBot, OAI-SearchBot and ChatGPT-User) and logs what your server says to each one.
- Looks for an llms.txt file.
An IN needs both: robots.txt lets the crawler in and the server answers it. Fail either one and it's an OUT.
Three ways sites lock out AI without meaning to
- A robots.txt line someone forgot about. One
Disallow: /under the wrong user agent and that crawler is gone, quietly. - Bot protection at the CDN or firewall. Cloudflare and other CDNs have one-click settings that block AI crawlers, and security plugins do it too. robots.txt says yes, the server says 403.
- Content that only exists after JavaScript runs. Most AI crawlers don't run scripts. If the words-with-JS-off number is tiny, they're reading an empty shell.
Not every bot matters the same
Training crawlers (GPTBot, ClaudeBot, CCBot and friends) grab pages to train future models. Search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) index pages so an assistant can cite them. Fetch-on-demand bots (ChatGPT-User, Claude-User, Perplexity-User) grab a page the moment someone asks about it. Blocking training is a business decision. Blocking the other two means the assistant can't quote you, full stop.
Getting in is step one
A clean scoreboard here means the crawlers can read you. Whether ChatGPT, Gemini, Perplexity, Claude and AI Overviews actually name you is a different question, and that's what the tracker answers, prompt by prompt.