Disallow rule in your robots.txt that names an AI crawler, or catches it through a wildcard group, tells that crawler to stay away. The AI Crawler Checker reads the robots.txt rules that apply to one page and shows what they say for 15 published crawler controls. Search access and training preferences are shown separately, because they are different choices.
The checker reads robots.txt and matches its rules to your page. It does not test a verified crawler, login restrictions or bot protection. An allowed rule does not guarantee that an engine can read or cite the page.
How to use the checker
Go to areyoufoundbyai.com/tools/ai-crawler-checker, paste the full address of a page and select Check permissions. Use the full page URL: rules can differ between pages on the same site.The 15 crawler controls it checks
Every control below is one its provider publicly documents, and the checker links each result to the provider’s own documentation. The checker groups them by purpose:
GPTBot and ClaudeBot cover training, separately from their providers’ search crawlers. Google-Extended covers Gemini training and grounding; it does not control inclusion in Google Search. For ChatGPT-User and Perplexity-User, robots.txt may not apply: these are fetches made when someone asks an assistant to open a page.
Two AI vendors this check cannot assess. xAI (Grok) and DeepSeek publish no crawler user-agent, so there is nothing for a
robots.txt rule to name. Their absence from the list is neither permission nor a block. It is a limit on what robots.txt can show.What the results show
For the exact page you entered, the result states when it was checked, whetherrobots.txt was read successfully, was missing, or was unavailable, and a link to the rules file. Each crawler then gets one of these states:
- Allowed by rule: the rules allow this crawler on this page. This includes cases with no matching disallow rule.
- Disallowed by rule: a
Disallowrule covers this crawler for this page. - Unknown: no rule could be established, for example because
robots.txtwas unavailable. - Not established by robots.txt: robots.txt does not settle access for this control.
Allowing AI crawlers
If a crawler you want shows as Disallowed by rule, open Rule and source to find the line that blocks it, then change that rule or add an explicit group for the crawler inrobots.txt. Only add the groups you actually want, and decide search access and training separately. Run the checker again afterwards to confirm the result:
What if my firewall blocks crawlers even though robots.txt allows them?
What if my firewall blocks crawlers even though robots.txt allows them?
A permissive
robots.txt is necessary but not sufficient. Some web application firewall and CDN security rules block user-agents by name or by traffic pattern before robots.txt is ever consulted, and the checker does not test bot protection. If a crawler shows as Allowed by rule but you suspect it is still blocked, check your Cloudflare, Fastly or server firewall rules, and look for the crawler’s user-agent string in your server access logs.Should I allow all crawlers, or can I be selective?
Should I allow all crawlers, or can I be selective?
You can be selective. Blocking a training crawler while allowing the same provider’s search crawler is a legitimate choice, which is why the checker shows the two separately. Allowing a crawler does not guarantee that the engine will name you, and other sources can also lead an engine to name your business.