> ## Documentation Index
> Fetch the complete documentation index at: https://areyoufoundbyai.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Check Which AI Crawlers Your robots.txt Allows

> Check the published robots.txt rules for 15 search, training and user-request crawler controls on any page, with search access and training preferences shown separately.

Before an AI engine can quote a page, its crawler has to be allowed to fetch it. A `Disallow` rule in your `robots.txt` that names an AI crawler, or catches it through a wildcard group, tells that crawler to stay away. The AI Crawler Checker reads the robots.txt rules that apply to one page and shows what they say for 15 published crawler controls. Search access and training preferences are shown separately, because they are different choices.

<Note>
  The checker reads robots.txt and matches its rules to your page. It does not test a verified crawler, login restrictions or bot protection. An allowed rule does not guarantee that an engine can read or cite the page.
</Note>

## How to use the checker

Go to [areyoufoundbyai.com/tools/ai-crawler-checker](https://areyoufoundbyai.com/tools/ai-crawler-checker), paste the full address of a page and select **Check permissions**. Use the full page URL: rules can differ between pages on the same site.

## The 15 crawler controls it checks

Every control below is one its provider publicly documents, and the checker links each result to the provider's own documentation. The checker groups them by purpose:

| Crawler | Who it is | Group |
| - | - | - |
| OAI-SearchBot | ChatGPT search | Search |
| Claude-SearchBot | Claude search | Search |
| PerplexityBot | Perplexity search | Search |
| Googlebot | Google Search | Search |
| Applebot | Apple Search, Siri and Spotlight | Search |
| bingbot | Bing Search, which grounds Copilot | Search |
| GPTBot | OpenAI model training | Training and grounding |
| ClaudeBot | Anthropic model training | Training and grounding |
| Google-Extended | Gemini training and grounding | Training and grounding |
| Applebot-Extended | Apple foundation model training | Training and grounding |
| meta-externalagent | Meta AI training and indexing | Training and grounding |
| CCBot | Common Crawl, the corpus many models train on | Training and grounding |
| ChatGPT-User | ChatGPT user requests | User requests |
| Claude-User | Claude user requests | User requests |
| Perplexity-User | Perplexity user requests | User requests |

GPTBot and ClaudeBot cover training, separately from their providers' search crawlers. Google-Extended covers Gemini training and grounding; it does not control inclusion in Google Search. For ChatGPT-User and Perplexity-User, robots.txt may not apply: these are fetches made when someone asks an assistant to open a page.

<Note>
  **Two AI vendors this check cannot assess.** xAI (Grok) and DeepSeek publish no crawler user-agent, so there is nothing for a `robots.txt` rule to name. Their absence from the list is neither permission nor a block. It is a limit on what `robots.txt` can show.
</Note>

## What the results show

For the exact page you entered, the result states when it was checked, whether `robots.txt` was read successfully, was missing, or was unavailable, and a link to the rules file. Each crawler then gets one of these states:

* **Allowed by rule**: the rules allow this crawler on this page. This includes cases with no matching disallow rule.
* **Disallowed by rule**: a `Disallow` rule covers this crawler for this page.
* **Unknown**: no rule could be established, for example because `robots.txt` was unavailable.
* **Not established by robots.txt**: robots.txt does not settle access for this control.

Open **Rule and source** on any row to see the matched rule and its line number, the applicable user-agent group and the provider's documentation.

The checker also makes one simulated page request using the OAI-SearchBot user-agent label and reports the HTTP status. That is a simulation, not a verified crawler request, and does not establish actual crawler access.

## Allowing AI crawlers

If a crawler you want shows as **Disallowed by rule**, open **Rule and source** to find the line that blocks it, then change that rule or add an explicit group for the crawler in `robots.txt`. Only add the groups you actually want, and decide search access and training separately. Run the checker again afterwards to confirm the result:

```text theme={null}
# Search crawlers
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Googlebot
Allow: /

User-agent: Applebot
Allow: /

User-agent: bingbot
Allow: /

# Training and grounding (your choice)
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: meta-externalagent
Allow: /

User-agent: CCBot
Allow: /

# User requests
User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Perplexity-User
Allow: /
```

You can still disallow paths you do not want fetched (internal search pages, cart endpoints, API routes) inside each group.

<Accordion title="What if my firewall blocks crawlers even though robots.txt allows them?">
  A permissive `robots.txt` is necessary but not sufficient. Some web application firewall and CDN security rules block user-agents by name or by traffic pattern before `robots.txt` is ever consulted, and the checker does not test bot protection. If a crawler shows as **Allowed by rule** but you suspect it is still blocked, check your Cloudflare, Fastly or server firewall rules, and look for the crawler's user-agent string in your server access logs.
</Accordion>

<Accordion title="Should I allow all crawlers, or can I be selective?">
  You can be selective. Blocking a training crawler while allowing the same provider's search crawler is a legitimate choice, which is why the checker shows the two separately. Allowing a crawler does not guarantee that the engine will name you, and other sources can also lead an engine to name your business.
</Accordion>

<Tip>
  Run the checker after every significant `robots.txt` change and whenever you deploy a new CDN or firewall configuration. Then [run the free scan](https://areyoufoundbyai.com/) to see whether ChatGPT and Gemini name your business.
</Tip>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.