AI crawler checker: inspect your site's robots.txt policies
An AI crawler checker can help interpret the public robots.txt rules for selected AI-related agents and policy controls. That is a policy check, not proof that a genuine provider bot can retrieve every page. Firewalls, authentication, and other infrastructure can affect the actual request.
Use the results to identify the next investigation. An allowed policy does not guarantee a citation, and a training restriction does not automatically mean search access is blocked.
- OpenAI, Anthropic, Google, Perplexity and more
- Training, search and user roles separated
- Allowed, disallowed or unknown
Check my robots.txt policy
How the check works.
robots.txt read
We fetch your public robots.txt and apply RFC 9309 matching rules for each listed agent or policy control.
Separate observation
We make one normal request for your homepage and report it on its own. We never send requests pretending to be a provider bot.
Policy status
See which agents your robots.txt allows, disallows, or leaves unknown, grouped by training, search and user-triggered roles.
What does the checker examine?
The check retrieves the site's public robots.txt file and interprets applicable rules for the agents and controls listed in the report. The result identifies the URL checked, the matching rule, and whether the conclusion is allowed, disallowed, or unknown under the implemented parser.
The checker does not impersonate provider bots or gain their verified network identity. A separate normal request for your homepage is labeled as its own observation. A normal fetch cannot establish how every verified bot is treated.
How should you read the results?
Read the purpose as well as the policy. A training crawler and a search crawler have different roles. A summary such as three controls disallowed should not be presented as three assistants unable to recommend you.
If the file cannot be fetched or interpreted reliably, the result is shown as unknown. A timeout is not labeled blocked, and a missing observation is not labeled allowed. The report explains what evidence is available.
Review the exact host and path. A policy for the main domain may differ from a subdomain or another section. A site-wide overview is a starting point, not a full audit of every URL.
Selected agents and controls
| Name | Role to distinguish | Practical implication |
|---|---|---|
| GPTBot | Potential OpenAI model-training use | Not the same as ChatGPT search access |
| OAI-SearchBot | OpenAI search discovery | Review search access independently |
| ChatGPT-User | User-triggered OpenAI retrieval | Not a universal search inclusion control |
| ClaudeBot | Anthropic training crawler | Separate from Claude search and user agents |
| Claude-SearchBot and Claude-User | Search and user-triggered Anthropic roles | Check each documented role separately |
| PerplexityBot and Perplexity-User | Search and user-triggered Perplexity roles | Requested fetches and crawling can behave differently |
| Google-Extended | Policy token for specified Gemini uses | Not a separate HTTP crawler or Search inclusion switch |
| Applebot-Extended | Policy control for Apple model-training use | Does not itself crawl pages |
| CCBot | Common Crawl dataset crawler | Evaluate dataset policy separately from live AI search |
Roles checked November 11, 2026 against provider documentation in the AI crawler audit guide.
Should you allow or block each one?
Make a deliberate policy decision based on purpose. A business may support search discovery while limiting potential training uses, subject to the provider's documented controls. Do not assume that allowing every AI-related name guarantees recommendations.
Keep private material protected with proper access controls. robots.txt is not authentication. Review changes with the responsible website owner and avoid broad security exceptions merely to improve a summary indicator.
Why can access fail when robots.txt looks permissive?
A CDN challenge, firewall rule, rate limit, login, or unavailable page can prevent retrieval through a particular route. The policy file and the enforced response are separate evidence.
Ask the host for verified provider requests and matching rule outcomes when actual access is the concern. Inspect the returned body as well as the status. The crawler audit guide explains those deeper checks.
Frequently asked questions
What is an AI crawler?+
It is an automated system used to discover or retrieve web information for an AI-related purpose. Different agents serve training, search, or user-requested tasks, and some familiar names are policy controls rather than crawlers. Check the provider's current documentation before treating every AI-related entry as the same kind of access.
How do I check whether GPTBot can access my site?+
Start by interpreting the relevant robots.txt policy, then inspect actual verified requests and infrastructure rules if you need evidence of real access. A normal checker fetch cannot assume GPTBot's identity. Keep the policy conclusion separate from whether a genuine provider request succeeded and whether the content could support a later answer.
Should I block AI crawlers?+
That depends on the intended use and your organization's policy. Separate training preferences, search discovery, and user-triggered access rather than make one blanket assumption. Keep private content behind appropriate authentication. Review the specific provider controls and business consequences before changing rules, and do not promise that a policy setting guarantees visibility.
Does blocking GPTBot stop ChatGPT Search from citing me?+
OpenAI documents GPTBot and OAI-SearchBot as different controls for different purposes. A GPTBot training preference should not be treated as the universal switch for ChatGPT Search. Review the current provider documentation and actual site access, then assess citations separately. A policy check alone does not prove inclusion or exclusion in every answer.
Why are bots blocked when my robots.txt allows them?+
The enforced request can be affected by CDN or firewall rules, authentication, rate limits, or page behavior. robots.txt describes policy for compliant agents, not every infrastructure outcome. Inspect verified requests and rule logs with the host, and check the response body. Avoid disabling protection broadly or assuming that every failed normal fetch represents a real bot block.
Move from a policy check to a useful diagnosis
Read the practical AI crawler audit, explore a technical AEO visibility audit, or run a free visibility scan for a separate view of sampled brand appearances.