Free AI Crawler Checker
Check your robots.txt for 17 AI crawlers, including GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and Bingbot. See which AI assistants can show your site, and whether you have llms.txt.
What this tool checks
- ✓Checks 17 AI crawlers against your robots.txt
- ✓Separates AI answer engines from AI training crawlers
- ✓Finds llms.txt
- ✓Spots noindex and noai tags on the home page
Questions people ask
Should I block GPTBot?+
GPTBot only collects pages for training. Blocking it does not remove you from ChatGPT search; that is OAI-SearchBot. If you want customers to find you in ChatGPT, never block OAI-SearchBot or ChatGPT-User.
Why might my site be blocked without me knowing?+
Some hosting firewalls, Cloudflare settings and SEO plugins add AI blocks by default. This check reads your live robots.txt so you can see what is really there.
Does allowing crawlers guarantee I show up?+
No, it only makes it possible. Clear pages that answer customer questions, reviews and mentions on other sites are what get you named.
More free tools
Want this done for you, every month?
Prophecius checks every page of your site, connects your Google Search Console, tells you what to fix in plain words, and runs an AI assistant that answers your customers on your website and WhatsApp.