🤖Free
Free Robots.txt and Sitemap Checker
Check a website’s robots.txt and XML sitemap. See if Google or AI crawlers are blocked, and whether the sitemap is found and readable.
What this tool checks
- ✓Reads robots.txt
- ✓Checks Google and AI crawlers are allowed
- ✓Finds and reads the XML sitemap
- ✓Counts the pages listed
Questions people ask
What is robots.txt?+
A small file at the root of your site that tells search engines and other bots which parts they may visit.
Should I block AI crawlers?+
If you want to be mentioned in ChatGPT, Perplexity and Google’s AI answers, let them in. Blocking them keeps your content out of those answers.
Do I need a sitemap?+
It is not required, but it helps Google find all your pages quickly, especially new ones.
More free tools
🩺On-Page SEO AuditA full check of one page: title, headings, images, speed signals and more.🔠H1 Tag CheckerSee whether a page has exactly one clear main heading.🪜Heading Structure CheckerSee the outline of a page from H1 to H6 and spot skipped levels.📏SEO Title Length CheckerCheck if a title fits on Google without being cut off.📐Meta Description Length CheckerCheck if a description fits on Google without being cut.🔎Google Result PreviewSee how your page will look in Google, on desktop and mobile.
See all 25 tools →Want this done for you, every month?
Prophecius checks every page of your site, connects your Google Search Console, tells you what to fix in plain words, and runs an AI assistant that answers your customers on your website and WhatsApp.