AI Crawler Checker
AI Crawler Checker is a free tool that reads your robots.txt and shows whether 13 AI bots, including ChatGPT and Perplexity, may read your website.
- Free
- No sign-up
- Checks a live page
More and more people ask ChatGPT, Claude or Perplexity instead of searching Google. Those tools can only mention your business if their bots are allowed to read your website. This checker reads your robots.txt and tells you, for 13 AI crawlers from OpenAI, Anthropic, Google, Apple, Meta, Amazon and others, whether each one can read your home page — and the exact line that decided it. It also checks for an llms.txt file.
Your workspace
Guide
How to use it
- 1Type your website address.
- 2Press Check AI crawlers. We read your robots.txt the way each bot would.
- 3Look at the Blocked rows. Check the purpose column: blocking a training bot is a choice; blocking a live-answer bot means AI assistants cannot quote or link you.
- 4To change a result, edit robots.txt (or your SEO plugin or Cloudflare setting that writes it) and run the check again.
What the results mean
- robots.txt
- A small public file at yoursite.in/robots.txt that tells bots which parts of your site they may read. Well-behaved bots, including the big AI companies, follow it.
- Training crawler
- Collects pages to teach future AI models. Blocking it keeps your text out of training, but does not stop AI tools answering questions about you.
- Search and live-answer crawlers
- Fetch pages right now to answer a question or show a source link. Block these and AI assistants cannot cite or link your website.
- llms.txt
- An optional plain-text file that gives AI tools a short, clear summary of your business and your key pages. Still new, but cheap to add.
Common questions
Should I block AI crawlers?
For most small businesses, no. Being mentioned when someone asks an AI assistant for a dentist or caterer in your town is free visibility. If you want to keep your writing out of AI training, block only the training bots and leave the search bots open.
I never blocked anything. Why are bots blocked?
Some hosts, security plugins and Cloudflare's 'block AI bots' setting add these rules for you. The rule column shows the exact line, which helps you find where it came from.
Does blocking Google-Extended remove me from Google Search?
No. Google-Extended only controls whether your content is used for Google's AI models such as Gemini. Normal Google Search uses Googlebot, which this setting does not touch.
Does this tool check the whole website?
It checks your home page, which is what matters most. A rule could still block a particular folder; our robots.txt tester can check any other address.
Guides on this topic
- How ChatGPT Picks Local Businesses to RecommendHow ChatGPT picks local businesses: what OpenAI documents about search, location and listings, what is only guessed, and what an owner can fix.
- How to Rank in Google AI Overviews: A 10-Step ChecklistHow to rank in Google AI Overviews: a 10-step checklist from Google's own guidance, from picking queries and answer-first pages to Google-Extended.
- How to Check if AI Assistants Mention Your BrandA monthly routine to check brand mentions in ChatGPT, Perplexity, Gemini and Google AI Mode, plus Google's and Bing's AI reports and AI visits in Analytics.
- Perplexity SEO: How to Get Cited in Its AnswersHow Perplexity picks sources, how to let PerplexityBot in, write passages it can quote and check whether it cites you, with what is documented and what is not.