Robots.txt Tester
Robots.txt Tester is a free tool that checks whether Googlebot, Bingbot or an AI bot may read a page under your robots.txt, and shows the line that decides.
- Free
- No sign-up
- Runs in your browser
Your robots.txt file tells search engines and AI bots which parts of your site they may read. One wrong line can quietly hide your whole site from Google, or let AI companies copy content you meant to protect. Paste your robots.txt, pick a bot and a page, and see whether it is allowed — plus the exact line that decided, worked out with the same rules Google uses.
Your workspace
Fill in the details above, then press the button to see your result.
Guide
How to use it
- 1Open yourwebsite.in/robots.txt in your browser, copy everything, and paste it into the box.
- 2Type the page you want to test — a full address or just the part after the domain, such as /services.
- 3Choose the bot: Googlebot for Google Search, Bingbot for Bing, or an AI bot like GPTBot or ClaudeBot.
- 4Read the result and the line that decided it. If an important page is blocked, remove or narrow that line.
What the results mean
- User-agent
- The name of the bot a group of rules is for. "User-agent: *" means every bot that has no group of its own. A bot follows only the one group that best matches its name, never two.
- Disallow and Allow
- Disallow asks a bot not to read addresses starting with that path; Allow makes an exception. When both match, the longer, more specific rule wins, and Allow wins a tie.
- * and $
- An asterisk matches anything, so /*.pdf matches every PDF. A dollar sign at the end means "the address ends here", so /*.pdf$ does not match /menu.pdf?v=2.
- Crawling versus indexing
- robots.txt controls whether a bot reads a page, not whether Google lists it. A blocked page can still appear in results with no description if other sites link to it. To keep a page out of Google, use a noindex tag instead.
Common questions
My page is blocked for Googlebot — is that why it is not on Google?
Very likely. Google cannot read a blocked page, so it cannot rank it properly. Remove or narrow the Disallow line the tool points to, then use URL Inspection in Search Console to ask Google to look again.
Should I block AI bots like GPTBot and ClaudeBot?
It is your choice. Blocking them stops those companies using your pages for training, but it can also mean AI assistants know less about your business when customers ask. Blocking AI bots never affects your Google Search ranking.
Why does Googlebot ignore my "User-agent: *" rules?
Because your file also has a group for Googlebot by name. A bot follows only its own group and skips the * group completely, so any rule you want Googlebot to obey must be repeated inside its group.
Does this tool fetch my live robots.txt?
No. It tests exactly what you paste, entirely in your browser. That makes it safe for testing changes before you upload them — paste the new version and check your important pages first.
Guides on this topic
- How to Rank in Google AI Overviews: A 10-Step ChecklistHow to rank in Google AI Overviews: a 10-step checklist from Google's own guidance, from picking queries and answer-first pages to Google-Extended.
- Perplexity SEO: How to Get Cited in Its AnswersHow Perplexity picks sources, how to let PerplexityBot in, write passages it can quote and check whether it cites you, with what is documented and what is not.
- Bing Webmaster Tools: Setup Guide (It Feeds ChatGPT)Set up Bing Webmaster Tools from Search Console, submit your sitemap, use IndexNow, read the AI Performance report, and learn what Bing means for ChatGPT.
- Favicon Guide: Sizes, Formats and Showing in GoogleWhat a favicon is, the sizes and formats you need today, how to add one in WordPress, Shopify, Wix or HTML, and why Google may still show the old one.