Robots.txt Generator
Robots.txt Generator is a free tool that makes the file telling Google which pages to read, linked to your sitemap, with a clear choice about AI crawlers.
- Free
- No sign-up
- Runs in your browser
robots.txt tells search engines which parts of your site to read. Most small businesses need a very permissive one — the common mistake is copying a file from somewhere that blocks Google entirely, which quietly removes a whole site from search.
Your workspace
Fill in the details above, then press the button to see your result.
Guide
How to use it
- 1Enter your website address so the file can point at your sitemap.
- 2Leave the blocked paths empty unless you have a specific reason. Most sites should.
- 3Decide about AI crawlers. Blocking them also stops AI assistants recommending your business.
- 4Save the result as robots.txt in the top folder of your site, so it loads at yoursite.com/robots.txt
Common questions
Do I even need a robots.txt file?
Not strictly. Without one, search engines assume they may read everything, which for most small business sites is what you want. It becomes useful for pointing at your sitemap and keeping crawlers out of admin or cart pages.
Does blocking a page in robots.txt remove it from Google?
No — and this catches people out. robots.txt stops crawling, not listing. A blocked page can still appear in results with no description. To keep a page out of Google, use a noindex tag and let it be crawled.
Should I block AI crawlers like GPTBot?
For a local business, usually not. Blocking them means AI assistants cannot mention you when someone asks for a dentist in your city — and that is increasingly where people ask. Publishers protecting paid content are a different case.