Robots.txt Explained: What to Block and What to Leave

By Neha·Published October 6, 2026

There is a small text file on most websites that almost no owner has ever opened. It is called robots.txt, it is usually four or five lines long, and one wrong line in it can hide an entire business from Google. Picture a furniture showroom whose site vanishes from search for two months because a developer left a single "Disallow" line behind after a redesign. Nobody notices, because the website itself looks perfectly fine. It is one of the most common quiet disasters on small business websites.

So this is a guide to that file: what robots.txt is, what it can and cannot do, and how to check yours in about five minutes.

What is robots.txt, in one paragraph

Search engines do not look at your site the way a person does. They send a program, usually called a crawler or a bot, that reads page after page. Before reading anything, a polite bot asks for one address first: yourwebsite.in/robots.txt. That file is your list of house rules. It says which parts of the site the bot may read and which parts it should skip.

Think of it as the board outside a housing society: "Visitors allowed, delivery staff use gate 2, no entry to the terrace." It does not lock any door. It only tells well-behaved visitors where they are welcome.

Reading a robots.txt file line by line

Here is a normal, healthy file for a small business site:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://yourwebsite.in/sitemap.xml

Four ideas cover almost everything you will ever see:

• User-agent names the bot the rule is for. A star means "every bot".
• Disallow names a folder or page the bot should not read. Here it is the WordPress login area, which nobody needs in search results.
• Allow makes an exception inside a blocked folder.
• Sitemap points to the list of your pages, so bots can find them faster. Our XML sitemap guide explains that file.

Now compare it with the file that sinks the furniture showroom:

User-agent: *
Disallow: /

That lone slash means "the whole site". Developers add it while a site is being built, so the half-finished version does not appear on Google. Then launch day arrives, everybody is busy, and the line stays.

Five things people believe about robots.txt that are wrong

"Blocking a page in robots.txt removes it from Google." It does not. It stops Google reading the page, but if other sites link to it, the address can still show up in results with no description. To keep a page out of Google, the page needs a noindex tag instead, and Google must be allowed to read the page to see that tag. The meta robots tag generator builds the tag and tells you in plain English what it will do.

"It keeps private pages safe." No. The file is public. Anyone can type your address followed by /robots.txt and read it. Listing a folder of customer invoices there is like putting up a sign that says "valuables in this room". Private pages need a password.

"Every site must block lots of things." Most small business sites need almost nothing blocked. A ten-page website for a dental clinic has no reason to hide any of those ten pages.

"If I have no robots.txt, Google will not read my site." The opposite. No file means no restrictions, and bots read everything they can find.

"Bots always obey it." The big, honest ones do: Google, Bing and the main AI companies. Spam bots and scrapers ignore it completely. Robots.txt is a request, not a wall.

What a small business should and should not block

A short, safe list of things worth blocking:

1. Admin and login areas, such as the WordPress admin folder.
2. Cart, checkout and "my account" pages on an online shop.
3. Internal search result pages, which can create thousands of thin, near-identical addresses.
4. Test or staging folders that should never have been public.

And things you should never block:

1. Your CSS and JavaScript files. Google needs them to see the page the way a customer on a phone does.
2. Your images, if you want them in Google Images.
3. Any page you want customers to find: services, products, contact, blog.
4. The whole site, for any reason, once it is live.

If you are unsure about a line, leave the page open. An extra page in Google rarely hurts. A missing one often does.

AI bots: should you let them in?

This is the new question. ChatGPT, Gemini, Claude and Perplexity all send their own bots, with names like GPTBot and PerplexityBot. You can allow or block each one separately in robots.txt.

For a local business, our view is simple: let them in. When someone asks an AI assistant "which bakery in Jaipur does eggless custom cakes", you want your site to be one of the pages it has read. Blocking AI bots made sense for news publishers worried about their articles being copied. A shop that wants customers has the opposite goal.

The AI crawler checker shows which of these bots your site currently allows. Our longer guide on making your business easier to find in ChatGPT covers what else helps. You may also hear about a separate file called llms.txt; our guide to whether your website needs an llms.txt file explains how it differs from robots.txt and why it is optional.

How to check your own file in five minutes

1. Open your browser and type your website address followed by /robots.txt. If you see a short text page, you have one. If you get a "not found" page, you have none, which is fine.
2. Look for Disallow: / on a line by itself. If it is there under User-agent: *, your site is hidden from search. That is the emergency case.
3. Pick your three most important pages and test each one. The robots.txt tester tells you whether Google, Bing or an AI bot is allowed to read a page, and which exact line makes the decision.
4. Check there is a Sitemap line. If not, add one.
5. If the file is a mess, start again. The robots.txt generator makes a clean one you can copy.

On WordPress, plugins such as Yoast and Rank Math let you edit the file from the dashboard. On Wix and Shopify, the platform writes it for you and the defaults are sensible. On a hand-built site, the file sits in the main folder next to your home page.

One more WordPress trap: in Settings, under Reading, there is a tick box called "Discourage search engines from indexing this site". It does the same damage as the bad Disallow line. Make sure it is not ticked.

When robots.txt is not the problem

If your pages are still missing from Google after the file is clean, the cause is somewhere else. The usual suspects are a noindex tag, no links pointing to the page, or a site so new that Google has not visited yet. The Google index checker shows which of your pages Google has saved and gives a plain reason for each one it has left out. Our guide for a website not showing on Google walks through every cause in order.

It is also worth running a full check once, because robots.txt mistakes tend to travel with other leftovers from a redesign, such as broken links and missing redirects. The SEO audit guide shows what to look at, and the guide to broken links and redirects covers the clean-up.

Frequently asked questions

Do I need a robots.txt file at all?

Not strictly. A site with no file is treated as fully open, which suits most small businesses. It is still worth having a simple one, mainly for the Sitemap line and to keep login and cart pages out of the way.

How long does Google take to notice a change in robots.txt?

Google usually re-reads the file within a day. Pages that were blocked can take longer to return to search results, from a few days to a few weeks, because Google has to visit each page again.

Can robots.txt improve my rankings?

Not directly. It cannot push a page higher. What it can do is stop a mistake from pushing you out completely, and on large sites it stops Google wasting time on useless pages.

Is robots.txt the same as a noindex tag?

No. Robots.txt controls whether a bot may read a page. Noindex controls whether a page may appear in results. Using both on the same page cancels out, because a blocked bot never gets to see the noindex instruction.