Upgrowly
← Blog

Should You Block GPTBot and Other AI Crawlers? A Plain Guide

By Neha

Most small businesses should not block AI crawlers across the board. Blocking a training bot such as GPTBot keeps your pages out of future model training but does not remove you from ChatGPT's answers. Blocking a search bot such as OAI-SearchBot or PerplexityBot does remove you from those answers, and that can cost a business customers.

So the real question is not "block or allow AI?" but "which bot, for which job?". Each AI company now runs separate bots for training, for search and for fetching a page when a user asks, and each one obeys its own line in robots.txt. This guide explains the difference using each company's own documentation, checked on 15 November 2026, and gives you ready robots.txt sections to copy.

Short answer: Allow the search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) if you want AI assistants to find and link your business. Blocking the training bots (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot) is a fair choice if you publish original content you do not want used for training; OpenAI and Google both say their training controls are separate from search. Google-Extended does not control AI Overviews. And robots.txt is a request, not a lock.

What are AI crawlers, and why are there so many?

An AI crawler is a program an AI company sends to read web pages, and most companies now run two or three of them, each with a different job.

Before reading your site, a well-behaved crawler checks your robots.txt, the small public file at yoursite.in/robots.txt that lists which bots may read which pages. If robots.txt is new to you, our robots.txt guide explains it line by line.

The jobs fall into three groups:

  • Training bots collect pages to build future AI models. Blocking them is about whether your writing becomes training material.
  • Search bots build the index an assistant searches when it answers a question with sources. Blocking them is about whether you can be cited and linked.
  • User-triggered fetchers open one page because a person asked the assistant to, for example by pasting your link into a chat. The companies say robots.txt may not apply to these, because a human asked for the visit.

Mixing these up is the most common mistake. A business owner reads "block AI bots to protect your content", switches on a blanket block, and quietly disappears from ChatGPT and Perplexity answers in their city.

Which bot does what? The documented list

This table uses only what each company says on its own crawler page; anything not documented is left out.

BotCompanyJob, as the company describes itIf you block it
GPTBotOpenAIUsed "to make our generative AI foundation models more useful and safe"Your content "should not be used in training" OpenAI's foundation models
OAI-SearchBotOpenAISearch for ChatGPTYou "will not be shown in ChatGPT search answers, though can still appear as navigational links"
ChatGPT-UserOpenAIVisits a page when a user asks ChatGPT a questionOpenAI says robots.txt rules "may not apply"; not used to decide what appears in search
ClaudeBotAnthropicTrainingYour future pages "should be excluded from our AI model training datasets"
Claude-SearchBotAnthropicIndexing for Claude's searchAnthropic says it "may reduce your site's visibility and accuracy in user search results"
Claude-UserAnthropicFetches pages for Claude usersStops Claude retrieving your page when a user asks
PerplexityBotPerplexitySurfaces and links sites in Perplexity's results; "not used to crawl content for AI foundation models"You drop out of Perplexity's results
Perplexity-UserPerplexityVisits a page when a user's question needs itPerplexity says it "generally ignores robots.txt rules"
Google-ExtendedGoogleControls use of your content for Gemini training and groundingNo effect on Google Search inclusion or ranking
Applebot-ExtendedAppleControls use of your content to train Apple's foundation modelsApple says your site stays discoverable in Spotlight, Siri and Safari
CCBotCommon CrawlBuilds Common Crawl's open web archiveYou stay out of its new crawls

Sources: OpenAI's crawler overview, Anthropic's crawler help page, Perplexity's bot documentation, Google's list of common crawlers, Apple's Applebot page and Common Crawl's CCBot page.

OpenAI states the key point plainly: "Each setting is independent of the others", so you can allow OAI-SearchBot while disallowing GPTBot.

Does blocking GPTBot remove you from ChatGPT?

No. OpenAI documents that GPTBot is for training and OAI-SearchBot is for search, and that the two settings are independent.

If you block GPTBot, OpenAI says that tells it your content should not be used to train its foundation models. Your pages can still be found by OAI-SearchBot and shown, with a link, when someone asks ChatGPT a question that needs a web search. Our guide to how ChatGPT recommends local businesses explains that search process step by step.

If you block OAI-SearchBot, the effect is far bigger. OpenAI says opted-out sites "will not be shown in ChatGPT search answers", apart from bare navigational links. For a dentist in Indore or a packers-and-movers firm in Gurugram, that means dropping out of an assistant your customers may ask before they call anyone.

Two smaller documented details are worth knowing:

  1. OpenAI says it can take about 24 hours after you change robots.txt for its search systems to adjust.
  2. ChatGPT-User, the fetcher used when a person asks ChatGPT to open a page, "is not used for crawling the web in an automatic fashion", and OpenAI says robots.txt rules may not apply to it because a user started the action.

What is not documented: how much weight a page gets in an answer, or whether allowing GPTBot today helps you appear in future answers. Anyone who promises either is guessing.

Does Google-Extended stop AI Overviews?

No. Google says Google-Extended controls use of your content for Gemini training and grounding, and that it does not affect Google Search; AI Overviews are part of Search.

This catches many people out. Google's crawler list says Google-Extended manages whether content Google crawls may be used for training future Gemini models and for grounding answers in the Gemini apps and Vertex AI. It adds: "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal".

For AI features inside Search, Google's AI features page says robots.txt rules for Googlebot are "the control for site owners to manage access to how their sites are crawled for Search", and that to limit what is shown you use "nosnippet, data-nosnippet, max-snippet, or noindex controls". Those controls also shrink or remove your ordinary Google listing, so they are rarely worth it for a business. Our guide to AI Overviews eligibility covers them, and our AI Mode guide covers the separate Search Console setting for generative AI features.

One thing to watch: the same crawler page notes that Google "intends to support the AIPREF shared machine-readable training preference signal" in early 2027. That is a stated plan, not a live feature, so for now Google-Extended is the token to use.

Should a small business block AI crawlers?

For most Indian small businesses, no: allow the search bots, and treat the training bots as a separate, optional decision.

A local business wants to be found. When someone in Kochi asks ChatGPT for "a CA who files GST returns for small shops near Kakkanad", you want your site to be one of the pages it can read and link. That is the same advice our guide to showing up in ChatGPT and AI search and our Perplexity SEO guide give, and nothing here changes it.

Blocking training bots is a reasonable choice in some cases:

Your situationSensible choice
Clinic, salon, shop, restaurant, local serviceAllow everything. Your pages describe services, not content you sell.
Freelancer or agency with a portfolio and blogAllow search bots; training bots are your call.
Coaching institute selling notes or test seriesAllow search bots; block training bots, and keep paid material behind a login.
Publisher, news site or paid newsletterBlock training bots; decide on search bots by how much traffic they send you.
Anything private, such as invoices or client filesDo not rely on robots.txt at all. Use a password.

The point of the table: blocking is a business decision about your content, not a security measure. Make it on purpose, and write it down so the next developer does not undo it.

Ready robots.txt sections you can copy

Pick the one that matches your decision, paste it into your robots.txt, and keep your existing Sitemap line.

One rule matters here, and the robots.txt tester follows it: a bot obeys only the single group that best matches its name. If you give GPTBot its own group, it ignores your User-agent: * rules, so repeat any folder blocks you still need inside its group.

Example 1: allow everything (the right choice for most local businesses).

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://yourwebsite.in/sitemap.xml

Example 2: allow AI search, block AI training.

# Training bots: keep my content out of model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

# Search bots: allowed, so assistants can cite and link me
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

# Everyone else, including Googlebot and Bingbot
User-agent: *
Disallow: /wp-admin/

Sitemap: https://yourwebsite.in/sitemap.xml

Example 3: keep one folder, such as paid study notes, out of AI training only.

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /notes/

User-agent: *
Disallow: /wp-admin/

Example 4: block all AI bots (publishers only; this removes you from AI search answers).

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

Example 4 does not touch Googlebot, so your normal Google listing and AI Overviews are unaffected. It also cannot stop user-triggered fetchers such as ChatGPT-User or Perplexity-User, which their companies say may not follow robots.txt. If writing the whole file feels risky, the robots.txt generator makes a clean one with a clear choice about AI bots.

Is robots.txt enough to stop AI bots?

No. Robots.txt is a public request that honest bots follow; it cannot force anyone to obey.

Google's own introduction to robots.txt says the file's instructions "cannot enforce crawler behavior to your site; it's up to the crawler to obey them". The companies above say their main crawlers follow it, but scrapers and badly built bots may not. Common Crawl also warns that some crawlers falsely identify themselves as CCBot.

So, in practice:

  1. Truly private pages need a password, not a Disallow line. Listing a folder in robots.txt actually tells everyone it exists.
  2. Firewalls can block bots by accident. Security plugins and "block AI bots" switches at hosting or CDN providers can shut out search bots without anyone editing robots.txt. Check those settings too.
  3. Blocking stops future crawling only. It does not pull pages out of models that were already trained, or out of archives already published.

How to check what your site allows today

Run a free check, read which bots are blocked, and compare that with what you actually want.

  1. Run the AI crawler checker. It reads your robots.txt and shows, for 13 AI crawlers from OpenAI, Anthropic, Google, Perplexity, Apple, Meta, Amazon, ByteDance and Common Crawl, whether each may read your home page, the line that decided it and whether the bot is for training or for search.
  2. If a search bot is blocked and you did not mean it, find the line, or the plugin or CDN setting that writes it, and remove it.
  3. Test important folders in the robots.txt tester: pick a bot and a page, and it shows the exact line that decides.
  4. For a wider view, the AI readiness checker scores how easy your whole site is for AI tools to read, including crawler access, llms.txt, your sitemap and structured data.
  5. Check your firewall or CDN bot settings once, because the tools above read robots.txt only.

You may also hear that an llms.txt file can block AI. It cannot; our llms.txt guide explains what that file is for.

Frequently asked questions

Does blocking GPTBot stop ChatGPT from mentioning my business?

No. OpenAI says GPTBot is for training and OAI-SearchBot is for search, and the two settings are independent. Block GPTBot and ChatGPT can still find and link your site through search, as long as OAI-SearchBot is allowed.

Will blocking AI crawlers hurt my Google rankings?

No. Rules for AI bots such as GPTBot or Google-Extended are separate from the rules for Googlebot, which is what Google Search uses. Google says Google-Extended is not a ranking signal and does not affect inclusion in Search. Only rules for Googlebot itself affect Google Search.

How do I block AI crawlers on WordPress?

Edit robots.txt through your SEO plugin, such as Yoast or Rank Math, and add the groups from the examples above. Then run the AI crawler checker to confirm the result, and look at any security plugin that may add its own bot rules.

Can AI companies still use my content if I block their crawlers?

Blocking stops future crawling by bots that obey robots.txt. It does not remove content collected before, and it cannot stop bots that ignore the file. User-triggered fetchers such as ChatGPT-User and Perplexity-User may not follow robots.txt at all, according to their companies.

How long does a robots.txt change take to work?

OpenAI says about 24 hours for ChatGPT's search, and Perplexity says up to 24 hours for its systems. Others do not publish a figure, so allow a few days and check again.

Should I block CCBot?

Only if you want to keep your pages out of Common Crawl's open web archive. A local business usually has no reason to. Common Crawl says its crawler checks robots.txt first, and documents User-agent: CCBot with Disallow: / as the way to block it.