Should I Block AI Crawlers in Robots.txt?

Updated 19 October 2026 · 10 min read · By Matthew Bertram

Google works differently. Its AI token is Google-Extended, and it isn't a crawler at all. Blocking it doesn't keep your pages out of AI Overviews, going by Google's documentation. A separate setting in Search Console does that.

Which AI crawler user agents can robots.txt block?

All of them, if their operators keep their word. A robots.txt group starts with a User-agent line naming a crawler, and the rules under it apply only to that crawler. These are the names that matter for AI as of October 2026.

We checked every row against the operator's own documentation on October 18, 2026. Bytespider is the thin one. The only first-party documentation we found is Toutiao's Chinese page, which never mentions AI, though Cloudflare's managed robots.txt still disallows it alongside known AI crawlers.

What's the difference between a training crawler and a search crawler?

What happens after the fetch. A training crawler collects pages to train future models, which then answer from what they learned, with no link to you. A search crawler builds an index that an AI engine searches when someone asks, and that's where citations come from.

OpenAI and Anthropic each run both kinds. So do Amazon and Meta. Perplexity documents no training crawler: PerplexityBot "is not used to crawl content for AI foundation models." Google and Apple crawl once and let a token decide how the data gets used, so there's no training crawler to block, only a token to set.

Treat every agent as a separate switch. OpenAI says each setting "is independent of the others," and Perplexity and Amazon say much the same.

What's the difference between GPTBot and ChatGPT-User?

Who started the visit. GPTBot crawls automatically to collect training data. ChatGPT-User shows up when a person's request sends ChatGPT to a page. OpenAI is plain about what that means for your file: "Because these actions are initiated by a user, robots.txt rules may not apply." Search belongs to a third agent, OAI-SearchBot.

Should you block GPTBot?

Block it if you object to training. That's all it controls. OpenAI's documentation says "Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models." What you give up is the training layer: what future models know about your business when they answer without searching. That layer runs on a slower clock than search.

Our own view at EWR Digital is to allow training crawlers on sites that want to be known, and to ask for credit in the site's terms instead of blocking. A model that has read your pages can describe you accurately, while one that hasn't can only repeat what others wrote about you.

We can't prove that view pays off. We know of no published measurement showing that being in training data raises a site's citation rate, and we haven't run one either.

Blocking works forward, not backward. Anthropic says a ClaudeBot block signals that "the site's future materials should be excluded from our AI model training datasets."

Does blocking GPTBot remove your site from ChatGPT?

No. ChatGPT's search answers come from OAI-SearchBot, and OpenAI treats the two as separate switches. Blocking OAI-SearchBot is what takes you out: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links."

What is Google-Extended?

A token, not a crawler. Google says Google-Extended "doesn't have a separate HTTP request user agent string," so it never shows up in your server logs. Google crawls with its usual agents and checks the token before using your content for the purposes it covers.

Those purposes, in Google's words: training "future generations of Gemini models that power Gemini Apps and Vertex AI API for Gemini," and grounding, which Google defines as "providing content from the Google Search index to the model at prompt time to improve factuality and relevancy," in Gemini Apps and Grounding with Google Search on Vertex AI.

Google's on-record position is that Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."

Does blocking Google-Extended hurt your rankings or AI Overview citations?

Neither, by Google's account. AI Overviews and AI Mode are part of Google Search. Google's page on AI features names robots.txt rules for Googlebot as the control for them, and it points to Google-Extended only for limiting "AI training and grounding in some of Google's other systems."

To leave AI Overviews and AI Mode without leaving Search, use the Search generative AI control in Search Console, available to every site since August 31, 2026. Google says sites set to exclude "won't receive any traffic or impressions from these features," but the control "doesn't affect AI training," so for the models behind those features Google points back to Google-Extended.

That's Google's on-record position. No outside test we know of has checked it.

Staying eligible isn't the same as being cited. Our guide to how AI Overviews choose their sources covers that second step.

How do you block AI crawlers in robots.txt?

Give each agent a group of its own. Leave Googlebot alone.

The file lives at the root of each host, so www.example.com/robots.txt doesn't cover blog.example.com. Anthropic's instructions say to add its rules "for every subdomain that you wish to opt out from." Never add a broad User-agent: * group that disallows everything. It blocks every crawler without a group of its own, and on most sites that includes Googlebot.

To decline training:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: CCBot
Disallow: /

The Amazonbot and meta-externalagent lines block more than training, as the table shows. Bytespider is left out on purpose, since Toutiao documents it as a search crawler. Add it if Cloudflare's AI label is enough for you.

To stay out of AI search as well, add these groups, which include the user fetchers even though several operators don't promise those will obey. The table leaves out Amzn-User and two Meta agents: meta-webindexer, which Meta says improves Meta AI's search results, and meta-externalfetcher, which fetches links for users and which Meta says "may bypass robots.txt rules." Google's AI Overviews and AI Mode need the Search Console control instead.

User-agent: OAI-SearchBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: Claude-SearchBot
Disallow: /

User-agent: Claude-User
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Perplexity-User
Disallow: /

User-agent: Amzn-SearchBot
Disallow: /

User-agent: Amzn-User
Disallow: /

User-agent: meta-webindexer
Disallow: /

User-agent: meta-externalfetcher
Disallow: /

To allow everything:

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

A missing file allows everything too, unless your CDN adds rules of its own.

Two operators borrow rules written for other crawlers. If your file names Googlebot but not Applebot, Apple's crawler follows the Googlebot rules. If it doesn't name Amzn-SearchBot but allows other search bots, Amazon says Amzn-SearchBot follows their rules.

Do AI companies respect robots.txt?

Mostly, and nothing forces them to. Cloudflare's documentation is blunt: "robots.txt compliance is voluntary. The file expresses your preferences, but it does not prevent crawlers from accessing your content at a technical level."

The gap is the user-triggered fetcher. Because a person asked for the page, most operators that run such a fetcher give it looser rules. OpenAI says robots.txt rules don't necessarily apply to ChatGPT-User. Perplexity says its user fetcher "generally ignores robots.txt rules." Amazon says Amzn-User "may not follow all robots.txt directives." Meta says its user fetcher can bypass robots.txt as well.

Anthropic is the exception. It says its bots respect "do not crawl" signals "by honoring industry standard directives in robots.txt," and it says site owners can control Claude-User through robots.txt too.

Then there's a public dispute. In August 2025, Cloudflare reported that Perplexity was using "a generic browser intended to impersonate Google Chrome on macOS when their declared crawler was blocked." Perplexity replied that Cloudflare had confused it with millions of daily requests from BrowserBase, a cloud browser service it says it uses only occasionally. Cloudflare ran the same test on ChatGPT and found that "ChatGPT-User fetched the robots file and stopped crawling when it was disallowed." We can't referee that dispute from outside.

Real enforcement takes a firewall or CDN rule. Cloudflare's advice is to "use robots.txt to express your preferences and AI Crawl Control to enforce crawler-specific blocks."

Does Cloudflare block AI crawlers by default?

For some sites, yes. Cloudflare announced in July 2025 that it was "changing the default to block AI crawlers unless they pay creators for their content." Its current documentation is more specific: from September 15, 2026, new domains get defaults that block bots classified as Training or Agent on pages that display ads, while Search stays allowed. Its page on blocking AI bots says the training block also catches crawlers that mix search and training, but its managed robots.txt page says mixed-use crawlers are excluded from the edge block that setting turns on. Check which one your zone uses.

Separately, its managed robots.txt setting adds Disallow rules for known AI crawlers to the top of your existing file, or serves a file of its own if you have none. Free-plan domains with no file and the setting off get Cloudflare's Content Signals Policy, which defines categories such as ai-train without stating a preference.

How do you test whether a bot is blocked?

Start with the live file. Open your robots.txt in a browser and look for two things: a group naming the bot, and a broad User-agent: * group that disallows everything. A CDN can add rules your server never wrote, as Cloudflare's managed setting does, so read the file as it's served.

Then check the layer in front of it. A firewall rule can turn a crawler away before it reads robots.txt. Google asks site owners to make sure crawling is allowed "by any CDN or hosting infrastructure," and Perplexity suggests allowlisting its bots so a firewall doesn't stop them.

Server logs show what arrived. They don't prove who sent it. Google's crawler documentation warns that the HTTP user agent string "can be spoofed," and Common Crawl says it is "aware of crawlers falsely identifying themselves as CCBot." Most operators publish a way to check. Some list their IP addresses. Toutiao says genuine Bytespider hostnames end in bytedance.com.

Don't test by sending requests that pretend to be a crawler. About fifteen test requests in a few seconds got our own IP address banned by a client's firewall in August 2026, and the ban nearly went into our report as proof that the site blocked AI crawlers. A spoofed request tests your firewall's reaction to a name, not the real crawler arriving from its real addresses.

FAQ

Is llms.txt the same as robots.txt?

No. Llms.txt is a markdown file at a site's root that summarizes the site and links its most important pages. Ahrefs, in a study of 137,210 domains, puts the difference plainly: "it is not a robots.txt-style directive: it controls nothing and blocks nothing."

Little reads it, either. In that study, 97% of the llms.txt files got no requests at all in May 2026. Slackbot alone fetched llms.txt files more than PerplexityBot did. SE Ranking found no correlation between llms.txt and AI citations across nearly 300,000 domains. Google's on-record position is that Search ignores such files, which "will neither harm nor help your site's visibility or rankings in Google Search."

None of the crawler pages linked in this guide says its crawler reads llms.txt. Our guide to AI SEO has the full verdict.

How long does a change take to work?

About a day for robots.txt. OpenAI says its search systems "can take ~24 hours" to adjust after a robots.txt update. Perplexity and Amazon give similar figures, though Amazon adds that its crawlers "fetch host-level robots.txt files or use a cached copy from the last 30 days." Google says a change to its Search Console control "generally takes a few days."

Does blocking CCBot keep your pages out of AI models?

It keeps them out of future crawls, which is all a robots.txt rule can do. Common Crawl describes its archive as "an open repository of web crawl data that is universally accessible and analyzable by anyone." Its CCBot page also points to an opt-out registry, which we haven't tested.

Want to know which AI crawlers your site lets in today, and whether your firewall agrees with your robots.txt? Book a discovery call with EWR Digital →

Keep reading: How Do I Know If ChatGPT or Google AI Overviews Are Citing My Site? · What's the Difference Between GEO and AEO? · What Is AI SEO, and How Is It Different From Traditional SEO?

Found an AI crawler in your server logs that isn't in this table, or one ignoring your robots.txt? Send us the log lines, with IP addresses if you can. We'll verify them the way each operator says to and, with your permission, add what we confirm to this guide.