robots.txt Does Not Prohibit AI Training: Which Robots to Block to Keep the Site in Neural Network Responses / Habr

ChatGPT, Claude, and Gemini use different bots for training and answering

OpenAI and Anthropic each have three bots. The first collects text for training, the second searches for pages for the service's search function, and the third visits a page when a user requests it.

OpenAI says these settings are independent. You can allow OAI-SearchBot and remain in ChatGPT search, but block GPTBot so that the text is not used for training. ChatGPT-User comes in response to user action, and according to OpenAI, robots.txt rules may not apply to it. The site is removed from ChatGPT search via OAI-SearchBot.Anthropic warns that if Claude-User is disabled, Claude will not be able to use your page in its response to a user query, and the site’s visibility may decrease.

Google has separated training from Search with the Google-Extended token. The token has no crawler of its own. It determines whether pages crawled by Google will be used to train future Gemini models and for grounding, that is, for supplying pages to the AI model while generating a response. Google-Extended does not affect inclusion in Search (Google documentation). Applebot-Extended works similarly: it does not crawl pages itself and only determines whether AI models can be trained on Applebot data. A site that blocks it remains in search results (Apple Support). Common Crawl, an open archive of downloaded pages, crawls websites with the CCBot bot and stops if this token is blocked (Common Crawl FAQ).

In Russia, the options are more limited. To prevent pages from appearing in Alice AI’s quick answers in Search, Yandex Webmaster recommends blocking YandexAdditional or YandexAdditionalBot. The change takes 2–14 days to take effect. According to the seodb reference, Yandex has no separate token exclusively for training: AI answers draw pages from the main index, so blocking YandexBot removes the site from both search and those answers. According to Flowstat’s observation as of October 2026, Sber has published neither the name of the GigaChat search crawler nor any robots.txt guidelines. It is not yet possible to name this crawler in the file.

Which robots.txt lines block AI training, and which remove a site from answers

It is not possible to block a site from AI entirely using robots.txt. If you try, you will most likely lose ground in the answers generated by neural networks, while content that has already been downloaded will remain in the training data—that is, in the set of texts used to train AI models.Training and responses need to be treated separately: crawlers used for training can be blocked if that is important to the business, while search crawlers from AI services and crawlers that arrive in response to a user request should be left open.

Exclude from training while remaining in responses:

``` User-agent: GPTBot User-agent: ClaudeBot User-agent: CCBot User-agent: Applebot-Extended Disallow: / ```

The first group asks that pages not be downloaded for training by the OpenAI, Anthropic, Common Crawl, and Apple crawlers. The second group is there for clarity: without a prohibition, these crawlers are allowed anyway. Google Search does not act on such a file and does not remove the site from ChatGPT and Claude search. It has no effect on pages these crawlers have already fetched and cannot stop crawlers that do not read robots.txt.

I did not include Google-Extended in this list. It blocks not only Gemini training but also grounding, so the prohibition may cost the site a place in Gemini responses. It should only be added if preventing training is more important to the business than those responses.

Do not appear in responses:

``` User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: Claude-SearchBot User-agent: Claude-User User-agent: YandexAdditional User-agent: YandexAdditionalBot Disallow: / ```

The cost of such a file is that the site will no longer appear in the responses of these services.With ChatGPT-User, the result isn’t guaranteed: OpenAI notes that the rules may not apply to it.

Which AI bots bypass robots.txt and how to verify this from server logs

Not everyone obeys the file. In August 2025, Cloudflare showed that when blocked, Perplexity changes its user agent and ASN (the network number from which the traffic originates) and sometimes does not even request robots.txt. The test was conducted on new domains with a blanket ban in robots.txt and rules specifically targeting PerplexityBot and Perplexity-User. Perplexity still obtained content by pretending to be the Chrome browser on macOS. According to Cloudflare, this covert circumvention was occurring across tens of thousands of domains, with millions of requests per day. Perplexity was removed from Cloudflare’s list of verified bots. Interestingly, the most compliant bot in the same test was ChatGPT-User: it requested robots.txt and stopped crawling, even though OpenAI itself warns that the rules may not apply to it.

You can check your own crawlers using the access log. Filter for lines containing the tokens and see which pages they requested after the appearance of Disallow:

`` grep -E "GPTBot|ClaudeBot|CCBot|OAI-SearchBot|ChatGPT-User" access.log ``

Each line contains the request time, path, server response code, and user-agent. A successful response to a request for a disallowed section after the appearance of Disallow means the bot downloaded the page, while a rejection or error means it did not receive the content. Check whether the bot requested /robots.txt and whether it continued accessing disallowed sections. You cannot identify spoofing by the token, so look for requests with the user-agent of an ordinary browser that are hitting the same pages at the same cadence.How GPTBot, ClaudeBot, and the others appear in logs is explained in an article about which AI bots actually crawl websites and what they request.

What, besides robots.txt, blocks content from AI models: server-side blocking, metatags, and TDMRep

Neither robots.txt nor user-agent blocking helps against a robot that calls itself a browser. The developer should be asked to check the protection against automated requests on the server or through the website’s content delivery service (a CDN such as Cloudflare): it distinguishes robots from customers using regular browsers based on traffic characteristics, such as the network from which the request originates and the cadence of requests. In the Cloudflare test, it worked: when the disguised crawler was blocked as well, Perplexity’s responses became less specific, with no details from the source page. IP blocking instead of using the Anthropic file can backfire: it prevents the robot from reading robots.txt, and the denial may not take effect.

• The nosnippet metatag. Apple provides it to prevent specific content from appearing in Siri and Apple Search answers to general questions.

• TDMRep. This W3C Community Group protocol (it is not a W3C standard) provides a way to reserve rights to TDM—the automated analysis of texts and data: opt-out is specified by the tdm-reservation parameter with the value 1 (protocol description). It has legal weight in the EU: Article 4 of Directive 2019/790 permits TDM for commercial purposes as long as the rights holder has not reserved those rights, including by machine-readable means.Outside the EU, this exception does not apply.

• Pay per crawl. Since July 2025, Cloudflare has had Pay Per Crawl in closed beta: a website can charge AI crawlers for each access to content.

Anthropic already honors a blanket ban on AI training in robots.txt, and Google promises to do so starting in 2027.

Starting October 30, 2026, Anthropic recognizes a machine-readable signal prohibiting AI training. The blanket ban looks like this:

`` User-agent: __usage_ai-training Disallow: / ``

Google plans to support the same AIPREF signal in early 2027. According to Lab Leaders, Google may treat such a ban as a Google-Extended ban, in which case the blanket directive could also cost a website its Gemini responses. The IETF is preparing the standard: a draft co-authored by Ilyes will add the Content-Usage rule and a same-named HTTP header to robots.txt, but the terminology has not yet been agreed upon.Until the standard has been adopted, training and responses must be closed separately, and each robot must be referred to by name.