Pick training only, all AI bots or your own list and copy the robots.txt lines, with each bot's purpose from its operator's own page where there is one. Nothing is uploaded.
To block AI crawlers, add one User-agent group per bot to your robots.txt and give each Disallow: /. Block AI training writes that for 7 bots that their operators describe as training or building datasets: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, meta-externalagent and Amazonbot. With comments on, that is 27 lines. Bytespider is not in that preset, because its operator documents nothing; tick it under Pick bots if you want it. Block all AI bots writes all 17 bots in the list, including Bytespider, the search bots and the user-request fetchers. Pick bots lets you tick your own set, Allow all writes Allow: / for each, and a list of paths such as /private/, /drafts/ turns Disallow: / into Disallow lines for just those paths. A bot that has its own group ignores the rules under User-agent: *, so add these groups to the file and do not replace it.
The operators separate crawlers by purpose, as of October 2026. OpenAI lists GPTBot for training, OAI-SearchBot for ChatGPT search and ChatGPT-User for pages visited when a user asks. Anthropic lists ClaudeBot for training data, Claude-SearchBot for search quality and Claude-User for user questions. Perplexity says PerplexityBot is for search results and not model training, and Perplexity-User supports user actions. Amazon lists Amazonbot, which may train Amazon AI models, plus Amzn-SearchBot and Amzn-User. Blocking a search bot can remove your pages from that product's answers, so the training preset leaves the search and user bots alone.
Google-Extended and Applebot-Extended are robots.txt controls, not crawlers. Google says Google-Extended has no separate user agent and does not affect inclusion or ranking in Google Search; it manages whether content Google crawls can be used for Gemini training and grounding. Apple says Applebot-Extended does not crawl pages and controls whether content Applebot crawled is used to train Apple's foundation models. Blocking either one therefore changes use of content that was already crawled and does not stop the regular Google or Apple crawler.
robots.txt is a request that a crawler may or may not follow. Several operators say so for fetchers started by a person: OpenAI says robots.txt rules may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores robots.txt, Meta says meta-externalfetcher may bypass it and Amazon says Amzn-User may not follow all directives. The table shows each statement next to the bot. Bytespider is listed because third parties report it as a training crawler, but no ByteDance documentation page was found, so its purpose is marked as not documented by the operator and nothing about it here is confirmed by ByteDance. To enforce a block, use your server or CDN rules, which this page does not write. Bot names change, so check the operator pages before you rely on the list.
Add a group with User-agent: GPTBot and Disallow: / on the next line. OpenAI documents GPTBot as the crawler for training content, and its robots.txt setting controls it. This page writes that group for you, with 6 other training bots, when Block AI training is chosen.
Add User-agent: ClaudeBot and Disallow: / to robots.txt in the top-level directory of each subdomain. Anthropic says its bots honor robots.txt and lists ClaudeBot for training, Claude-SearchBot for search and Claude-User for user questions.
No. Google says Google-Extended does not affect a site's inclusion or ranking in Google Search. It only controls whether content Google crawls may be used for Gemini training and grounding. Googlebot is a separate user agent.
OpenAI states that because ChatGPT-User actions are started by a user, robots.txt rules may not apply. The same kind of statement exists for Perplexity-User, meta-externalfetcher and Amzn-User. Block by server or CDN rules if you need to refuse them.
Only if you accept not appearing in that product's answers. OpenAI says sites that opt out of OAI-SearchBot will not appear in ChatGPT search answers, though they can still show as navigational links. That is why the training preset leaves search bots alone.
No. The options are processed in your browser and not uploaded. Nothing you choose or type is saved on this device either.