Which AI crawlers to allow, and what blocking them actually does
There is a real difference between the crawler that trains a model and the crawler that fetches a page to answer a question. Block the wrong one and you remove yourself from AI search results.
A lot of sites added AI crawlers to robots.txt in a hurry, usually after a conversation about training data. Some of those blocks are doing exactly what was intended. Others quietly removed the site from AI search results, which was not the intention at all.
The distinction that matters is not "AI" versus "not AI". It is training versus retrieval.
Two different jobs
A training crawler collects pages to build a model. Blocking it is a content licensing decision. Reasonable people land in different places on it, and the answer depends on what your content is worth and to whom.
A retrieval crawler fetches a page because a user just asked something and the assistant is looking for an answer to cite. Blocking it is not a licensing decision. It is opting out of being an answer.
They are different crawlers with different names, and robots.txt treats them separately.
The names
For retrieval, the ones that matter today are OAI-SearchBot for OpenAI's search features, PerplexityBot, and ClaudeBot. GPTBot sits closest to the line: OpenAI documents it as their web crawler, and blocking it is the broadest block you can apply to that ecosystem.
Google-Extended is worth understanding on its own terms, because it confuses people. It is not a crawler. It is a control that tells Google whether your already-crawled content can be used for Gemini training and grounding. Blocking it does not remove you from Google Search, and it does not stop Googlebot.
CCBot belongs to Common Crawl, which is a public dataset that many models are trained from. Blocking it is the clearest single lever on training use.
A defensible default
If you want to be found and cited, and you are not precious about training:
Allow everything. A plain `User-agent: *` with `Allow: /` already does this. You do not need to name AI crawlers to permit them, and adding a list of Allow rules for them changes nothing functionally.
If you want to be found but not used for training, allow the retrieval crawlers by name and disallow CCBot and Google-Extended. That is the position most publishers end up at.
If you block everything, understand precisely what you bought: your pages will not appear as sources in AI answers, and that is now a meaningful share of how people find suppliers.
The thing robots.txt cannot do
Some fetchers are user-triggered. When a person pastes your URL into an assistant and asks it to read the page, that request is made on the user's behalf, and by long-standing convention user-triggered fetches do not consult robots.txt. ChatGPT-User works this way, and so do several of Google's agentic fetchers.
This is not a loophole anyone is exploiting. It is the same reason your browser does not read robots.txt before loading a page you asked for. If you need to stop those requests, robots.txt is the wrong instrument and server-side access control is the right one.
Checking what you currently say
Open yoursite.com/robots.txt and read it. Look for any section naming one of the crawlers above, and check whether it ends in `Disallow: /`.
The most common finding is not a deliberate block. It is a rule somebody copied from a blog post in 2023, which blocked GPTBot before AI search existed as a thing to be found in, and which nobody has revisited since.
Auditing exactly this, what your site currently allows, what it should, and why, is part of the AI search visibility work we do for clients.
Tell us what needs building
Describe the problem, the system you have today, and the date it needs to be live. We reply with a written scope and a fixed price.