An AI crawler is not one type of visitor with one purpose. Some help people discover public pages, some fetch a page for a user and some collect material for model training. Treating them all the same can block useful visibility or grant access the business never intended.
01
Visibility and training are different decisions
A marketing team may want AI services to find and cite the website. A legal, security or leadership team may want to stop company content being used to train a model. Those positions can sound incompatible, but they often describe different forms of access.
The mistake is to begin with a blanket question: should we allow AI bots? That bundles search discovery, user-requested page visits and model training into one answer. The better starting point is to decide what the business wants from each use.
A public service page may benefit from appearing when somebody asks an assistant for relevant suppliers or an explanation of a problem. That does not automatically mean the company wants every training crawler to collect the same page. Equally, blocking every unfamiliar bot may reduce visibility without making private information safer.
The useful policy is selective. Make accurate public content available for the purposes the business supports, control model training separately where providers offer that choice and protect sensitive material with real access controls.
02
Start by separating three kinds of access
AI providers use different crawler names and their definitions can change. The business decision becomes easier when each one is mapped to its purpose rather than judged by whether its name contains AI, bot or user.
Search and indexing crawlers collect public pages so a service can return links, snippets or citations. User-requested fetchers visit a page because somebody has asked an assistant or agent to use it. Training crawlers collect material that may help improve future foundation models.
There is a fourth category that should not be treated as a crawler policy at all. Private portals, draft pages, customer records and internal systems need authentication, authorisation and infrastructure controls. A line in robots.txt is not a security boundary.
The table below turns one vague AI access question into four clearer business choices.
| Access purpose | What it enables | Typical control | Main business question |
|---|---|---|---|
| Search and indexing | Links, snippets and citations in search experiences | Search-specific crawler rules plus verification where needed | Do we want useful discovery and referrals? |
| User-requested fetch | An assistant visits a public page for a current user | Product-specific fetcher behaviour and public page controls | Should agents be able to use this public information? |
| Model training | Content may help improve a future foundation model | Training-specific crawler token or provider content signal | Are we comfortable with this form of reuse? |
| Private or internal access | No public discovery should be possible | Authentication, authorisation, WAF and network controls | Would exposure create customer, legal or operational risk? |
03
OpenAI makes the split unusually clear
OpenAI documents separate controls for OAI-SearchBot and GPTBot. OAI-SearchBot supports ChatGPT search features, including links and citations. GPTBot is used for content that may help train generative AI foundation models. A website can allow the search crawler while disallowing the training crawler.
That distinction matters for a business that wants its public expertise to be discoverable but does not want to opt into model training. The two choices do not have to move together.
ChatGPT-User is different again. It can be used when a person asks ChatGPT or a custom GPT to visit a page. OpenAI says this user-initiated agent may not follow robots.txt in the same way as an automatic web crawler. That is another reason sensitive information must sit behind proper access controls rather than a crawler instruction.
If OAI-SearchBot is newly allowed, OpenAI notes that search changes can take time to appear. The sensible test is therefore not an immediate prompt repeated five minutes later. Check the published rules, monitor server logs and referrals, and allow the service time to recrawl.
04
Perplexity and Google use different controls
Perplexity documents PerplexityBot as its search crawler. It discovers and links to websites for search results and is not used to build foundation AI models. Perplexity-User fetches a page in response to a user request and, like other user-initiated fetchers, is not guaranteed to follow robots.txt.
Perplexity also advises website operators to verify both the user agent and its published IP ranges. That is useful when a firewall or web application firewall is making the decision, because a user-agent name on its own can be copied by an unrelated crawler.
Google-Extended is not a general Google Search switch. Google describes it as a control for whether content crawled by Google may help train future Gemini models or be used for grounding in Gemini products. Blocking Google-Extended does not remove a page from Google Search and does not change its search ranking.
Google also says there is no separate technical recipe or special schema required for its AI search features. The familiar foundations still matter: crawlable pages, useful original content, accurate structured data where relevant, good page experience and clear internal links.
05
Robots.txt is a policy notice, not a locked door
A robots.txt file tells compliant crawlers which paths the site owner would prefer them not to request. It is useful for managing crawler behaviour, but it does not create authentication and it does not make a public URL private.
Cloudflare makes the same distinction in its current guidance. Robots rules express a preference. Complete enforcement needs a security control such as AI Crawl Control, a web application firewall rule, authentication or restricted network access.
That distinction becomes important when a crawler does not honour the rule, imitates another user agent or reaches the site through a user-requested tool. It also matters when a CDN, reverse proxy or security service silently applies a bot policy before the request reaches the application.
Never place customer data, internal documents, unfinished proposals or administration screens on a public URL and rely on robots.txt to hide them. If the information should not be available to an unauthorised person, the application must enforce that decision.
06
Make the public surface worth crawling
Allowing a crawler does not make weak content useful. The public page still needs to answer a real question with accurate information, a clear structure and something more valuable than a rearranged summary of other websites.
Give important pages stable canonical URLs, useful titles, meaningful internal links and current information. Put the main answer in page content that people and crawlers can reach without an unnecessary interaction. Semantic headings, accessible labels and sensible HTML help search systems, assistive technology and agents understand the same page.
Structured data can clarify supported page information, but it should match what a visitor can see. Adding every possible schema type or inventing attributes will not make a page more trustworthy. Google explicitly says its AI search features do not require special AI markup.
Keep preview, staging and unfinished content away from the public surface. A crawler cannot judge that a page contains last month's draft unless the website gives it a reliable signal or prevents access. Clear publishing controls are part of AI visibility work because they determine which version of the business becomes discoverable.
07
Choose rules that fit the business
A public consultancy or service website may decide to allow recognised search crawlers, permit public user-requested access and block training crawlers where a separate control exists. That can support discovery without treating every form of reuse as the same choice.
A publisher, research company or membership business may choose a different boundary because its public content is the product. A documentation site may welcome search crawlers but exclude preview builds and private support material. An online portal holding customer records should rely on authentication and permissions regardless of which crawlers are named in robots.txt.
There is no universal allow list that suits every organisation. Even the same business may apply different rules to marketing pages, public documentation, paid content and application areas.
Write the reason next to the rule. A future administrator should be able to see whether a crawler was blocked for training, security, licensing, capacity or a temporary incident. Without that context, the next update can reverse an intentional decision by accident.
08
Implement the decision without guessing
Begin with an inventory. Read the current robots.txt file, CDN bot settings, firewall rules, application middleware and any platform-level controls. A rule in one layer may be overridden or made irrelevant by another.
Use each provider's current documentation to map crawler names to purposes. Do not copy a two-year-old block list from a forum and assume every token still means the same thing. Where a provider publishes IP ranges, verify both the user agent and the source address before granting special treatment through a firewall.
Change one policy at a time and keep the previous configuration. Test that public pages still load for ordinary visitors and the intended crawler, while restricted areas remain unavailable without authorisation. Then monitor logs, crawl volume, referral data and visible citations rather than assuming the rule worked.
Review the policy regularly. Provider names, product behaviours and CDN defaults change. The website may also add new public sections, integrations or customer areas that need a different decision from the marketing site.
09
Controlled access is better than a blanket answer
AI crawler access should be a small, documented part of the website's publishing and security policy. It is not a one-off switch that guarantees citations, and blocking a token is not a substitute for protecting the application.
Separate the purposes first. Decide whether the business wants search discovery, user-requested access and model training. Make accurate public information easy to find for the uses you support. Put private information behind controls that do not depend on a crawler behaving politely.
That approach gives marketing, security and leadership a more useful conversation. Instead of arguing about whether AI is good or bad, the business can choose exactly what it wants each part of the website to allow.
If you need help reviewing crawler access alongside the wider website, I can assess the current setup, explain the trade-offs and turn the decision into controlled technical changes your business can revisit later.
Useful questions
AI crawler access checklist
- What exact outcome do we want: search discovery, agent access or model training?
- Which public paths should be available for that purpose?
- Are preview, staging, customer and administration areas protected by authentication?
- Does robots.txt separate search and training where the provider supports it?
- Is a CDN, firewall or hosting platform silently overriding the rule?
- Are recognised bots verified using official IP information where available?
- Does the public HTML contain the main content and accessible structure?
- Are canonical URLs, internal links, metadata and the sitemap correct?
- Are crawler logs, referrals and citations being monitored?
- Is there a scheduled review for provider documentation and policy changes?
