Answers · AEO & AI Search
Which AI crawlers should an insurance agency website allow?
Last reviewed: · Strategic AI Architects Data Desk · reviewed by Mike Moore
Allow every agent that produces citations and referrals — the search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot, Bingbot) and the user-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) — because blocking any of them removes your agency from the AI answers your prospects are already reading; the training-only crawlers (GPTBot, ClaudeBot, CCBot, meta-externalagent, plus Google's Google-Extended token) carry no documented search-visibility penalty either way, and for a local agency whose entire goal is to be found, we allow those as well.
The qualifier: this is a visibility recommendation, not a universal one. A publisher whose content is the product has a real licensing argument for blocking training crawlers. An insurance agency is the opposite case — its content exists to make the agency get found and called — so blocking the bots that feed AI answers is paying a cost to defend nothing. Every user agent below is verified against the vendor's own published documentation, fetched live on August 18, 2026.
The actual user agents, verified
| User agent | Vendor | What it does | robots.txt | Our call |
|---|---|---|---|---|
| GPTBot | OpenAI | Crawls content for training generative models (token GPTBot/1.4) | Honored | Allow |
| OAI-SearchBot | OpenAI | "Used to surface websites in search results in ChatGPT's search features" (OAI-SearchBot/1.4) | Honored | Allow — this one is your ChatGPT visibility |
| ChatGPT-User | OpenAI | Fetches a page when a ChatGPT user asks about it (ChatGPT-User/1.0) | May not apply — user-initiated | Allow |
| ClaudeBot | Anthropic | Collects web content that may contribute to model training | Honored | Allow |
| Claude-User | Anthropic | Fetches pages when Claude users request web content | Honored per Anthropic's docs | Allow |
| Claude-SearchBot | Anthropic | "Navigates the web to improve search result quality" | Honored; blocking reduces search visibility | Allow |
| anthropic-ai | Anthropic (legacy) | Older token still listed in many robots.txt files; absent from Anthropic's current crawler documentation, which lists the three agents above | Harmless to list; not the operative control | Allow (and rely on the three current tokens) |
| PerplexityBot | Perplexity | Surfaces and links websites in Perplexity search results; documented as not used for training (PerplexityBot/1.0) | Honored | Allow |
| Perplexity-User | Perplexity | Visits pages to answer a user's question (Perplexity-User/1.0) | "Generally ignores" it — user-initiated | Allow |
| Google-Extended | A control token, not a crawler: governs use of crawled content for Gemini training and grounding. "Does not impact a site's inclusion in Google Search nor is it used as a ranking signal." | Honored (as a token) | Allow | |
| Bingbot | Microsoft | Builds the Bing index, which Microsoft Copilot's web answers draw from | Honored | Allow — blocking it exits Bing and Copilot at once |
| CCBot | Common Crawl | Builds the nonprofit open web corpus (CCBot/2.0) that many AI training datasets draw from | Honored | Allow |
| meta-externalagent | Meta | Crawls "for use cases such as training foundation AI models or improving products by indexing content directly" (meta-externalagent/1.1) | Honored | Allow |
| meta-externalfetcher | Meta | Fetches individual links for user-requested agentic tasks (meta-externalfetcher/1.1) | "May bypass" it — user-initiated | Allow |
The distinction that decides everything is crawl-for-answers vs. crawl-for-training. Blocking OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot, or Bingbot takes you out of the respective answer surface — that is each vendor's own documentation, not speculation. Blocking the training crawlers changes only whether future models learn from your pages. Google-Extended is the cleanest illustration: Google states outright that it has zero effect on Search inclusion or ranking, and AI Overviews eligibility runs through ordinary Googlebot indexing.
First-hand practice: every site we build ships with all of the above allowed, an llms.txt map, and a callable WebMCP endpoint — and the audit we run on prospective clients' sites checks this exact list, because a WAF default silently blocking these agents is one of the most common citation-killers we find.
Sources
- OpenAI. "Bots" — GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot: purposes, tokens, robots.txt behavior. Verified live 2026-08-18. platform.openai.com.
- Anthropic. "Does Anthropic crawl data from the web, and how can site owners block the crawler?" — ClaudeBot, Claude-User, Claude-SearchBot. Verified live 2026-08-18. support.claude.com.
- Perplexity. "Perplexity crawlers" — PerplexityBot, Perplexity-User, robots.txt behavior, not-for-training statement. Verified live 2026-08-18. docs.perplexity.ai.
- Google Search Central. "Google's common crawlers" — Google-Extended's function and the quoted no-ranking-impact statement; Googlebot's separate role. Verified live 2026-08-18. developers.google.com.
- Meta. "Meta Web Crawlers" — meta-externalagent, meta-externalfetcher: purposes and robots.txt behavior. Verified live 2026-08-18. developers.facebook.com.
- Common Crawl. "CCBot" — user agent, robots.txt compliance, open-corpus purpose. Verified live 2026-08-18. commoncrawl.org.
- Microsoft Bing. Webmaster documentation on Bing's crawlers (bingbot). bing.com/webmasters.
Related questions
What is the difference between a crawl bot and a training bot?
Search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot, Bingbot) build the indexes that AI answers cite from — blocking them removes you from answers. Training crawlers (GPTBot, ClaudeBot, CCBot, meta-externalagent) collect content for training future models — blocking them is a content-licensing stance with no documented effect on whether you appear in today's search or AI answers. User-fetch agents (ChatGPT-User, Claude-User, Perplexity-User, meta-externalfetcher) retrieve a page because a human asked; several vendors document that these may not honor robots.txt since the request is user-initiated.
Does blocking Google-Extended hurt my Google rankings or AI Overviews visibility?
No — Google's crawler documentation states it directly: 'Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal.' It controls whether your content trains future Gemini models and feeds grounding. AI Overviews eligibility runs through ordinary Googlebot indexing instead, so blocking Googlebot is what would make you invisible.
Where do these blocks actually happen — is robots.txt enough?
Check three layers. robots.txt is the declared policy, and the reputable crawlers document honoring it. But WAF and CDN bot-protection rules (Cloudflare's AI-bot blocking, for example) can block these agents at the network layer regardless of what robots.txt says — many agencies block AI crawlers without knowing because a security default did it. And user-fetch agents may bypass robots.txt by design, so a hard block requires the WAF layer. Audit all three before concluding you're open.
Should a Medicare or health agency treat this differently?
The crawler list is the same, but remember these bots read pages, not minds: letting an AI engine cite your Medicare content is a visibility win, while what actually creates health-privacy exposure is the tracking layer — pixels firing on quote forms — which is a separate problem with its own answers on this site. Allowing GPTBot does not send anyone's PHI anywhere; a Meta Pixel on your quote form might.
Want to know where your own site stands? Run the free Audit — a live AEO audit plus a HIPAA tracking scan — or browse all answers.