Should insurance agents block GPTBot or ClaudeBot?

Almost certainly not. If you're a licensed agent hoping to get named when someone asks an assistant "who can help me with Medicare in my county," blocking AI crawlers bolts shut the door you want to walk through. But the question deserves better than "don't," because "AI crawler" is three different things wearing the same coat — and the toggle that blocks one usually blocks all three, including, in one configuration, Googlebot.

Why publishers block — and why you're not a publisher

The case for blocking is real, and it wasn't invented by cranks. It was built by publishers whose business model is ad-supported pageviews. For a news outlet, the article is the product: when an assistant summarizes it, the reader gets the value and the publisher gets nothing.

An insurance agency website is not that business. Yours is a brochure and a lead-capture form — nobody visits it to consume content for its own sake. So when an assistant reads your FAQ page and tells a prospect "an independent agent in your area, [Your Agency], covers this — here's their page," that isn't leakage. That's the outcome you're paying a website to produce. And there's nothing to protect anyway: plan facts and enrollment windows come from CMS and are public by definition, while the proprietary asset in your business — your book, your relationships, your license — never lived on a web page.

There isn't one AI crawler. There are three kinds.

Every major operator publishes separate user-agents for separate jobs, and that distinction is everything. Training crawlers collect content that may build models. Search crawlers index pages so the assistant can find and link them later. User agents fetch one page in real time because a person just asked something that needs it.

Those descriptions come from the operators' own docs — OpenAI, Anthropic, Perplexity, Google. Remember one row: OAI-SearchBot stands between your pages and ChatGPT's search results.

Does blocking GPTBot hurt my Google rankings?

No — the one place the pro-blocking crowd is completely right. GPTBot and Googlebot are different companies running different pipelines, so disallowing GPTBot has no effect on how Google crawls, indexes, or ranks you. Google says the same about its own training crawler: "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."

But rankings usually aren't what's really being asked. The real question is whether blocking costs you anything you care about — and it does, if you block the search and user agents. Those decide whether an assistant can retrieve your page while composing an answer about your market.

Blocking training won't keep you out of AI answers — and allowing it won't get you in

Two levers get confused constantly. Separated:

Blocking training does not remove you from AI answers. AI Overviews and AI Mode are surfaces inside Search, fed by pages Googlebot already crawled. Google's documentation on AI features says a page needs only to be "indexed and eligible to be shown in Google Search with a snippet," and points owners to nosnippet, data-nosnippet, max-snippet, and noindex. Note the trade: those apply across Search, so suppressing your AI snippet suppresses your ordinary one too.

Allowing training does not put you in AI answers either. Letting GPTBot crawl you today doesn't mean ChatGPT mentions your agency tomorrow; training runs are infrequent and not a citation mechanism. When an assistant names a local agent, it's nearly always because a retrieval step just fetched a page and quoted it — which is why page structure matters far more than robots.txt ever will. Our AEO guide is the long version of that argument.

The Cloudflare default that changes on September 15, 2026

On July 1, 2026, Cloudflare announced new AI traffic controls for all customers, free plans included, using the same three categories. Starting September 15, 2026, newly onboarded domains get Training and Agent crawlers blocked by default on pages that display ads; Search stays allowed.

For most agency sites that default is harmless — you don't run display ads on a brochure site. What matters is that if you're on Cloudflare, these controls now exist in your dashboard whether you asked for them or not.

The setting your website builder may have already flipped

This is the version we actually find on agent sites: nobody decided anything, a checkbox got ticked. Wix, for example, offers a block-AI-crawlers setting whose documented list includes CCBot, GPTBot, ChatGPT-User, and BingAI. Look at the third one — ChatGPT-User fetches a page because a person asked about it. Block it, and a prospect who types your agency's name into ChatGPT gets an assistant that can't open your site.

Go read your own /robots.txt right now. If AI user-agents are disallowed there and you didn't put them there, your platform did.

What we put in robots.txt on an agency site

Allow the answer engines; disallow pages never meant to be indexed:

User-agent: *
Allow: /
Disallow: /thank-you
Disallow: /client-portal

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

Sitemap: https://youragency.com/sitemap.xml

An explicit Allow for a bot nothing is blocking is redundant — the wildcard already permits it. We write them out anyway: it documents an intentional decision and makes an accidental platform-level block obvious on sight. Ours is public: foundagents.ai/robots.txt. It's also a different job from llms.txt — robots.txt governs access, llms.txt describes content.

When blocking actually makes sense

Three honest exceptions, none of them an agency homepage:

• Genuinely gated material. A paid course, a members-only library. Blocking is reasonable — but put it behind a login, since robots.txt is a request, not a lock.
• Client portals and anything with personal data. Never rely on robots.txt here; use authentication. A disallowed URL is still a published URL — you've just handed a list of interesting paths to anyone who reads the file.
• Staging and demo sites. Use noindex plus HTTP auth, not a crawler rule.

Notice what's missing: your county pages, your FAQ, your about page, your blog. Blocking those is paying for a billboard and throwing a sheet over it. Allow them all, and spend the energy on making them worth quoting — nobody can promise you a citation, and a page with nothing verifiable on it won't earn one even with the welcome mat out. To find out whether that describes your site, run the 10-minute test; what we build into every site is the fix for what it turns up.

Frequently asked questions

Should an insurance agent block AI crawlers?

In almost every case, no. The blocking argument was built for publishers whose business is ad-supported pageviews, where an AI summary replaces the visit they get paid for. An insurance agency website is a brochure and a lead-capture tool, not the product being sold — being quoted by an assistant, with the agency named, is the outcome the site exists to produce. The narrow exceptions — gated content, client portals, staging sites — need real authentication rather than a robots.txt rule.

Does blocking GPTBot hurt my Google rankings?

No. GPTBot is OpenAI's crawler and has nothing to do with Googlebot, so disallowing it does not affect crawling, indexing, or ranking in Google Search. Google says the same about its own AI training crawler: "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." The real cost of blocking is not to your rankings — it is to whether AI assistants can read and cite you.

What is the difference between GPTBot and OAI-SearchBot?

They are two different OpenAI crawlers with two different jobs. GPTBot collects content that may be used to train OpenAI's foundation models, and OpenAI states that disallowing it signals a site's content should not be used for that training. OAI-SearchBot is documented as the crawler "used to surface websites in search results in ChatGPT's search features" — it is the one that determines whether ChatGPT can find and link your pages. Blocking GPTBot but allowing OAI-SearchBot is a coherent middle position; blocking both removes you from ChatGPT's search surface.

Does blocking AI crawlers remove my site from Google's AI Overviews?

No, and this is the most common misunderstanding. Google's AI Overviews and AI Mode are surfaces inside Google Search that draw on pages crawled by Googlebot, so blocking Google-Extended does not remove you from them. Google's documentation says a page must be "indexed and eligible to be shown in Google Search with a snippet" to appear, and points site owners to the nosnippet, data-nosnippet, max-snippet, and noindex controls to limit what is shown. Those controls apply across Search generally, so there is no surgical AI-Overviews-only opt-out.

Do AI crawlers actually respect robots.txt?

The major published crawlers say they do. Anthropic states its bots "respect 'do not crawl' signals by honoring industry standard directives in robots.txt," and OpenAI and Google publish robots.txt tokens for their crawlers. But user-triggered fetchers are treated differently: Perplexity documents that Perplexity-User "generally ignores robots.txt rules" because a person initiated the request, and OpenAI notes robots.txt controls do not necessarily apply to ChatGPT-User. Robots.txt is a published preference, not an access control: anything that must stay private needs authentication.