Does ChatGPT recommend your agency? A 10-minute test
Open ChatGPT in a Temporary Chat, ask five questions the way a buyer in your county would ask them, and write down every business the answer names. Repeat in Perplexity and Google's AI Mode. That's the whole test, and it takes about ten minutes. What matters is doing it clean — most agents run it wrong, get a flattering answer, and conclude they're fine.
Why "ChatGPT, do you recommend my agency?" proves nothing
Three things make the obvious version of this test useless.
You named yourself. Once your agency is in the prompt, the model is summarizing a business you handed it. That's reading comprehension, not recommendation. A prospect never types your name — that's the entire problem you're trying to measure.
Your account knows you. ChatGPT's memory draws on a continually updated synthesis of context from your past chats to personalize responses. If you've spent six months discussing your book of business with it, you are not seeing what a stranger sees. OpenAI's fix is built in: a Temporary Chat doesn't access or create memories.
One answer is noise. These systems generate a response rather than reading off a fixed leaderboard, so the same question can produce different names on Tuesday than it did on Monday. Five questions and a repeat run tell you something. One question tells you nothing — in either direction.
The ten-minute test, step by step
- Start clean (1 min). ChatGPT → new Temporary Chat. Or Settings → Personalization → Memory, off.
- Write five buyer questions (3 min). Name a real county or city in each. Mix urgency: turning 65, lost employer coverage, comparing plans, needs a local human, wants a second opinion on what they already bought.
- Ask each one in its own fresh chat (2 min). Never mention your agency. Don't argue with the answer or ask follow-ups — you're taking a reading, not having a conversation.
- Log names and links (2 min). Two columns: businesses named, and the source URLs the answer cites. The links are the more useful column — they tell you which pages in your market are considered quotable.
- Repeat in a second and third engine (2 min). Perplexity and Google's AI Mode at minimum. Different engines run different indexes and different crawlers, so a split result is normal and diagnostic.
- Check Search Console. Google now offers a Generative AI performance report showing impressions for your site in AI Overviews and AI Mode. Google has said it's rolling out to a subset of site owners, so it may not be in your property yet.
How to read what you got
| What the answers showed | What it actually means | First move |
|---|---|---|
| National quote sites only, no local businesses named | The engines found nothing citable in your market. This is the most common result — and the biggest opening. | Publish genuinely local, answer-shaped pages for the counties you serve. |
| Local competitors named, you're absent | Something about their sites is machine-readable and yours isn't. Usually structure, not effort. | Open the cited pages. Look at their headings, their FAQs, their about page. Copy the shape, never the words. |
| You're named in one engine, invisible in another | An index or crawl-access difference, not a quality verdict. | Check robots.txt for a blanket AI-bot block before you touch the content. |
| You're named, but the description is wrong or stale | The engines are quoting old or thin pages about you — possibly a directory listing, not your site. | Fix your own pages first, then your business listings, so the accurate version is also the easiest one to quote. |
| You're named consistently with correct details | It's working. It is also not permanent. | Keep publishing and re-run this test quarterly. |
Before you blame your content: are you blocking the crawlers?
We see this constantly. A security plugin or a hosting dashboard offers to "block AI bots," someone clicks yes in 2024, and the agency quietly removes itself from the pool. The crawlers are documented and separate:
- OAI-SearchBot — OpenAI documents it as the crawler used to surface websites in ChatGPT's search features. Disallow it and you're out of those answers. GPTBot is a different bot, for model training.
- PerplexityBot — Perplexity's search crawler, which the company says complies with limits set in robots.txt.
- Googlebot — still the control for Google's AI features. Google states that blocking Google-Extended does not control whether you appear in AI Overviews or AI Mode; those follow ordinary Search crawling and the nosnippet family of preview controls.
Ten seconds to check: load yourdomain.com/robots.txt and read it. If you see Disallow: / under any of those user agents, that's your answer, and it's a one-line fix.
What this test can't tell you — and who's lying about that
Be honest with yourself about the limits, because vendors won't be.
There is no rank tracker for AI answers. Impressions in Search Console are the closest thing to real measurement, and Google explicitly cautions against third-party tools that claim to expose internal Google metrics. Anyone selling you an "AI ranking dashboard" with a number in it is selling you a sampled prompt test with a chart on top — which is fine, as long as you know that's what it is.
Google is also blunt that there's no magic markup: structured data isn't required for generative AI search, and no special schema.org type unlocks AI Overviews. We still put schema on every page we build, and we still publish an llms.txt — Google says it ignores files like that, and no major provider has committed to honoring it. It's a cheap file that forces you to state plainly what you do. It is not the strategy. The strategy is being the clearest, most verifiable answer to a question a real person asks, which is what the AEO guide for insurance agents walks through end to end.
And nobody — us included — can promise you a citation. What we can say is what's observable: the agencies getting named locally have pages shaped like answers, real details a machine can verify, and a publishing cadence that gives crawlers a reason to come back. The ones that aren't, usually don't. If your test came back empty, the five-move fix is the next thing to read, and what you get spells out what we do about it if you'd rather not do it yourself.
Questions agents ask about this test
How do I check if ChatGPT recommends my insurance agency?
Open a Temporary Chat in ChatGPT so past conversations and memory can't personalize the answer, then ask five questions in your prospect's words — for example, who can help me compare Medicare plans in a named county. Never mention your agency name. Log which businesses get named and which pages are cited across all five answers, then repeat the same questions in Perplexity and Google AI Mode. The pattern across engines is the result; a single answer is noise.
Why does ChatGPT give me a different answer every time I ask?
These systems generate answers rather than reading from a fixed ranked list, so wording, timing, and the pages retrieved that moment all change the output. Personalization adds more variance: ChatGPT's memory draws on context from your past chats unless you turn it off or use a Temporary Chat. This is why a prompt test needs several questions and a repeat run, and why no one can honestly sell you a guaranteed AI ranking.
Can I see how often my site appears in Google's AI answers?
Partly. Google Search Console has a Generative AI performance report showing impressions for your site in generative AI features on Search, including AI Overviews and AI Mode. Google has said the report is being rolled out to a subset of website owners, so it may not appear in your property yet, and Google warns against third-party tools that claim to expose internal Google metrics.
Does blocking AI crawlers keep my agency out of AI answers?
It can, and agents block them by accident more often than on purpose. OpenAI documents OAI-SearchBot as the crawler used to surface websites in ChatGPT's search features, and sites that disallow it won't appear in those answers. Blanket "block all AI bots" settings in security plugins and hosting dashboards are the usual culprit — check your robots.txt before you conclude the content is the problem.
Run the test. Then let's look at the result together.
Bring your five questions and what came back. We'll tell you honestly whether you need a rebuild or just a publishing cadence — and what it would cost either way.
Book a free 15-minute demo →