How often does ChatGPT crawl my website?
Nobody publishes a schedule. OpenAI, Anthropic, Perplexity and Google all document which crawlers they run and what each one does, and none of them commits to how often it comes back to a given site. The only cadence figure in OpenAI's crawler documentation is that it "can take ~24 hours from a site's robots.txt update for our systems to adjust." For a small agency site the honest answer is: periodically, not continuously — and a new page is realistically days to weeks from being quotable, not hours. Here's what each operator documents, and how much of the wait you can shorten.
"Crawling" isn't one thing, and that's why the answer feels vague
"How often does ChatGPT crawl my site" is usually three different questions with three different answers, because every major operator runs separate user-agents for separate jobs:
| Kind | Examples | What it does | Rough cadence |
|---|---|---|---|
| Training | GPTBot, ClaudeBot, Google-Extended | Collects content that may be used to build models | Sporadic; the model itself only changes when it's retrained |
| Search | OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot | Indexes pages so the assistant can find and link them later | Repeating, but on the operator's own schedule |
| User-triggered | ChatGPT-User, Claude-User, Perplexity-User | Fetches one page in real time because someone just asked | On demand — potentially seconds after you publish |
OpenAI is explicit that ChatGPT-User is "not used for crawling the web in an automatic fashion" and "not used to determine whether content may appear in Search." Perplexity draws the same line: its user-driven agent fetches only what's needed for the question in front of it, and that content isn't stored for training. So the fast path exists — an assistant can read your page moments after you publish — but only when something already points it at your URL. The slow path decides whether you get named when nobody mentioned you first.
How long until a new page can actually be quoted?
Google is the only operator publishing usable numbers, and because AI Overviews and AI Mode run on the regular Search index, its numbers set the floor for a large share of AI answers. Google's guidance on asking for a recrawl says plainly that "crawling can take anywhere from a few days to a few weeks," and that "requesting a crawl does not guarantee that inclusion in search results will happen instantly or even at all." Its AI-features documentation is broader still: crawling "can take anywhere from several days to several months, depending on how often our systems determine a page needs to be refreshed."
That last clause is the whole game. Refresh frequency is earned, not scheduled. A site that publishes something worth re-reading every few weeks gets checked on a tighter loop than one whose last update was a 2023 headshot — and the loop tightens gradually, over months, not on the day you decide to care.
For ChatGPT there's no equivalent figure, which is itself informative: OpenAI documents eligibility, not timing. Allow OAI-SearchBot, confirm your host or CDN isn't blocking OpenAI's published searchbot IPs, and you're in the pool. Nothing promises a date, and anyone quoting you one is guessing.
How do I know if GPTBot has visited my site?
Stop estimating and go look. Every crawler above identifies itself in its user-agent string, so your server or CDN access logs hold the answer for your actual domain. Filter for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Googlebot and you'll see dates, paths and response codes. Operators also publish IP lists — Perplexity's are at perplexity.com/perplexitybot.json — so you can verify a request claiming to be a bot really is one, since a user-agent string is trivially spoofed.
If your vendor can't give you access logs or a bot-traffic report, notice that. It means you can't tell whether AI crawlers reach your site at all, or catch it when a plugin, firewall rule or builder toggle starts turning them away. We cover those toggles in should insurance agents block GPTBot or ClaudeBot? — the short version is that the most common cause of "AI never mentions us" is a setting nobody remembers flipping.
Can I force a re-crawl?
Not on demand, and not for the AI assistants specifically — there is no "recrawl my URL" button from OpenAI, Anthropic or Perplexity. What you can do is make discovery cheap:
- Keep an XML sitemap that tells the truth. Accurate
lastmoddates on genuinely changed pages, and none you bumped just to look busy. - Submit the URL in Google Search Console. URL Inspection requests a crawl for one page. There's a quota, resubmitting doesn't help, and it guarantees nothing — but it's the one direct lever a site owner has.
- Link the new page from pages that already get crawled. An orphan page waits for a sitemap fetch; a linked page gets found on the next pass over pages already on the schedule.
- Consider IndexNow. It notifies participating engines — Bing, Yandex, Naver, Seznam, Yep and others — the moment a URL changes. Its own FAQ is blunt that "submitting a URL does not guarantee immediate indexing," and Google is not a participant.
- Don't make crawlers work for it. Content that only appears after JavaScript runs, interstitials and slow responses all cost you. Anthropic honors a
Crawl-delaydirective for ClaudeBot — easy to leave at an accidental value and throttle a crawler for months.
What actually moves your crawl rate
The uncomfortable part: re-crawl frequency is a lagging indicator of everything else. You can't buy it, and what raises it is what makes a site worth citing anyway — publishing on a real cadence, being linked from somewhere other than your own footer, and giving crawlers pages that meaningfully changed. That's the mechanism behind the six-month minimum on our monthly service: an answer-engine-ready site in week one is not the same asset in month six.
The honest counterpoint: for a brand-new domain in a small county, none of this may show a visible result for a couple of months, and nobody — us included — can promise a citation on a date. What we can tell you is whether the delay you're seeing is normal or a fixable problem on your end. Those are very different diagnoses, and most agents never learn which one they have. The 10-minute prompt test is the cheapest way to get a baseline before you start counting weeks.
Frequently asked questions
How often does ChatGPT crawl my website?
OpenAI does not publish a re-crawl frequency, and it varies by site. What OpenAI does document is that ChatGPT uses several separate agents: GPTBot crawls content that may be used to train foundation models, OAI-SearchBot is "used to surface websites in search results in ChatGPT's search features," and ChatGPT-User fetches a page in real time when a person asks something that needs it. OpenAI also states it can take roughly 24 hours from a robots.txt update for its systems to adjust, which is the only cadence figure it publishes. For a small business site, assume periodic rather than continuous crawling, and check your server logs for the actual pattern.
How long does it take for AI to index new website content?
Days to weeks is the realistic range for a small business site, and no one can promise a specific date. Google documents that "crawling can take anywhere from a few days to a few weeks," and in its AI features guidance says crawling "can take anywhere from several days to several months, depending on how often our systems determine a page needs to be refreshed." Because AI Overviews and AI Mode draw on the regular Search index, a page must be indexed and snippet-eligible before it can be used there at all. Assistants that fetch pages live at question time can reach a new page sooner, but only if something points them to it.
How do I know if GPTBot has visited my site?
Look in your server or CDN access logs for the crawler's user-agent string — GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot and Googlebot all identify themselves. Most hosts and CDNs expose these logs or a bot-traffic report; if yours does not, that is a reason to move. Operators also publish IP ranges you can verify against, so you can confirm a request claiming to be a given bot really is one. A few weeks of logs will show your site's real pattern far better than any published average.
Can I force ChatGPT or Perplexity to re-crawl a page?
Not directly. Neither OpenAI nor Anthropic nor Perplexity offers a public "recrawl this URL" button for site owners. What you can do is make the page easy to discover: keep an accurate XML sitemap, link the new page from pages that already get crawled, and submit the URL through Google Search Console's URL Inspection tool — though Google warns that "requesting a crawl does not guarantee that inclusion in search results will happen instantly or even at all." IndexNow notifies participating engines such as Bing and Yandex immediately, but its own FAQ notes that "submitting a URL does not guarantee immediate indexing."
How long until new content appears in Google AI Overviews?
There is no separate queue for AI Overviews — Google states that to be eligible as a supporting link in AI Overviews or AI Mode, "a page must be indexed and eligible to be shown in Google Search with a snippet," and that there are no additional requirements or special optimizations. So the timeline is your normal Google indexing timeline: a few days to a few weeks for most pages, longer for a new or low-authority domain. Getting indexed is the gate; appearing in any given AI answer is never guaranteed.
If you'd rather not audit this yourself, it's the standing job on every site we build — sitemap, robots.txt, schema, a monthly publishing cadence, and someone actually reading the crawler logs. That's what the monthly service on every Found Agents plan buys, and why the minimum term is six months.
Find out what the assistants say about your market today
We'll run the prompt test for your counties, check whether the AI crawlers can actually reach your current site, and tell you which of the two problems you have.
Book a free 15-minute demo →