Do AI Search Engines Actually Read Your Website?
By
Izzy Gregorio
·
6 minute read
Definition · Technical
Do AI Search Engines Actually Read Your Website?
Some do, some do not, and they do different jobs. Blocking the wrong one removes you from answers without protecting anything.
By Izzy Gregorio · Updated August 2026 · 9 min read
In short
GPTBot and CCBot gather training data. OAI-SearchBot and PerplexityBot retrieve live pages to answer questions right now. Google-Extended is not a crawler at all, it is a permission setting that fetches nothing. AI crawler access is the cheapest and most-skipped fix in this discipline, and a robots.txt copied from a template can remove a business from AI answers entirely with nothing in any dashboard to report it.
Know the difference
The crawlers are not doing the same job
Treating them as one category is how businesses block the wrong thing.
| Crawler | Belongs to | What it does | Blocking it means |
|---|---|---|---|
| GPTBot | OpenAI | Collects data used for model training | Excluded from future training, not from live answers |
| OAI-SearchBot | OpenAI | Builds the index behind ChatGPT search results and citations | Excluded from ChatGPT search results. This is the one that matters |
| ChatGPT-User | OpenAI | Fetches a specific page live during a conversation | Real-time reads stop mid-conversation |
| PerplexityBot | Perplexity | Live retrieval and indexing | Excluded from Perplexity answers |
| ClaudeBot | Anthropic | Retrieval for Claude | Excluded from Claude answers |
| Bingbot | Microsoft | Builds the Bing index, one input among several to ChatGPT retrieval | Broad exclusion, well beyond Bing itself |
| CCBot | Common Crawl | Open crawl corpus used by many models | Excluded from a dataset many systems draw on |
| Google-Extended | Not a crawler. A permission token covering Gemini training and grounding | You stay in Google Search and in AI Overviews, and drop out of Gemini grounding |
Google-Extended is the one people get wrong most often, in both directions. It does not fetch anything. It has no user-agent string. Googlebot does the crawling, and the token only tells Google whether the resulting content may be used for Gemini training and grounding.
Blocking it does not remove you from AI Overviews. Those are features inside Search, and Google states plainly that Google-Extended does not affect inclusion in Search.
Plenty of publishers added that line believing they had opted out of Google's AI features. They opted out of Gemini and stayed exactly where they were in AI Overviews.
Three controls, not one
If you actually want out of Google's AI features
There are three separate levers with three different scopes, and most published guidance still describes a world with two. The third one is new.
| Control | What it removes you from | Collateral damage |
|---|---|---|
| Google-Extended in robots.txt | Gemini model training and grounding in Gemini Apps and Vertex AI | None. Search inclusion and ranking unaffected |
| nosnippet, data-nosnippet, max-snippet | AI Overviews and AI Mode | Also removes your ordinary search snippet. All or nothing |
| Search Console AI features setting | Google's AI features, and nothing else | None. This is the control that did not previously exist |
Google acknowledged the all-or-nothing problem with the snippet route and added the Search Console option in response. If you have been avoiding an AI opt-out because it meant losing your search snippet, that constraint no longer applies. Most guidance on this topic predates the change.
Run this now
The four-gate check
Being read is a sequence of four gates. Failing any one of them produces the same outcome, which is silence.
- 1
Permission
Open yourdomain.com/robots.txt in a browser. Read it yourself rather than asking whether it is fine. Look for Disallow lines under any of the user agents above, and for a blanket rule catching every agent.
- 2
Delivery
Confirm the page returns real content to a plain request. If your content only appears after JavaScript executes, some retrieval systems see an empty shell. Gemini is the notable exception, because it renders through Googlebot's infrastructure. Do not assume the others do.
- 3
Indexation
Search site:yourdomain.com in both Google and Bing. A large gap between the two counts is an indexation problem, and Bing's index is one of several inputs feeding ChatGPT retrieval.
- 4
Extractability
Read the first hundred words of a page in isolation. If a lifted passage would not make sense without the surrounding page, there is nothing for a system to cite even after it reads you.
Gates one through three are technical and usually fast. Gate four is editorial, and it is where most sites actually fail.
The distinction that matters
Crawling is not citing
A system reading your page is the entry requirement. It is not the outcome.
Roughly 77 percent of the sources cited in AI answers about a brand are off-page, per Omniscient Digital's review of more than 23,000 citations. Muck Rack's December 2025 analysis found 82 percent of AI citations come from earned media.
So a perfectly crawlable site with no third-party corroboration is legible and unconvincing. The engine can read your claim about yourself and has nothing to check it against.
Access gets you considered. Corroboration gets you cited.
That distinction reorders most budgets. Crawler access is a one-hour fix. Citation supply is a program.
Keep going
Get the next breakdown before it is public.
Crawler names change, rendering behavior changes, controls get added, and nothing notifies you when your site quietly stops being read. What shifted, what to re-check, and the fixes worth an hour of someone's time.
No spam. Unsubscribe anytime.
The tempting mistake
Should you block AI crawlers to protect your content?
Some publishers do, and for a large media company with a licensing strategy that can be a defensible business decision. For almost every business selling a product or service, it is self-harm.
Here is what blocking actually accomplishes. Your content stops being read. The question still gets asked. The answer still gets assembled, from your competitors' pages and from third-party sources describing your category. You lose the citation and the referral, and you do not recover the click, because the click was never going to a page the reader did not know existed.
You also do not stop being discussed. Roughly four out of five citations about a brand come from pages the brand does not own. Blocking your own site removes the one source you control from a conversation that continues without you.
If the concern is training data specifically, there is a middle position. Block GPTBot and allow OAI-SearchBot. You withhold the training material and keep your eligibility to be cited in ChatGPT answers. The two are controlled independently.
The only thing blanket blocking reliably prevents is being the source of your own description.
Translate it
What breaks by site type
| If your site is | The gate that usually fails | The fix |
|---|---|---|
| A WordPress site an agency built years ago | Gate 1. Inherited robots.txt with blanket disallows written before AI crawlers existed | Read the file. One line, one deploy |
| A React or single-page app | Gate 2. Content renders client-side and returns an empty shell | Server-side rendering or prerendering for crawlers |
| A large ecommerce catalog | Gate 3. Google indexed deeply, Bing barely | Submit to Bing Webmaster Tools, import from Search Console |
| A well-maintained marketing site | Gate 4. Everything is readable and nothing is liftable | Move answers to the top, cut backward references |
| A site behind an aggressive firewall or bot filter | Gates 1 and 2 together. Crawlers blocked at the edge, invisible in robots.txt | Check server and CDN bot rules, not only robots.txt |
That last row catches people. A robots.txt can be perfectly permissive while your CDN quietly rejects the same crawlers at the edge, and nothing in the file will tell you.
One question worth sitting with. You can check all four gates in under an hour, and most businesses have never checked the first one. So the real question is not whether AI reads your site. It is who decided the answer to that, and when, and whether anyone has looked since.
Common questions
AI crawlers, answered
What is GPTBot?
GPTBot is OpenAI's crawler for collecting web data used in model training. It is separate from OAI-SearchBot, which builds the index behind ChatGPT search results, and from ChatGPT-User, which fetches a specific page live during a conversation. Blocking GPTBot affects training inclusion. Blocking OAI-SearchBot affects whether you appear in ChatGPT search results.
Does blocking Google-Extended remove me from AI Overviews?
No. AI Overviews and AI Mode are features inside Google Search, governed by Googlebot and snippet controls. Google-Extended covers Gemini model training and grounding in Gemini Apps and Vertex AI. Google states it does not impact inclusion in Search and is not a ranking signal. Publishers who added the rule believing they had opted out of Google's AI features opted out of Gemini and stayed in AI Overviews.
How do I actually opt out of Google's AI features?
There are three separate controls. Google-Extended in robots.txt covers Gemini training and grounding only. The nosnippet family removes you from AI Overviews but also removes your ordinary search snippet. A Search Console setting removes a site from Google's AI features and nothing else, which is the option that did not previously exist and that most published guidance predates.
How do I check if my site blocks AI crawlers?
Open yourdomain.com/robots.txt in a browser and read it. Look for Disallow rules under GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Bingbot, and CCBot, and for a blanket rule that catches all agents. Also check CDN or firewall bot rules, which robots.txt will not reveal.
Does JavaScript break AI crawling?
It can. If content only appears after client-side JavaScript executes, some retrieval systems receive an empty shell. Gemini renders JavaScript through Googlebot's infrastructure, so it is the notable exception. Server-side rendering or prerendering removes the risk across all of them.
Should I block AI crawlers to protect my content?
For most businesses, no. Blocking stops your pages from being read while the questions still get asked and answered from competitor and third-party sources. You lose the citation and the referral without recovering traffic, since roughly 82 percent of AI citations already come from earned media. If training data specifically is the concern, block GPTBot and allow OAI-SearchBot.
If AI can read my site, will it cite me?
Not automatically. Access is the entry requirement, not the outcome. Roughly 77 percent of sources cited about a brand are off-page, per Omniscient Digital's review of more than 23,000 citations, so corroboration from third parties is what converts readability into citation.
After the four gates
Find out whether readable is translating into cited
Run the four gates yourself, and you should. But passing all four only means you are eligible. The AI visibility audit from Conspicuouz Creative Group tells you whether eligibility is producing anything, by putting your category's real questions to ChatGPT, Claude, Gemini, and Perplexity and recording who gets named instead of you.