Can AI agents actually read your website?

By the SEOCells team, OmCells R&D Labs LLP · Published 8 November 2026

Often they cannot. AI crawlers and agents are commonly blocked by robots.txt rules, by firewalls and bot protection, or see an empty page because the content only appears after JavaScript runs. When that happens, answers are built from other sites instead. Check robots.txt, your bot settings and your raw HTML.

Key points

AI companies publish the names of their crawlers; robots.txt lets you allow or block each one.
Search crawlers (used to cite pages in answers) and training crawlers are often separate and can be treated differently.
Firewalls and bot protection can block AI agents even when robots.txt allows them.
Many AI crawlers do not run JavaScript, so the main text should be in the HTML the server sends.

Why would an AI agent fail to read my site?

There are three usual reasons, and owners are often unaware of all three. A robots.txt rule says “no”. A firewall or bot-protection service quietly turns the visitor away. Or the agent gets in, but the page it receives is an empty shell because the real content is drawn in later by JavaScript.

The result is the same in each case: the engine cannot use your page, so it answers from someone else's. A booking platform, a directory, a competitor.

Which crawlers matter, and what are they called?

Each AI company documents its crawlers. The important detail is that most run separate crawlers for different jobs, and you can treat them differently. Check the official pages listed below for current names, since they change.

Common AI crawlers (check the official documentation for changes)
Company	User agent	What it is for, per the company's docs
OpenAI	OAI-SearchBot	Finding pages to show and link in ChatGPT search
OpenAI	GPTBot	Collecting content that may be used to train models
OpenAI	ChatGPT-User	Visiting a page when a user's request needs it
Anthropic	ClaudeBot	Collecting content that may be used to train models
Anthropic	Claude-SearchBot, Claude-User	Search results and user-requested visits
Perplexity	PerplexityBot	Indexing pages to show and link in Perplexity answers
Perplexity	Perplexity-User	Visiting a page when a user's request needs it
Google	Google-Extended	A control token for use in Gemini models; Google says it does not affect Google Search

Note the Google line. Google's AI Overviews and AI Mode use the normal Googlebot and Search index, so blocking Google-Extended does not remove you from them, and blocking Googlebot removes you from Search entirely.

What should robots.txt say?

That depends on what you want. Many businesses are happy to be cited in answers but would rather not have their content used for training. In that case you can allow the search crawlers and disallow the training ones. Here is an example for a public site, not a recommendation for every site:

# Allow AI search crawlers so pages can be cited
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

# Opt out of model training (optional)
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Disallow: /account/
Disallow: /checkout/

A few practical points. robots.txt is a public request, standardised in RFC 9309, not a lock; reputable crawlers follow it. A crawler follows the most specific group that names it, so a named group replaces your “*” rules for that crawler rather than adding to them. And we regularly find a blanket “Disallow: /” left over from a staging site. Read your live file at yoursite.com/robots.txt.

What about firewalls and bot protection?

This is the one owners miss most. Many hosting firewalls, content delivery networks and security plug-ins now offer a switch to block AI bots, and some turn it on by default. Others challenge any visitor that does not behave like a browser. Your robots.txt can say “welcome” while the firewall says “go away”.

Check the bot or security settings in your hosting, CDN and security plug-in. Look for anything labelled AI bots, AI scrapers or bot fight mode, and decide deliberately. If your server logs are available, search them for the user agents above and see what status codes they received.

Does JavaScript stop AI crawlers?

It can. Google renders JavaScript, though with a delay, and explains the details in its JavaScript SEO documentation. Many AI crawlers do not render at all: they read the HTML the server sends and stop. If your prices, product descriptions or answers are injected by scripts after the page loads, those crawlers see nothing useful.

A quick test: open the page, choose “view page source” (not the inspector), and search for a sentence from your main text. If it is not there, a crawler that does not run scripts will not see it either. The fix is usually server-side rendering or pre-rendering, which most modern site builders and frameworks support.

What is a sensible checklist?

Read your live robots.txt and confirm public pages are not disallowed for the AI search crawlers you want.
Check firewall, CDN and security plug-in settings for AI-bot blocking.
View the page source of your key pages and confirm the main text is in the HTML.
Keep private areas (accounts, checkout, admin) disallowed and protected by login, not by robots.txt alone.
Do not block public pages you want customers to find through AI answers.
Recheck after site redesigns, host moves and security updates.

An /llms.txt file is a newer proposal for giving language models a short guide to your site. It is optional and not a substitute for readable pages. The free SEOCells AEO Score checks whether AI crawlers can read the page you give it.

Questions answered

Should I block GPTBot?

That is a business choice. GPTBot collects content for model training; OAI-SearchBot is the crawler OpenAI says is used to show pages in ChatGPT search. Many sites block the first and allow the second.

Does blocking Google-Extended remove me from AI Overviews?

No. Google says Google-Extended does not affect Google Search; AI Overviews and AI Mode use the normal Search crawler and index.

My robots.txt allows AI crawlers. Why can't they read my site?

A firewall, CDN or security plug-in may be blocking them, or the main content may only appear after JavaScript runs. Check bot settings and the page source.

Is robots.txt enough to keep content private?

No. It is a public request that reputable crawlers follow. Anything private should be behind a login.

Do I need an llms.txt file?

It is an optional proposal and does not replace readable HTML pages. Fix access and rendering first.

Sources

Search engines and AI assistants change often; check the linked documentation for the latest details. Nothing in this post is a promise of rankings, citations or results.

Check whether AI crawlers can read one of your pages — the first AEO Score is free.

Get a free AEO Score