Your Site Can Be Excellent and Still Invisible to AI

After years of working on organic visibility, one thing keeps proving itself: the most expensive problems are the ones nobody checks, because they look too basic to be worth checking. The team writes content, refines the structure, builds authority. Then it turns out that a single line in the server config, set five years ago by someone who left the company long ago, has been holding the door shut the whole time.

Right now the entire industry is talking about how to write content for AI search. Almost nobody starts with the question of whether AI can reach that content at all. That is the wrong order. Refining content for a system that cannot fetch it is the most expensive way to lose a quarter.

Below are four places where access tends to be closed:

• Three layers of blocking - robots.txt, the X-Robots-Tag header, the meta robots tag
• Bot protection - Cloudflare, WAF, anti-scraping rules
• Content loaded via JavaScript - the bot receives an empty shell
• The sitemap and load time - the dull items that decide whether a page gets fetched

You do not need server access to work through this list. You will come out of it with the right questions for your IT team.

It is not one block, it is three layers

Every guide stops at robots.txt. In reality, the instruction "do not index this" can sit in three independent places, and any one of them is enough to make the site disappear:

• robots.txt - a text file in the domain root. The best known and, against expectation, the least often guilty, because it is visible and someone eventually looks at it.
• The X-Robots-Tag HTTP header - set at the server, the CDN, or some layer in front. It does not appear in the page markup, and you will not see it by clicking "view source". This is the single most common reason for the situation where robots.txt is spotless and the site is still invisible.
• The meta robots tag in the HTML - the classic scenario: the staging environment had a global noindex so it would not end up in search engines, it rode along with the deployment to production, and nobody caught it.

These layers do not override one another in any intuitive way, and there is no hierarchy you can rely on. All three have to be checked independently.

The usual sequence goes like this. Someone reports that the company does not show up in AI answers. The team opens robots.txt, sees it is clean, closes the matter, and goes back to working on content.

Over the following months they produce articles that no bot ever fetches, because the block sits one floor up, in a config nobody looked at. The cost of that mistake is not one line. It is a quarter of the team's work.

While you are at it, check whether the block covers only part of the site. Rules can apply to specific directories, and sometimes the one excluded is exactly where the blog or the knowledge base lives - the content most worth citing.

Not every bot does the same thing, and that is the heart of it

This is where most of the mistakes happen, so I will spend longer on it.

The five systems in play right now - ChatGPT, Gemini, Claude, Perplexity and Google AI Overviews - do not mean five bots in a one to one relationship. Each provider sends out several, with different names and genuinely different jobs:

• OpenAI - GPTBot collects content that may be used to train models; OAI-SearchBot is what surfaces sites in the search results inside ChatGPT; ChatGPT-User visits a page live when a user has asked a question.
• Anthropic - ClaudeBot gathers training data; Claude-SearchBot builds the search index; Claude-User visits the page in response to a specific user question.
• Perplexity - PerplexityBot indexes pages so they can be shown and linked in answers; Perplexity-User visits the page live. Perplexity states plainly that neither of the two collects content for training models.
• Google - the Google-Extended token in robots.txt governs whether content may be used to train Gemini models and to ground answers.

Why does this matter so much? Because blocking GPTBot and ClaudeBot does not remove you from ChatGPT or Claude answers. Those are training crawlers. Publishers who blocked them for licensing reasons often believe they have thereby disappeared from AI search. They have not.

It also works the other way round, and that case is far more dangerous: a company that wants to be cited and checked only GPTBot may have the door shut on precisely the bots that determine presence in answers.

One more thing about Google. It states explicitly that Google-Extended does not affect a site's inclusion in Google Search and is not a ranking signal. That is worth knowing, because blocking the token is sometimes described as an SEO risk. It is not. It is a decision about Gemini, and it deserves to be made deliberately rather than in passing.

The report in the Pro package checks this for you. As part of the GEO audit we go through the crawlers one by one and split them into two groups: the ones responsible for your visibility in answers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User, Claude-User) and the training ones (GPTBot, ClaudeBot, Google-Extended). You get it name by name: who has access, who is blocked, and which of the three layers the block sits in.

The practical conclusion is simple. Before you block or unblock anything, work out what you actually want - staying out of model training is one decision, being cited in answers is an entirely different one:

• You do not want to feed training sets but you do want to be cited - block the training crawlers only (see below) and leave open the ones responsible for search and live retrieval.
• Presence in answers is what you care about - OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User and Claude-User have to be open. Those are the names to check first, not GPTBot.

In practice the first scenario looks like this:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Everything else, including OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User and Claude-User, stays untouched.

One caveat worth knowing: Perplexity notes that Perplexity-User generally does not follow the rules in robots.txt, because its visit is triggered directly by the user who asked the question. Bots acting on a user request are sometimes treated differently from crawlers acting on their own initiative - do not assume that one entry in the file settles the matter with every provider.

Bot names change more often than anything else in this area: new ones appear, roles shift. This is not a configuration you set once.

The block nobody meant to put up

After X-Robots-Tag, the second most common cause of silent invisibility is bot protection:

• Cloudflare and similar protective layers
• WAF and anti-scraping rules
• traffic limits, meaning rate limiting

Nobody configures that with AI in mind. It gets configured against scrapers and abuse, and AI systems land in the same net, because from the rule's point of view they look identical: automated, unusual user agent, a lot of requests.

The symptom is distinctive: the page works perfectly in a browser but cannot be fetched programmatically. If anyone at the company has ever said "the tool can't reach our site, but it works fine for us", this is exactly that case.

We check this with every report: when our crawler cannot fetch the page, we name bot protection as the first suspect, because in practice it usually is.

What the bot sees before the page assembles itself

Modern sites often load content via JavaScript once the shell is already in place. The user notices no difference. The bot sometimes sees an empty shell.

It helps to think of two different versions of the same page: the raw HTML that comes from the server, and the HTML after scripts have run, which is what a human sees. The larger the gap between them, the more content sits out of reach of some bots.

The usual casualties are:

• asynchronously loaded product descriptions
• content tucked inside tabs
• reviews
• price lists
• anything that appears only after a click

You can check this without server access. Open the page source - not the inspector, the source - and look for your key paragraph in it. If it is not there, it is not in the version the bot receives either.

This distinction is easy to get wrong, because the browser inspector shows the page after the scripts have run, and everything looks correct there. That is why so many teams are convinced the content is present.

None of this means you have to rebuild the site. It is enough for the most important pages to deliver their substantive content straight from the server. The rest can load afterwards.

This is part of the Pro report as well. Viewing the source by hand answers the question "there or not there", but says nothing about how much content actually falls away. That is why we show both versions of the page side by side: how much text is visible before JavaScript runs and how much after. Alongside it we include an excerpt of what the bot actually receives - the raw HTML, exactly as it looks before the page assembles itself. That tends to be the moment the discussion ends, because instead of arguing about percentages, everyone looks at an empty shell and sees precisely what the bot sees.

The sitemap, the boring measure that works

Four conditions, all of them mundane:

• the sitemap exists
• it is declared in robots.txt
• it is current
• it is not empty

There is nothing interesting to say here, and that is exactly why this point gets skipped for years. Working through it takes fifteen minutes and requires nobody's approval.

Speed, and let us be honest about it

Core Web Vitals - LCP, CLS, INP - are the weakest link on this list, and I am not going to pretend otherwise.

The connection between page speed and citability in AI answers is far looser than it is in classic SEO. The industry currently sells every old item from SEO audits as "critical for AI", and readers are starting to sense it.

The real mechanism is different and narrower: a bot fetching a page live in order to answer a user's question has limited time. A page that loads too slowly simply does not fit inside it. That is reason enough to care about speed, but let us not confuse it with a ranking signal.

The list to send to IT

Copy it and send it as questions, not as instructions:

1. Do we block any of the AI bots in robots.txt? Which ones exactly, by name?
2. Do the server, the CDN, or any layer in front add an X-Robots-Tag header anywhere?
3. Does any section of the site have meta robots with noindex left over from staging?
4. Does our bot protection reject requests by user agent? What exactly is on the block list?
5. Is the key content in the raw HTML, or loaded via JavaScript?
6. Is the sitemap current and declared in robots.txt?

If you want to work through this yourself, that list is all you need - a few hours of work and no budget.

If you would rather have it checked straight away, we run exactly this audit in the Pro package (and in Premium, which includes it). The GEO audit is 22 checks, and among them is every point on the list above:

• which bots have access, which are blocked, and on which of the three layers
• how much content is left of your page once JavaScript is subtracted, together with an excerpt of what the bot receives
• whether bot protection is cutting off the AI crawlers
• whether the sitemap is in place and whether the page fits within the time available

Separately we measure your brand's actual presence in ChatGPT, Gemini, Claude, Perplexity and Google AI Overviews - and show how you compare with your competitors.

But even if you do nothing with us, go through the list. It is the cheapest item in all of GEO, and the only one where a single line of configuration can cost you everything.