Quite possibly, and probably not on purpose. Firewalls, security plugins and CDN bot rules routinely refuse AI crawlers as unrecognised traffic, and since mid-2025 some providers have shipped that block switched on by default for new sites. A blocked crawler produces no error and no warning. The only symptom is never being mentioned in an AI answer, which looks exactly like a marketing problem and is not one.
Every other piece of advice about AI visibility assumes the engine can read your site. If it cannot, rewriting your homepage, adding schema and publishing comparison pages all achieve exactly nothing, and you will not find out for months.
It is also the cheapest thing on the list to rule out. One command, one answer.
Ask your site for its homepage while claiming to be an AI search crawler, and compare what you get with what a browser gets. In a terminal:
curl -s -o /dev/null -w "%{http_code}\n" \
-A "Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)" \
https://yoursite.com/200 means the crawler is served normally. 403 means it is refused. 503 usually means a bot challenge, which a crawler cannot solve and will treat as a refusal.
Run it again with a normal browser user agent. If a browser gets 200 and the crawler does not, something in front of your site is making that decision, and it is almost never something you configured deliberately.
This is the part that catches people who have already looked into it. robots.txt is a polite request, published at a URL, that a well-behaved crawler reads and chooses to follow. A firewall rule is a refusal, applied before the request reaches anything that could serve a file.
So a site can have a robots.txt that explicitly welcomes every AI crawler, and block all of them anyway. The crawler never gets far enough to read the welcome. If your test above returned 403, editing robots.txt will not change it.
Blanket "block AI bots" settings treat these as one thing. They are not, and the distinction is the whole decision.
Training crawlers read pages to improve a future version of a model. GPTBot, CCBot and Google-Extended are in this group. Blocking them is a legitimate choice about your content, and it does not affect whether you get cited today.
Search and assistant crawlers fetch a page because somebody has just asked a question and the engine is deciding what to say. OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-User and Claude-SearchBot are in this group, along with ordinary Googlebot and Bingbot. Blocking these removes you from answers.
Two of those deserve special mention. Googlebot matters twice over, because Google AI Overviews are assembled from the regular search index rather than a separate AI crawler. And Bingbot matters more than most people assume, because ChatGPT's search runs on Bing's index.
In rough order of how often it turns out to be the culprit: a CDN or firewall bot-fighting setting that challenges anything it does not recognise; a managed security rule shipped on by default, which is why a site nobody has touched can still be blocking; a WordPress security plugin with an AI-bots toggle somebody switched on; a rate limit strict enough that a crawler working through your pages trips it; and finally robots.txt, which is the first place everyone looks and the last place it usually is.
Re-run the test to confirm, then wait. The crawler has to come back on its own schedule, and the engine still has to have a reason to name you rather than someone else.
If access was the whole problem you will see movement in weeks. If nothing changes, the block was not the cause and the question becomes which of the other reasons applies.
Only partly, and that is the trap. robots.txt is a request the crawler chooses to honour, and it is read from a page the server has already agreed to serve. If a firewall refuses the request before that, robots.txt never enters the conversation. Allowing a bot in robots.txt while your firewall blocks it means the bot is blocked.
Separate the two kinds first. Training crawlers read your pages to improve a future model, and blocking those costs you nothing in visibility. Search and assistant crawlers fetch a page to answer a question somebody is asking right now, and blocking those removes you from the answer. Most blanket blocks catch both, which is how sites end up invisible without meaning to be.
The ones that fetch pages to answer live questions: OAI-SearchBot and ChatGPT-User for ChatGPT, PerplexityBot and Perplexity-User, Claude-User and Claude-SearchBot, plus ordinary Googlebot and Bingbot. Googlebot matters twice over, because Google AI Overviews are built from the regular search index rather than a separate AI crawler.
Days to a few weeks. The crawler has to come back, fetch the pages, and the engine has to have a reason to cite you. Unblocking removes the floor under the problem, it does not by itself put you in an answer.
Citeon is in private beta. Tell us your domain and we'll be in touch when there's a place for you.