The only reliable way to know whether ChatGPT can read your website is to test it the way ChatGPT does: check your robots.txt for OpenAI's crawlers, then fetch your pages using their user agents and look at what actually comes back. Plenty of sites that look perfect in a browser are blank, blocked, or half-missing to AI — and their owners have no idea.
Which OpenAI bots are you testing for?
OpenAI operates several crawlers, and they do different jobs:
- GPTBot gathers content for training future models.
- OAI-SearchBot builds the index behind ChatGPT's web search — the one most tied to whether you get cited and linked in answers.
- ChatGPT-User fetches a page in real time when a user asks about it in a conversation.
They obey robots.txt rules separately. A site can welcome one and block the others without meaning to, so test all three names.
Does robots.txt tell the whole story?
Start at yourdomain.com/robots.txt and read what's there. You're looking for lines like "User-agent: GPTBot" followed by "Disallow: /" — an explicit block — or a blanket "Disallow: /" applied to every user agent, which is a surprisingly common leftover from a staging site or an overzealous plugin setting.
But robots.txt is only a statement of policy. It tells polite crawlers what you'd like; it says nothing about what your server actually does. A firewall or CDN can block bots your robots.txt welcomes, and a misbehaving bot can ignore rules you've set. So check it, fix anything obviously wrong, and then move to the real test.
How do you run a real fetch test?
Open a terminal and request your homepage the way OpenAI's crawler would, by setting the user agent. With curl, the test looks like this, all on one line:
curl -I -A "Mozilla/5.0 AppleWebKit/537.36 (compatible; GPTBot/1.2; +https://openai.com/gptbot)" https://yourdomain.com/
The -I flag asks for just the response headers. Read the first line:
- 200 means the server said yes. Good start.
- 403 or 401 means something — usually a firewall or bot-protection rule — refused the request.
- 503, or a tiny response about "verifying your browser," usually means a challenge page no crawler can pass.
Then drop the -I and fetch the full page. Search the output for something only your real content contains — your phone number, a service name, a sentence from your homepage. If the status is 200 but your content isn't in the response, you've found a different problem.
What about JavaScript rendering?
This is the blind spot that catches modern sites. Most AI crawlers fetch raw HTML and do not execute JavaScript. If your pages are built client-side — a React or Vue single-page app, or a builder that injects content with scripts — the crawler may receive a technically valid page containing a loading spinner and little else. Your browser runs the scripts and shows a beautiful site; the bot sees an empty shell.
The quick check needs no terminal: right-click your page, choose "View page source" (not "Inspect"), and search for a sentence you can see on screen. If it's not in the source, it doesn't exist for most AI crawlers. The fix is server-side rendering or prerendering, which is a bigger conversation — but you can't have that conversation until you know.
What does "invisible to AI" look like in practice?
From the outside, the symptoms are easy to shrug off. Ask ChatGPT about your business and it politely says it can't access the site — or, worse, confidently describes a years-old version of your services pulled from stale third-party sources. Ask it for recommendations in your category and it cites competitors whose sites it can read. A roofing company in a mid-size market can dominate the map pack and still be a rumor to AI engines, because visibility to Google and visibility to generative engines are separate problems with separate plumbing.
What should you do if you fail the test?
Work the list in order: fix explicit robots.txt blocks first, then investigate your firewall or CDN's bot settings if you're seeing 403s, then tackle rendering if your content isn't in the raw HTML. Each fix is verifiable with the same curl test, which is the beauty of this approach — no guessing. And since Google's AI Overviews ride on ordinary Googlebot crawling, keeping your classic SEO hygiene intact covers that front while you sort out the newer bots.
If you'd rather not live in a terminal, this whole battery — robots rules, live fetches as each major AI crawler, and rendering checks — is the sort of thing Speak Local runs automatically and re-checks over time. However you do it, run it. The businesses that show up in AI answers are, first and unglamorously, the ones whose sites AI can actually read.
