More and more people are asking an assistant instead of typing into a search box. “Who does vehicle wraps near Tupelo?” “Is there a place in Iuka open late that sells wine?” Whatever answer comes back was read off somebody’s website.
If the crawlers those assistants use can’t read yours, you’re not in the running. Not ranked low — absent.
We went through every site we host last week looking for exactly this, and we found sites quietly refusing the crawlers that feed ChatGPT, Claude, Perplexity and Google’s AI answers. Nobody chose that. It came in with a theme, or a plugin, or a file somebody pasted in years ago and forgot.
Here’s how to check your own. Type your domain into a browser followed by /robots.txt — so yourbusiness.com/robots.txt. You’ll get a plain text file. Look for any line that says Disallow: / underneath a name like GPTBot, ClaudeBot, PerplexityBot, Google-Extended or CCBot. If you see one, that crawler has been told to go away.
Two things worth knowing. A file that doesn’t exist at all is usually fine — it means nothing is being blocked. And a file that’s there but blocking ad networks and scrapers is often deliberate and harmless. It’s the AI and search crawlers you care about.
The other trap is harder to spot: a physical robots.txt file sitting on the server overrides whatever your SEO plugin thinks it’s publishing. Your plugin shows one thing in the dashboard, visitors get another. We found that on twenty-six sites.
If you’re not sure what you’re looking at, send us the address and we’ll look for free. It takes a minute and it’s the kind of thing that costs you customers silently for years.