AI
Is Your Website Invisible to AI Assistants?
Many B2B websites are invisible to AI assistants because the crawler never sees the content. In one audit, all 28 pages of a specialist company's site served AI crawlers the same 3.5 KB shell, scoring 32/100 for AI-search readiness. Before rewriting anything, check what a crawler receives. Then fix rendering, crawler access, structured data and answer-first copy, in that order.
Key Takeaways
- AI search is a rendering problem before it is a content problem.
- Test what a crawler receives, not what your browser shows.
- Most findings usually trace back to one or two architectural causes.
- Answer-first copy and structured data make pages easier to quote, but only once they are visible.
What the crawler actually sees
Browsers run JavaScript. Many AI crawlers do not. A site built as a client-side single-page app can look complete to every human visitor while serving crawlers almost nothing.
That is what case study 08 found: every one of 28 pages returned the same 3.5 KB shell, one headline and no body text, to AI-crawler user agents. The site scored 32/100 for AI-search readiness, and most of 25+ findings traced back to that one decision.
A five-minute test
- View the raw HTML. Fetch a page without a browser (or use 'view source', not 'inspect') and check whether your actual copy is in it.
- Check robots rules. Make sure the crawlers you want, such as GPTBot, PerplexityBot and ClaudeBot, are not blocked, and that no noindex header is sent sitewide.
- Check canonicals. Each page should point to itself on your real domain, not to a placeholder or another site.
- Look for structured data. Organization, service and article markup tell machines what the page is about.
We had the same problem
Clarafold's own site scored 25/100 on its first audit: a sitewide noindex header, canonicals pointing at a domain we didn't own, no structured data and no llms.txt. All of it was invisible in a browser.
The fixes were unglamorous: remove the header, correct the canonicals, add JSON-LD to every page, publish llms.txt, and open every page with a plain 40–80 word answer. You are reading the result.
Fix in this order
1. Rendering. Make sure the content is in the HTML the server sends, through server-side rendering, prerendering or static pages.
2. Access. Allow the crawlers you want; remove accidental noindex and broken canonicals.
3. Structure. Add schema markup and an llms.txt that points to your most useful pages.
4. Copy. Lead each page with a direct answer to the question it exists to answer. Assistants quote clear, self-contained statements.
Hand developers files, not opinions
The audit in case study 08 ended with a ready-to-apply pack: a prerender script, server configuration, structured data, sitemap, robots.txt, llms.txt and corrected meta descriptions, each with a one-line command to verify it worked. That turned an abstract 'improve AI visibility' into one to two days of developer time.
GTM Strategy includes this kind of visibility check, because positioning that buyers' assistants can't read isn't working yet.
Put It To Work