There is a specific failure that keeps appearing in modern B2B sites, and it is expensive precisely because it is invisible to everyone who checks the page the obvious way.
The comparison page — the one with the feature matrix, the one buyers actually use — is rendered client-side. Open it in a browser and it is excellent. Fetch it the way a non-browser client does and the main content area contains an empty div and a loading state.
Why this hits AI crawlers harder
Google has executed JavaScript for years, with caveats about queueing and budget. The crawlers that feed AI answer systems are a more varied population, and several of them do substantially less with JavaScript than Googlebot does. A page that Google eventually renders may be a blank document to a retrieval pipeline that fetches once and moves on.
Stated as a hypothesis
We cannot see inside any provider's crawling stack, so treat the specifics as a working assumption rather than a documented fact. What is directly observable is what your server returns to a given user agent without JavaScript execution — and that observation alone is usually enough to justify the fix.
The second half of the problem
Rendering is only one of two ways to be unreadable. The other is bot management. Edge platforms are very good at distinguishing browsers from everything else, which is exactly what you asked them to do, and the rules were usually written before several of these crawlers existed.
- A JavaScript challenge returned to a non-browser user agent is, from the crawler's perspective, a page with no content.
- Rate limits tuned for scrapers will also throttle a legitimate crawler working through your sitemap.
- Blanket blocks on 'unknown' user agents catch new crawlers by definition, since every crawler is unknown until someone adds it.
- Geographic restrictions can exclude the regions crawling infrastructure operates from.
How to test it in ten minutes
You do not need a tool for the first pass. Fetch your most commercially important pages without a browser, with a few different user agents, and read what comes back.
# What does a non-browser client actually receive?
curl -sL -A "Mozilla/5.0 (compatible; ExampleBot/1.0)" https://example.com/compare/x-vs-y \
| grep -c "<tr" # or whatever marks your real content
# Compare against a browser user agent
curl -sL -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36" \
https://example.com/compare/x-vs-y | grep -c "<tr"
# And check the status codes, not just the body
curl -sL -o /dev/null -w "%{http_code}\n" -A "ExampleBot/1.0" https://example.com/pricingIf the counts differ materially, or the status codes differ by user agent, you have found something worth more than a quarter of content production.
Then check what you are actually allowing
Separately from rendering and bot management, read your robots.txt properly. Different tokens control different things, several of them were added at different times, and in our experience the rules in place are frequently inherited from a template that nobody has reviewed since.
Blocking model training crawlers while allowing search crawlers is a coherent, defensible position. Blocking both by accident, and then investing in content marketing, is not. The point is that it should be a decision somebody made on purpose.
Why this is the first thing we look at
Because it is fast, cheap, unambiguous and provable. Content and authority work takes months and involves judgement. Discovering that your pricing page returns a 403 to everything without a browser fingerprint takes an afternoon and is not a matter of opinion.