Most of Your Audience Can't Run JavaScript. Testing What AI Crawlers Actually See

Kyle Tyacke, Director of Strategy and Innovation

For years, every developer program has validated its site against one reader: Googlebot.

Googlebot became proficient at JavaScript, the rendering question felt closed, and the industry moved on to content strategy. Meanwhile, a second class of readers arrived, took over the majority of web traffic, and somehow never learned to run JavaScript.

The result is a failure mode that produces no visible symptoms. Rankings hold, Core Web Vitals hold, and analytics look normal, while the thing that breaks is your presence inside an AI-generated answer that a buyer reads without ever loading your page. A quickstart that ranks first on Google can return an empty shell to the AI systems now shaping which vendors get shortlisted.

Almost nobody is testing for this, because it doesn’t yet appear in any standard SEO or performance report. But if you have ten minutes and a browser with JavaScript switched off, you can identify whether your highest-value pages are legible to the machines reading them, which makes the missing test the real finding here.

The Data

• Automated requests passed human requests for the first time in internet history, hitting 57.5% of HTML traffic as of June 3, 2026, roughly double the bot share a year earlier. (Cloudflare, 2026)

  
  

• A joint analysis from Vercel and MERJ of more than 500 million GPTBot fetches found no evidence of JavaScript execution. GPTBot, ClaudeBot, and PerplexityBot download script files and never run them. Googlebot remains the exception, and Gemini inherits its rendering capability. (Moore, 2024)

  
  

• In a benchmark of 2,400 agent runs across 20 documentation sites, requests to pages

  that did not exist fell from 2.23 per task on standard HTML to 0.11 once each page linked to an llms.txt file. Answer accuracy stayed in the mid-to-high 90s across every format, so the gain was efficiency rather than correctness. Pages stripped down to plain text with no navigation performed worse than standard HTML, because agents started guessing at addresses. (Shah, 2026)

  

• An audit of 399 sites across 11 industries found zero industries clearing the 60-point pass threshold for machine legibility, with a weighted mean of 41.3 out of 100. The largest single drag was whether content answered a buyer question in a self-contained, quotable block. These sites volunteered for a free audit, so treat 41.3 as directional and the failure point as the finding. (Kaliber, 2026)

  
  

• AI referrals sit near 1% of total sessions in cross-industry benchmarks and convert at roughly 4x to 5x organic search. (Conductor, 2025; Semrush, 2025; Opollo, 2026)

  
  

The Analysis

Three gaps compound here. The first is a navigational gap. Agents do not know which pages you have, so when a link is missing, they guess at the addresses and often get it wrong. The wasted effort is theirs to absorb, so this looks like someone else’s problem at first glance, but it’s not. An agent working under a step or context limit that cannot find your page does not stop and report failure.

It substitutes. It answers from its training data, which may describe an API version you replaced a year ago, or from a third-party page it can navigate more easily, which may be a competitor’s comparison. Your documentation stops being the thing that describes your product.

Accuracy held in the benchmark only because agents were given room to keep retrying, and real budgets are tighter than a benchmark’s. Agents also do not go hunting for help. An llms.txt file listing every page on the site changed nothing in the benchmark until the pages themselves linked to it, because agents follow the links they encounter and do not probe for files that nobody told

them about.

The second is a rendering gap, and it makes the first one worse. A page that builds itself in the browser has no links in its response for an agent to follow, and no words either. What comes back to a crawler that can’t run JavaScript is an empty template. There is nothing to quote and nothing to cite. The exposure concentrates on exactly the surfaces developer programs care about most: documentation, API references, quickstarts, and pricing. Those are the pages most likely to be built as interactive applications, with tabbed content, in-page search, and spec tables that appear only after the browser finishes loading.

The third is a citability gap, and it is not technical at all. The page structure that converts best for humans is often the least citable for machines. Marketing pages build tension deliberately, stating the benefit up top and the specifics three sections down. An AI system pulling an answer from that page finds no self-contained statement of what the product does or what it costs, so it cites a competitor whose documentation says the thing plainly.

What This Means for Developer Marketing Teams

Build the check into the release process. Load your highest-value pages with JavaScript switched off and treat anything that disappears as a defect, because that is what a crawler receives. Then ask your team for a plain-text version of the same page, and check whether its opening answers the buyer’s question without the rest of the page around it. The first test catches the rendering gap, and the second catches the citability gap. Automate the first in your continuous integration pipeline, the automated gate that runs on every build, because rendering breaks silently, and a quarterly audit will not catch it.

Measure the bot side before optimizing anything. Ask engineering to break down server logs by which AI crawler made each request, then track how often those requests land on a page that does not exist. A rising error rate is an early warning that your site structure is confusing them, and it appears well before any drop in citations.

Sequence the fixes. Getting your main content into the page the server sends comes first,

because nothing else matters if the content never arrives. An llms.txt file comes second and

is the cheapest win available: a plain-text page at your site root listing every URL with a one-line description, linked from every page so agents meet it while working. Publishing it without those links does nothing, and Google has stated that Search does not use it, so treat it as an agent efficiency play rather than a ranking one. Rewriting pages so a self-contained answer sits in the first 120 words comes third, and it takes the most judgment.

Start with documentation. Docs describe how the product behaves, and behavior is what an

evaluation agent needs to judge fit, which makes them the first place to run all of this, rather

than the last. The structural work buys eligibility. The content work earns the citation.

For more information read the full Developer Marketing Insights Report.