I run a monthly Ahrefs audit on a handful of sites in the tour and charter world. Yesterday one of them came back with a warning I had never seen before.
Health score 69. Forty-three errors. Nothing catastrophic on the Google side — some 404s, some redirect chains, the usual pruning work. Then, buried in the technical panel, a new row:
Slow server response for AI crawlers — 42 URLs
Forty-two. That was almost every meaningful page on the site.
The site loads fine in a browser. It passes Core Web Vitals. Googlebot has never complained. And yet Ahrefs was telling me that an entirely separate class of bots — the ones that feed ChatGPT, Perplexity, and Google’s AI Overviews — were timing out or bailing on nearly the whole thing.
That is a different problem than “your site is slow.”
Why the two categories of bot behave differently
Googlebot is patient. It has been indexing your site for years. It knows your averages, it knows when you deploy, and if a page is slow this crawl it will come back and try again later. Google has infinite budget to be forgiving because a slow crawl on Tuesday still lands in the index by Friday.
AI crawlers do not work like that. Most of them are not indexing your site — they are fetching your page on demand, in the middle of answering a user’s question, with a strict timeout budget measured in seconds. ChatGPT’s browsing tool, Perplexity’s fetcher, Bing’s Copilot, Google’s SGE fetch layer — they are all “read this URL right now, right now, right now.” If your server takes four seconds to send the first byte, the answer they give the user does not include you.
What was actually slow
I opened the site’s origin logs and grepped for the AI user agents Ahrefs was simulating. Three patterns showed up.
First, uncached dynamic pages. The site is on a CMS that renders every page server-side per request. The homepage was fast because it stays warm. The interior pages — the ones the AI bots actually visit when answering a specific question — were cold every time. Cold means “hit the database, run the template, render, return.” Two to four seconds of TTFB on a bad request, occasionally six.
Second, oversized origin assets. The CMS was pulling full-resolution hero photos through its own PHP layer instead of letting the CDN handle them. Every AI crawl was pulling a 3.2 MB image through the origin.
Third, blocking third-party scripts on server render. A “reviews” widget was calling an external API during page render. When that API was slow, the whole page was slow. Googlebot has re-visited enough times that it usually gets a good render. An AI bot that visits once, ever, was rolling the dice.
0
URLs flagged
0s
worst-case TTFB
0
Ahrefs score
0
Google warnings
The fix is not “make the site faster”
That is the reflex, and it is wrong. Making the frontend faster does not help. The AI bot is asking the origin. The origin is what has to be fast.
The playbook that actually moves this number:
Edge cache the HTML, not just the assets. Push rendered HTML for public URLs into a CDN with a sensible stale-while-revalidate policy. AI bots hit an edge node and get a byte in 40ms. If your CMS won’t do this natively, put Cloudflare or Bunny in front of it and set a Cache-Control: public, max-age=300, stale-while-revalidate=86400 on the pages you care about.
Serve images through the CDN, always. Never let a request-time renderer touch a 3 MB photo. Signed URLs, direct-to-CDN, srcset with real dimensions. This is 2015 advice that a lot of legacy sites never took.
Move third-party calls out of the render path. Reviews, weather, availability — none of that belongs in the server-side render of a page that AI bots visit once. Hydrate it client-side after the HTML is on the wire. The bot has your content by then and does not care about the widget.
Confirm the AI bots can actually reach you. Some CDNs and WAFs default to blocking or challenging unknown user agents. GPTBot, PerplexityBot, ClaudeBot, OAI-SearchBot, Google-Extended — check your firewall logs for 403s and 429s on those agents. A robots.txt that allows them is not enough if Cloudflare is 1010-ing them at the edge.
Here is the minimal robots.txt addendum I now paste into every audit deliverable:
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Google-Extended
Allow: /
That file is necessary but not sufficient. If your firewall does not know about those user agents, they still get blocked. Whitelist them at the WAF layer too.
An AI crawler that times out on your page is not going to try again in an hour. It has already told the user something else.
— the load-bearing sentence
The bigger shift
For fifteen years, SEO was a game of “does Google see this and does Google like this.” That game still exists — the site I was auditing gets almost all of its traffic from Google. But an entirely new game is running in parallel now, and it has different rules.
The AI game is one visit, right now, no retry. Your content has to be reachable and returnable inside a few hundred milliseconds. The AI bot does not care that your page renders beautifully once it arrives. It cares whether it arrives at all.
If you are auditing a site and you have not looked at TTFB from an AI-crawler user agent, you are looking at half the picture. The other half is already deciding whether your business shows up when someone asks a chatbot about your industry.
Forty-two URLs, silently absent from every AI answer surface on the internet, is a lot of business to be missing without a single Google warning telling you.