Is your website infrastructure ready for AI crawlers?
When organisations ask us how to prepare their websites for AI search, the conversation usually starts with content. Is the information clear? Does the website demonstrate genuine expertise? Does it answer the questions people are asking? Can AI systems understand what the organisation does?
These things matter, and we’ve covered them elsewhere in our AI Hub. But there is another part of AI readiness that receives considerably less attention: the infrastructure underneath the content.
AI systems need to be able to access and interpret your website. At the same time, the rapidly growing number of AI crawlers accessing the web creates a new performance challenge.
The objective isn’t simply to let every AI bot crawl everything, nor is it to block automated traffic indiscriminately. A modern website needs to make useful content accessible to the AI systems you want to reach, while protecting its resources from inefficient, excessive or unwanted automated traffic.
For WordPress websites, that makes rendering, caching, CDN configuration, bot management and ongoing technical management increasingly important.
Make sure your content is actually accessible
One of the first things we would check on a website being prepared for AI search is whether its important content is present in the HTML returned by the server.
Modern websites can use JavaScript to construct or insert content after the initial page has loaded. A human visitor may see a complete page even though some of that information was absent from the original HTML response.
Different search engines, AI crawlers and retrieval systems have different capabilities, and website owners should not assume that every crawler will execute JavaScript and reconstruct a page in the same way as a browser.
The safer principle is straightforward: if information is important enough that you want search engines and AI systems to understand it, make sure it is available in the server-returned HTML.
This is particularly worth checking on websites using JavaScript-heavy front ends or client-side rendering. A simple initial test is to view the page source rather than the rendered page. If the principal content is missing from the original HTML and only appears after JavaScript runs, that deserves further investigation.
This isn’t exclusively an AI-search issue. It has long been sound technical practice, but AI search gives us another reason to pay attention to it.
AI crawlers can create significant server load
For a well-configured WordPress website, ordinary pages can often be served efficiently from a CDN or full-page cache. Dynamic requests are different. Searches, filtered listings, carts, checkout processes and parameter-driven URLs may require PHP execution and database queries on every request.
At bot scale, that distinction matters.
Managed WordPress provider Kinsta recently reported 7.67 million requests from five bots to WooCommerce add-to-cart URLs within a single 24-hour period. It also documented a bot loop where one filtering rule intercepted 550 million requests over 30 days.
These requests do not need to be malicious to cause problems. Crawlers can follow links and URL variations repeatedly, including combinations of parameters that generate effectively endless numbers of URLs.
A crawler methodically reading cached articles may have relatively little performance impact. The same crawler repeatedly hitting dynamic endpoints can consume substantial server resources while gaining little useful information.
Good infrastructure therefore needs to distinguish between useful crawling and expensive, unnecessary automated traffic.
Blocking all AI crawlers isn’t necessarily the answer
For organisations interested in AI-search visibility, indiscriminately blocking AI crawlers may be counterproductive.
Different AI-related crawlers also perform different functions. Some are associated with model training, while others support search, retrieval or user-requested access. OpenAI, for example, distinguishes between GPTBot and OAI-SearchBot, with the latter used to surface websites in ChatGPT search.
This means “allow AI” or “block AI” is becoming too simplistic a choice. Website owners increasingly need to decide which automated systems they want to allow, which they want to restrict and which areas of the website those systems genuinely need to access.
Your CDN and firewall matter
This brings the CDN, firewall and bot-management layer directly into the AI-search discussion.
Services such as Cloudflare can identify and manage automated traffic before it reaches the underlying WordPress server. Cloudflare now provides dedicated AI Crawl Control facilities for analysing AI crawler activity, including request volumes, crawlers, paths and whether requests are being allowed or rejected.
There is an important balancing act here. Configure bot protection too aggressively and useful AI-search crawlers may be blocked. Configure it too permissively and inefficient crawlers may consume resources or repeatedly access parts of the website that provide them with little value.
If bot-management or firewall rules have recently been tightened to address performance, attacks or hosting costs, it is worth checking that the AI crawlers you actually want haven’t been caught by those changes.
AI crawler traffic needs monitoring
Crawler configuration shouldn’t be treated as a one-off exercise. Website owners increasingly need to know which AI crawlers are visiting, how often, which URLs they request, whether they are repeatedly hitting dynamic endpoints and whether useful crawlers are being blocked.
Tools such as Cloudflare’s AI Crawl Control are beginning to provide this visibility.
The important point is not the particular tool. It is that AI crawler behaviour is becoming another part of website traffic that needs to be understood and managed.
The landscape is changing too quickly for a configuration made today to be assumed correct indefinitely.
Good WordPress engineering still matters
Many of the measures that help with AI crawling are simply good website engineering. Accurate sitemaps, sensible canonical URLs and appropriate handling of filters and query parameters reduce unnecessary crawling. Effective caching reduces server work, while removing obsolete plugins and themes reduces complexity and improves maintainability and security.
There is no meaningful universal rule about how many plugins a WordPress website should have. What matters is their quality, what they do and how the complete website performs.
None of this is uniquely about AI. AI crawlers simply provide another reason to get the fundamentals right.
This is another argument for managed WordPress
AI readiness is not a plugin that can be installed once and forgotten.
Important content needs to remain machine-accessible. Caching needs to work effectively. Dynamic endpoints need protection from unnecessary traffic. CDN and firewall rules need to distinguish between useful and unwanted automated access, and crawler behaviour needs to be monitored as it changes.
These requirements sit alongside conventional WordPress responsibilities such as software updates, vulnerability monitoring, backups, access control and performance management.
This is why we increasingly see managed WordPress as a governance decision rather than simply a hosting product. Someone needs to be responsible not only for keeping the server running, but for understanding what is reaching the website, how the application is behaving and whether its configuration remains appropriate as the wider web changes.
AI has made that responsibility broader.
AI search readiness isn’t only about content
Good content remains fundamental to AI search, but producing excellent information is of limited value if the systems you want to reach cannot reliably access it. Equally, opening a website indiscriminately to every crawler makes little sense if automated traffic is consuming resources needed by genuine visitors.
The objective is balance: make important content easy for appropriate AI systems to find and understand, make the website efficient enough to serve it, control access to expensive or unnecessary functionality, and monitor what automated systems are actually doing.
For organisations interested in AI search, the question is therefore no longer only:
It is also:
Is our website infrastructure ready for AI?
'
'