The importance of infrastructure in the age of AI bots
AI is changing the way people discover and interact with websites, but there is another consequence receiving considerably less attention: AI systems are becoming significant consumers of website infrastructure.
We have recently seen this first-hand while investigating repeated periods of instability on the WordPress website of a well-known visitor attraction.
The website was not experiencing a conventional cyberattack. Instead, automated crawlers had discovered part of its calendar system that allowed them to generate an effectively unlimited number of URLs.
They kept crawling.
In under nine hours, around 19,000 requests hit nearly 17,000 distinct calendar URLs. Almost none could be served from cache, meaning WordPress had to generate a complete page for nearly every request.
The result was substantial unnecessary server load and intermittent website availability.
It provides a useful example of a problem we expect website owners to encounter increasingly often.
Not all website traffic is equal
When somebody visits a normal page on a well-configured WordPress website, there is a good chance much of the work has already been done. Caching allows the server or CDN to return a previously generated version of the page rather than asking WordPress and its database to rebuild it every time.
Dynamic requests are different. Search results, filters, calendars, shopping baskets and other interactive functionality can generate URLs that need to be processed by WordPress on every request. One such request is normally insignificant. Thousands arriving in rapid succession are not.
This distinction becomes particularly important with automated crawlers. A bot requesting 10,000 cached pages may cause relatively little difficulty. A bot making 10,000 requests that each trigger PHP execution and database queries can create a very different infrastructure problem.
The bots were not necessarily attacking the website
This is an important distinction. Much of the traffic we investigated was not obviously attempting to compromise the website. Crawlers had simply found links they could continue following.
The calendar allowed navigation through dates, with different URL parameters producing different pages. From the crawler’s perspective, there was always somewhere else to go. From WordPress’s perspective, each was another page to generate.
This is sometimes described as a crawler trap or infinite crawl space. Faceted product searches, calendars, dynamically generated archives and combinations of URL parameters can all create similar problems.
AI has not invented this problem. Search-engine crawlers have been capable of encountering crawler traps for years. What is changing is the number, variety and intensity of automated systems crawling the web.
AI bots add to an already complicated crawler landscape
Traditional search engines such as Google and Bing have been crawling websites for decades. Website owners now have another growing group to consider: crawlers operated by AI companies and services.
These bots do not all have the same purpose or behaviour. Some gather information for search and retrieval, some are associated with training models, and others may access websites on behalf of individual AI users.
That makes a blanket policy increasingly difficult. Blocking all bots would be disastrous for conventional search visibility. Blocking all AI crawlers could similarly restrict an organisation’s visibility through emerging AI-search services. But allowing every automated system to crawl every possible URL without restriction is not sensible either.
The question is no longer simply whether bots should be allowed. It is which bots should have access to which parts of the website, at what rate, and at what cost to the underlying infrastructure.
Fix the underlying problem, not just the symptom
Our immediate task was to stabilise the website, but simply blocking whichever crawler happened to be causing the most traffic would not have addressed the underlying weakness.
We therefore changed the calendar functionality itself. Navigation was restricted to a sensible date range. Requests outside that range were rejected early with a lightweight response, before WordPress performed the work required to render a complete page. We also adjusted the site’s crawling instructions to discourage legitimate crawlers from following unnecessary calendar URL patterns.
This is an important principle in bot management. If a website exposes an effectively infinite number of computationally expensive URLs, blocking one crawler may only solve the problem until another one discovers them.
Good application architecture should reduce the opportunity in the first place.
But the infrastructure in front of WordPress matters too
Code-level fixes are only one layer of protection. A modern, business-critical WordPress website should not have to process every incoming request before deciding whether that request is useful.
This is where a CDN and web application firewall (WAF) become important. A properly configured WAF can inspect and control traffic before it reaches WordPress. Rules can be applied to particular traffic patterns, automated clients, countries, URLs or other characteristics without changing the configuration of the underlying web server.
Rate limiting can help prevent one source from making excessive numbers of requests. Bot-management tools can provide visibility into automated traffic. CDN caching can prevent legitimate repeated requests from unnecessarily reaching the origin server at all.
The principle is defence in depth: deal with unwanted or excessive traffic as far away from the WordPress application as practical, while keeping the application itself efficient and appropriately configured.
Robots.txt is useful, but it is not a security control
One of the measures we took was to update the website’s robots.txt file to tell legitimate crawlers not to explore unnecessary calendar URL patterns.
That helps responsible crawlers use their crawl budget more efficiently, but it is important to understand what robots.txt does and does not do. It is an instruction, not an access-control mechanism.
A crawler can choose to ignore it. Malicious bots almost certainly will, and even legitimate automated systems do not necessarily behave identically.
So robots.txt can be part of good crawler management, but it cannot replace proper server, CDN or firewall controls.
Monitoring is becoming increasingly important
Perhaps the most significant lesson is that organisations need better visibility into automated traffic.
A website can appear to have a “performance problem” when the application itself is perfectly capable of serving its normal human audience. The problem may actually be an automated system making tens of thousands of requests to functionality that ordinary visitors barely use.
Without appropriate logging and monitoring, the distinction is not obvious.
For a managed WordPress provider, questions such as these are becoming increasingly relevant: Which bots are visiting the website? Which URLs are they requesting? How much of that traffic is being served from cache? Which requests are reaching WordPress? Are crawlers repeatedly accessing expensive dynamic functionality? Are useful search and AI crawlers getting through? And are unwanted bots being stopped before they consume significant resources?
This is becoming part of normal website operations.
Cheap hosting and managed infrastructure are not the same thing
There is a wider commercial point here too.
In the current economic climate, we understand why organisations are examining hosting costs. This is particularly apparent in the charitable and non-profit sectors, where funding and budgets are under considerable pressure.
But comparing hosting services purely by disk space, bandwidth and monthly price misses an increasingly important part of the equation.
Infrastructure needs management.
When a WordPress website experiences unusual automated traffic, somebody needs to determine whether the problem is malicious activity, a legitimate crawler behaving inefficiently, a flaw in the application, inadequate caching, a firewall configuration problem or some combination of these. Then somebody needs to know what to do about it.
A mass-market hosting account may keep a server running. That is not the same as having a competent team actively responsible for the performance, security, and resilience of the website running on it.
The age of AI makes infrastructure more important, not less
There is a tendency to think about AI readiness primarily in terms of content: writing useful information, structuring it clearly and making sure AI systems can find and understand it.
All of that matters. But those same AI systems are also generating traffic.
As the number of automated agents and crawlers increases, websites need infrastructure capable of distinguishing useful access from wasteful or harmful activity.
That means sensible application architecture, effective caching, appropriate CDN and WAF protection, bot management, monitoring and people who understand how those components work together.
The incident we investigated was not caused by one dramatic vulnerability. It resulted from automated crawlers finding functionality that allowed them to request almost unlimited numbers of expensive pages.
The solution was not simply to “block the bots”. It was to understand what they were doing, remove the underlying crawler trap, reduce unnecessary server work and strengthen the infrastructure around the website.
In the age of AI bots, website infrastructure is no longer just about making pages load quickly. It is increasingly about deciding who and what gets access to your website, what resources they are allowed to consume, and how you keep the service available for the people it was actually built for.
'
'