Something occurred to me recently that feels embarrassingly obvious in hindsight.

A lot of WordPress sites now see requests from things like GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot. We recognize the name, maybe see it in a security log or access log, and mentally translate that into “OpenAI, Anthropic or Perplexity visited my site.”

But that is not what the log actually proves. What it proves is that somebody sent a request containing that User-Agent. Anyone can send User-Agent: GPTBot.

This isn't even a new problem. Fake Googlebot traffic has existed for years, which is why Google provides ways to verify whether something claiming to be Googlebot actually originates from Google. AI crawlers seem to have brought the same old problem back in a new form.

On WordPress this can become more than just a misleading log entry. Imagine you block generic scrapers but allow known AI crawlers, exclude certain bots from rate limiting, bypass page cache for specific crawlers, allow some crawlers access to content that ordinary scraping tools should not receive, or simply analyze AI crawler activity to see which providers are accessing which URLs.

If the identity behind those rules is based only on the User-Agent, a scraper may only need to call itself GPTBot or ClaudeBot to receive the same treatment.

The consequences can range from merely annoying to fairly significant. A fake bot can scrape content while masquerading as a crawler you intentionally allow. It can consume PHP, database and server resources that you thought were being used by a legitimate provider. It can also distort your WordPress, security or access-log analysis, so a report saying “GPTBot requested 500 pages” may not mean OpenAI requested those pages at all.

If your site grants known AI crawlers different access than unknown scrapers, a spoofed identity can potentially turn that trust decision against you.

Even people who simply block all AI bots have an interesting problem here. If you see 1,000 “GPTBot” requests and block them all, how many were actually OpenAI? Maybe all of them. Maybe none of them. Without verification, you don't know.

That seems like an important distinction: “This request claims to be GPTBot” is not the same statement as “This request has been verified as coming from GPTBot.”

Security plugins are very good at blocking traffic, rate limiting it and detecting suspicious behavior. But blocking is a policy decision. Verification is an identity question. I’m starting to think WordPress has mostly concentrated on the first question while barely discussing the second one for AI crawlers.

So I'm curious: when you see GPTBot, ClaudeBot, OAI-SearchBot or PerplexityBot in your WordPress or server logs, do you actually verify the identity, or have you basically been trusting the User-Agent?

Google's documentation on the underlying problem, for reference:
https://developers.google.com/search/docs/crawling-indexing/googlebot

When a website has performance problems, the usual response is to optimize what is already there.

Minify CSS.
Delay JavaScript.
Remove unused styles.
Compress assets.
Improve the PageSpeed score.

But there is a more basic question:

Why deliver something that a device does not need in the first place?

A phone does not necessarily need the same frontend resources as a desktop. Yet conventional responsive web design often sends the same CSS and JavaScript to every device and lets the browser decide which parts are actually used.

That works. But it also means we may spend time optimizing resources that never needed to be delivered.

There is an older concept called RESS - Responsive Design + Server-Side Components.

The idea is simple:

Desktop → desktop.css
Tablet → tablet.css
Phone → mobile.css

The server decides which resource set is appropriate before the page reaches the browser.

And this does not mean abandoning responsive web design. Each device-specific stylesheet can still use normal media queries and responsive techniques.

RESS simply adds an earlier decision:

First decide what the device actually needs. Then optimize what remains.

Maybe "performance optimized" should therefore mean more than minification, caching and delayed loading.

Maybe users should also start asking theme and plugin developers:

Does your solution avoid delivering frontend resources that the device will never need?

EDIT: Judging from some comments, my CSS example may have been too narrow. RESS is not about the server replacing media queries. I made a small demo showing the actual idea: the server delivers different PHP, HTML and assets depending on the device, while responsive design can still be used normally inside each version.

Demo: https://www.imedes.dev/ress/

Full disclosure: I’m the developer.

Most cache-crawler setups start with a normal sitemap. But an SEO sitemap answers “which URLs exist?”, not “which URLs are actually worth warming?”

URLs Most Wanted records the URLs real visitors request, tracks their demand, validates them server-side, and generates a separate sitemap for the built-in LiteSpeed Cache crawler.

The idea is simple:

- URLs with demonstrated visitor demand can be warmed first.

- Rarely used URLs do not have to consume crawler resources repeatedly.

- Cacheable visitor paths such as pagination or filtered listings can be included even when they do not belong in a normal SEO sitemap.

The plugin does not replace LiteSpeed Cache or its crawler. It only gives the crawler a demand-based source of URLs.

Ask ChatGPT about URLs Most Wanted

It is free and available on WordPress.org:

https://wordpress.org/plugins/urls-most-wanted-for-litespeed-lscache/

I’m curious whether demand-based cache warmup would be useful on the WordPress sites you manage, or whether you generally prefer to crawl every available URL.

https://preview.redd.it/212zspfv8kch1.png?width=1319&format=png&auto=webp&s=7c6f2ab21d3844397ce16d3bfebbf6e287a1b697

I’ve been looking again at something that has bothered me about Wordfence for years.

Wordfence runs long-running PHP processes, especially its scanner. On LiteSpeed servers, those processes must not simply be terminated when the client connection disappears or certain connection timeouts are reached.

That part is perfectly reasonable. The problem is how Wordfence handles it. Wordfence recommends adding a rule like this to .htaccess:

RewriteRule .* - [E=noabort:1] 

or even:

SetEnv noabort 1 

The important part here is .*.

This does not protect only the Wordfence scanner. It applies noabort to every matching PHP request on the site.

LiteSpeed itself explicitly warns against applying noabort globally and recommends restricting it to the specific scripts or requests that actually require long-running execution.

And there is a very good reason for that.

noabort changes how LiteSpeed handles PHP processes when a connection disappears. Together with noconntimeout, a PHP process can continue running far beyond what many administrators would normally expect.

And before someone points at PHP's max_execution_time: on LiteSpeed/LSPHP that is not necessarily the hard process lifetime limit people assume it is.

I have tested this myself. (I am a LiteSpeed developer)

A request can reach its configured PHP execution time while the associated process continues running. The LiteSpeed-side mechanism that can impose a hard process-time limit is LSAPI_MAX_PROCESS_TIME.

You can observe the difference directly at process level. So the issue is not that Wordfence needs noabort. The issue is this:

Why does a security plugin remove a server-side process protection globally when only a small number of its own requests actually need that exception?

And there is another part of this story that I think matters even more:

This is not a newly discovered edge case. Wordfence has been aware of this issue for at least a decade.

I raised this problem directly with Wordfence years ago. LiteSpeed has also made clear for a long time that global noabort should be avoided when the exception can be restricted to the requests that actually require it.

Yet the broad configuration is still being recommended. That history is what makes this particularly difficult to understand. Software contains mistakes. Security software contains mistakes too. That is not the issue.

The issue is when a security vendor is made aware that its own configuration unnecessarily weakens a server-side protection, the server vendor explicitly warns against that same configuration, a technically straightforward way to scope the exception exists, and the unsafe recommendation remains in place for years.

At that point, this is no longer just an overlooked configuration detail. It is a consciously unaddressed security trade-off imposed on Wordfence users. And technically, the solution is not complicated.

With mod_rewrite, noabort can be enabled only for the exact Wordfence request that actually requires it.

Conceptually:

RewriteCond %{QUERY_STRING} ... RewriteRule ^wp-admin/admin-ajax\.php$ - [E=noabort:1,E=noconntimeout:1] 

instead of:

RewriteRule .* - [E=noabort:1] 

Same Wordfence functionality. Very different security boundary.

Millions of WordPress users install Wordfence specifically because they trust it to improve the security of their sites. A security plugin should not unnecessarily weaken process controls for unrelated PHP code just because one of its own components needs an exception. And if a security vendor has known about that problem for at least a decade and still does not fix it, users should at least have the option to fix it themselves.

I've noticed that my server receives a surprisingly large number of requests every day for random URLs that don't exist.

Every one of those requests eventually results in a 404, but WordPress still has to bootstrap, load plugins and process the request before it can determine that the page doesn't exist.

With enough of these requests, that seems like a lot of unnecessary PHP work for something that will only return a 404 anyway.

How do you deal with this?

Do you just accept it as normal WordPress overhead, handle these requests before they reach WordPress, or use some other approach?

https://preview.redd.it/4k06jdd5ovkh1.png?width=1491&format=png&auto=webp&s=03eb110d01cb42c41a8b9887fd9de3d5d472f514