Search and you'll find several prior discussions, e.g.:
Bots eating up all the bandwidth
Excluding Bots from accessing anything on my board
Claudbot attack
Many use .htaccess and blocking rules.
I recommend Cloudflare and whitelisting their server IPs with iptables and dropping all others. You can then tune the web access firewall (WAF) to filter offenders--the biggest now being AI bots. In fact, they have a whole category dedicated to them. You can also block "stealth" bots by their behavior, e.g., any IP address making 10 page requests in 10 seconds gets put in timeout.
The advantage of a CDN proxy, whether Cloudflare or other, is that "bad" traffic doesn't even reach your site. It's blocked by their servers. The other advantage is caching of common resources like images and JavaScript files. My motto on speed optimization is simple: Your server's fastest response is the one it doesn't have to do.
Bots eating up all the bandwidth
Excluding Bots from accessing anything on my board
Claudbot attack
Many use .htaccess and blocking rules.
I recommend Cloudflare and whitelisting their server IPs with iptables and dropping all others. You can then tune the web access firewall (WAF) to filter offenders--the biggest now being AI bots. In fact, they have a whole category dedicated to them. You can also block "stealth" bots by their behavior, e.g., any IP address making 10 page requests in 10 seconds gets put in timeout.
The advantage of a CDN proxy, whether Cloudflare or other, is that "bad" traffic doesn't even reach your site. It's blocked by their servers. The other advantage is caching of common resources like images and JavaScript files. My motto on speed optimization is simple: Your server's fastest response is the one it doesn't have to do.
Statistics: Posted by HB — Fri Jan 24, 2025 2:36 am