I'm running 3.3.14 and have been dealing with a large amount of bot traffic since the 13th. It was enough that Apache was running out of memory after 5-10 seconds of uptime. The site is currently stable, but I thought that I would make a note of what I did for anyone else running into the same problem, and also to hear any suggestions.
1) I have been doing User Agent filtering to ban most bots for more than a year now. The current bot traffic mostly does not have User Agent strings.
2) At the peak of the bot DoS, when the site had been down for several hours, I turned apache back on. The site went down and it immediately began throwing memory errors in the log, but the bots were all still there hammering away. I sampled 5 minutes of traffic, and banned the /16 of every IP address involved with iptables -j DROP, for about 10% of the internet. This allowed me to put the site back up (at the cost of blocking a number of real users).
3) I then took most of the forums private and turned off the ip banning. This allowed the site to come back up. The site is still private
4) Even with the private forums, I continued to have load problems. CPU alerts were coming in from my provider, and search was down. Because phpbb's logging options do not allow me to associate the apache logs with user logs, I added some code to session.php to capture more information about users in the php error log:
I added the following code just before the comment // Did the session exist in the DB?
And I added the following just inside the following if statement:
This allowed me to make some determinations about the bots
a) The bots all seemed to pass a session id in the target URL "sid=<a number>". The session ids were all unique and did not exist in the database. I experimented with blocking them at the apache level with mod_rewrite, but it broke login
b) In the period I sampled, there were 100,000 bot users from 80,000 unique IP addresses, compared to 5,000 unique anonymous web users, compared to 20-30 logged in users.
5) Combining all the above, I made a more targeted list of ~1500 /16 IP blocks and blocked those through iptables -j DROP. The site seems to be down to normal load, and the traffic seems only to be 2:1 bot users to anonymous web users.
1) I have been doing User Agent filtering to ban most bots for more than a year now. The current bot traffic mostly does not have User Agent strings.
2) At the peak of the bot DoS, when the site had been down for several hours, I turned apache back on. The site went down and it immediately began throwing memory errors in the log, but the bots were all still there hammering away. I sampled 5 minutes of traffic, and banned the /16 of every IP address involved with iptables -j DROP, for about 10% of the internet. This allowed me to put the site back up (at the cost of blocking a number of real users).
3) I then took most of the forums private and turned off the ip banning. This allowed the site to come back up. The site is still private
4) Even with the private forums, I continued to have load problems. CPU alerts were coming in from my provider, and search was down. Because phpbb's logging options do not allow me to associate the apache logs with user logs, I added some code to session.php to capture more information about users in the php error log:
I added the following code just before the comment // Did the session exist in the DB?
Code:
if (!isset($this->data['user_id'])){ error_log('session id did not exist:' . strval($this->session_id));}Code:
error_log('user id: ' . strval($this->data['user_id']));a) The bots all seemed to pass a session id in the target URL "sid=<a number>". The session ids were all unique and did not exist in the database. I experimented with blocking them at the apache level with mod_rewrite, but it broke login
b) In the period I sampled, there were 100,000 bot users from 80,000 unique IP addresses, compared to 5,000 unique anonymous web users, compared to 20-30 logged in users.
5) Combining all the above, I made a more targeted list of ~1500 /16 IP blocks and blocked those through iptables -j DROP. The site seems to be down to normal load, and the traffic seems only to be 2:1 bot users to anonymous web users.
Statistics: Posted by textkit — Fri Jan 24, 2025 2:01 am