Support

Admin Tools

#40900 Follow up on #40867 Many emails sent at the same time, for the same blocked IP address

Posted in ‘Admin Tools for Joomla! 4 & 5’
This is a public ticket

Everybody will be able to see its contents. Do not include usernames, passwords or any other sensitive information.

Environment Information

Joomla! version
4.4.5
PHP version
8.2
Admin Tools version
Latest

Latest post by nicholas on Monday, 08 July 2024 07:44 CDT

5uwebsite

Hi there,

 

Thank you for your help on ticket #40867. I thought I had correctly configured Admin Tools so I closed it. However I am still getting about 300 emails for the same IP blocking in just a few minutes.

May I know whether this is the correct setting for sending maximum 1 email per hour for blocking message?

https://mrkr.io/s/668866dfa674d7e635649bfe/0

 

If that is the case, could you help me to inspect why we got about 300 emails in just a few minutes?

Thank you and admin login information can be provided upo  requested.

 

 

nicholas
Akeeba Staff
Manager

The problem here is what I told you about the eventual consistency of the database. Since there are hundreds of requests coming in at the same time, all these requests will NOT see that other requests being processed concurrently (at the same time) have already blocked that IP address. They will also not see that other concurrently processed requests have sent emails, therefore the sending limits will not be enforced.

The best thing you can do is disable sending emails for each blocked request.

It might sound counter-intuitive but this will result in the IP of the attacker getting blocked faster in this case (after several dozen requests instead of several hundred). The reason is that sending emails takes up a lot of server resources. By sending emails you make everything slower as there's only so much a server can do in a given amount of time – the limit is hard i.e. imposed by the capabilities of the hardware and the physical network it's connected on.

Also keep in mind that sending emails for blocked requests is something we only recommend as a troubleshooting step. It is very useful when you have just set up Admin Tools to see why you keep getting blocked when you do not have the experience which tells you the exact same information is available in Admin Tools, Web Application Firewall, Blocked Requests Log. This is the reason this feature exists and why it's enabled by default. It's training wheels. Once you get the hang of it, you are meant to disable it because it has an adverse impact on the performance of your site and the effectiveness of the WAF during a real world attack.

I am going to add a permanent backend notification when this feature is enabled, letting you know it's a set of training wheels and let you disable it with a single click in the next version of Admin Tools (well, and an option to hide the message because some people want to do the wrong thing despite all warnings; as per the cardinal UNIX rule, my job is not to ask the user why they want to shoot their feet, but to provide the biggest, meanest gun which will shoot both of their feet clean off the first time).

Moreover, I am trying a workaround to this issue in another piece of software we make, Panopticon. I am using a database trick which should prevent this kind of situation by providing a more consistent view of the database. The key word here is “should”. The MySQL documentation uses rather cagey wording describing how table locking will perform under these conditions. I want to see it in action with a more technical audience before using the implicit transaction for InnoDB tables trick in a product aimed at the general market like Admin Tools. That is to say, I have not given up on this kind of issue just because database servers work in a certain way, I am doing my research and development. If I am right about what I am trying here, issues like that will no longer exist with Admin Tools by Q1 2025.

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

5uwebsite

Hi Nicholas,

Thanks again for the super fast and detailed reply, during the weekend! Just can't believe how impressive the Akeeba support is, compared to all other developers and companies we work with. Always resolve your issue with great support!

 

Yes. I agree that we should probably turn the email notifications off by default or make it one-click to turn off such notifications, if we cannot control such concurrent emails. It might be a bit difficult for most people to find where to turn it off, if my understanding is correct:

https://mrkr.io/s/66899322aef110cf65f57cd6/0

 

Or, alternatively, do you think we could wait five minutes before sending the emails? While many people will be beneifted from the immediate email notifications, a three to five minute wait is not a big deal if they found that they were locked out. If waiting a few minutes could let us build a list of blocked ip addresses and just send out one email, will that solve the problem?

 

I am not a technician so I could only think of non-technical solutions. Hopefully that works and I could contribute something to this excellent tool.

 

Thanks again!

 

nicholas
Akeeba Staff
Manager

t might be a bit difficult for most people to find where to turn it off

It should be fairly easy. The option I am talking about is "Email this address on blocked request" a bit further down the page from where you pointed at. As documented, if you empty this field, no emails will be sent every time a request is blocked.

Or, alternatively, do you think we could wait five minutes before sending the emails?

This solves nothing, really. The database concurrency problem is not solved, and as soon as the attack is being blocked your server gets an even greater load for sending dozens to hundreds of emails. A solution introducing a different problem isn't a solution. Hence the work on solving the database concurrency and consistency issues.

Preliminary tests show that using table locks won't solve the problem entirely, but it will reduce the number of extraneous requests to the number of concurrent threads being processed on the server i.e. between 2 to 10. So, if you configured Admin Tools to block an IP after 3 blocked requests in 1 minute you will get between 5 and 13 before it's blocked (and no more emails are sent) instead of the previous situation of a few hundreds. The drawback is that if you have a busy server which is struggling to keep up with legitimate requests you are introducing a further delay of between a fractional millisecond to low single digits milliseconds. In the grand scheme of things we're talking about under 1% performance impact at worst; on the vast majority of sites it will be orders of magnitude smaller than the natural variance in server response. I told you I was working on it; as it happened, that's what I was working on yesterday :D

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

5uwebsite

Thanks again, for the super fast and detailed reply! I thought the only purpose of this email is to let the administrator know that someone was blocked, and if that is you, here is the solution to login. Sorry I am not a tech savvy user, so my opinion could not help. What I am thinking, is simply hold the emails before sending, so that it will not send out the same email for the same blocking. 

 

You are professional, so I am pretty sure you will get the best solution out for the world's users relying on Admin Tools. Thanks again!

nicholas
Akeeba Staff
Manager

Yes, you are one of the few people who understand what the email is for :) What nobody understands – and I take responsibility for not having explained it in the documentation – is that it's only meant to be used during the initial configuration, and for troubleshooting.

When someone is configuring Admin Tools for the first time on a site it's absolutely expected that they will experience some false positives. Finding these and deciding what to do with them – exceptions, contact the extension's developer, etc – is part of the configuration process. If we could magically detect everything automatically there would be no configuration options. These false positives will definitely end up blocking the person doing the configuration. The emails are there to provide an easy way out with the Rescue Mode.

Likewise, it's possible that making a big change on the site, or installing a new version of an extension, may result in something which is throwing a lot of false positives. Or you may have some users complaining they get blocked but you can't figure out how or why. Using these emails is a good way to fix these issues.

I hope this is more clear now. I am also going to update the documentation with this information so it's a bit more clear why these features exist and how they are meant to be used.

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

Support Information

Working hours: We are open Monday to Friday, 9am to 7pm Cyprus timezone (EET / EEST). Support is provided by the same developers writing the software, all of which live in Europe. You can still file tickets outside of our working hours, but we cannot respond to them until we're back at the office.

Support policy: We would like to kindly inform you that when using our support you have already agreed to the Support Policy which is part of our Terms of Service. Thank you for your understanding and for helping us help you!