Support

Admin Tools

#41278 AWS Elastic Load Balancer Health Checker Triggers Critical File Modified Email Notification

Posted in ‘Admin Tools for Joomla! 4 & 5’
This is a public ticket

Everybody will be able to see its contents. Do not include usernames, passwords or any other sensitive information.

Environment Information

Joomla! version
5.2.0
PHP version
8.2.23
Admin Tools version
7.6.2

Latest post by nicholas on Sunday, 03 November 2024 09:08 CST

mbond87

Hi Nicholas,

I am running a Joomla site in AWS that is behind an Application Load Balancer. In this load balanced configuration (I'm running 2 EC2 instances).  As part of the load balancer operation the devices initiates a Health Check to confirm that an EC2 instance if healthy.  Unfortunately, whatever is being sent is Critical File Modified notification and I get 3 or 4 emails per minute to my inbox indicating that the critical files have been modified.  However, when I go manually check the files that the notification claims have been modified, I don't see any evidence that this is true. The last modified dates have not changed and the file contents seem to be exactly the same.

For comparison, I am running a Joomla site in AWS that is behind a Network Load Balancer. I do not have the same problem with Network Load Balancer as I do with the Application Load Balancer.  Now the Network Load Balancer operates at Layer 4 while the Application Load Balancer operates at Layer 7.  I don't know if that makes a difference or not.

Unfortunately, the Critical File Modified emails are so persistent that I am forced to disable the System - Admin Tools plugin.  Of course that means the Admin Tools component is not working.

So, my questions for you are:   

  1. What triggers the Critical File Modified email notification and why might it be firing when the files do not appear to have been modified?
  2. Is there any way to whitelist the IP address for the Health Checker that prevents the emails from firing while still protecting my site?

I have attached an image of the notification for you to see.  

Any suggestions you have for what I should check will be greatly appreciated!

Thanks.

Michael

 

 

nicholas
Akeeba Staff
Manager

For your reference, the documentation for this feature is in https://www.akeeba.com/documentation/admin-tools-joomla/web-application-firewall.html#waf-configure-joomla-hardening under "Monitor Critical Files".

You can temporarily disable this feature instead of Admin Tools in its entirety. Don't throw the baby out with the bath water.

As per the documentation, the files being checked are index.php, administrator/index.php, configuration.php, and each of your site's templates' index.php, error.php and component.php files (not every template has all files).

Either whatever you are doing modifies one of these files even if only by updating its last modified date, or it messes up the #__admintools_storage table with the key criticalfiles which is where the information about these files is stored.

Frankly, I consider it unlikely that any of that happens because of the health check. I believe the health check simply exposed a problem in your configuration.

You basically have an application server cluster. How are the files synchronised between the cluster nodes? For example rsync by default does not update the last modified time; you have to explicitly use the -t option (yeah, I know…). If you have a cluster sharing a single database (naturally!), but each node has a different last modified time for its own copy of each file then this situation would cause the problem you described.

Let's say we have index.php in two nodes, Node A and Node B. The last modified time on Node A is 1730358773 and on Node B it is 1722399171; I'll shorten them as '773 and '171 for readability purposes. A request is made and it is handled by Node A. There is no saved information, so it reads the last modified time ('773) and saves it to the database. Another request is made, and it is handled by Node B. The database says the file's last modified time should be '773 but it is '171. An email is sent, and the database is updated with the '171 value. Another request comes in and it is handled by Node A again. The database says the last modified time should be '171 but it is '773. An email is sent, and the database is updated with the '773 value. And so on and so forth.

I have a suspicion that your load balancer wasn't working before, all requests were handled by the same server. When you added the health check it started making HTTP requests to your application server nodes. So, every time the health check runs it goes Node A, to Node B etc. That's why you get these periodic emails.

The correct solution is to sync your filesystems completely, including the last modified time, not because of Admin Tools, but because Joomla itself relies on the last modified time of certain files to operate correctly.

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

mbond87

Thanks Nicholas.  You were correct.  The underlying problem was the sync process between the two servers.  Once I fixed that, the issue was resolved.

Thanks again.

Michael

nicholas
Akeeba Staff
Manager

You're welcome! I am glad my approximately 20-year-old frustrations with rsync came to good use :)

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

Support Information

Working hours: We are open Monday to Friday, 9am to 7pm Cyprus timezone (EET / EEST). Support is provided by the same developers writing the software, all of which live in Europe. You can still file tickets outside of our working hours, but we cannot respond to them until we're back at the office.

Support policy: We would like to kindly inform you that when using our support you have already agreed to the Support Policy which is part of our Terms of Service. Thank you for your understanding and for helping us help you!