Support

Admin Tools

#32781 Had to turn Admin Tools off - It's Blocking Citrix Users

Posted in ‘Admin Tools for Joomla! 4 & 5’
This is a public ticket

Everybody will be able to see its contents. Do not include usernames, passwords or any other sensitive information.

Environment Information

Joomla! version
n/a
PHP version
n/a
Admin Tools version
n/a

Latest post by on Friday, 08 May 2020 17:17 CDT

GuiGuy
After the last Admin Tools update (5.6) a small group of our users reported that when they attempted to link to https://mbea.com.au they were greeted by a 403 error page. After considerable effort between myself, the affected users and our hosting system we have identified

  • The five affected users were all coming from organisations running on Citrix
    The 403 was being generated by the CMS, specifically admin tools


I worked my way through what I could find online, notably
* Disabled all of Admin Tools Geoblocks
* Allowed Anonymous proxies
* Replaced .htaccess with the stock standard Joomla one

None of that resolved the issue.

My knowledge is insufficient to dig further, so more out of frustration I have now totally disabled Admin Tools. Once I did that the affected users were able to access the site.

Are you able to suggest a way forward from here? For example would removing Admin Tools and then re-installing help me at all?

Thanks

nicholas
Akeeba Staff
Manager
Disabling your security as a whole is never a good idea. It's the equivalent of torching your house to make a smoke detector stop beeping because of a low battery.

I suppose that by "Citrix" you mean their desktop virtualization software which is presumably installed somewhere under your organization's control. This would mean that all of these users connecting to your site appear to be coming from the same IP address or IP address cluster (where Citrix' server software is installed). This means that one user doing something to get themselves blocked would impact all of the other users connecting through the same Citrix server since they all appear to be coming from the same IP address. That IP address was either auto-blocked or someone with Super User privileges manually blacklisted it in Admin Tools' IP Blacklist in error.

Unblocking these IPs and then adding them to "never block these IPs" would address your issue.

If unsure, you can clear all auto-blocked IPs, the IP autoblocking history, the security exceptions log and the IP blacklist. Then you can have a user connecting through Citrix virtual desktop trigger a security exception and see which IP gets logged in the Security Exceptions Log. You can then put it in the "Never block these IPs" list to prevent it from being blocked.

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

GuiGuy
Sorry, I should have mentioned that I did not see the IP addresses of the affected users, which we have, blocked anywhere in the WAF. They were also whitelisted as you mentioned.

I understand it's a bad idea to turn the admin tools, but what is the point if users cannot access the site?

I have at the moment turned it back on and will decide what to do next week.

Cheers

nicholas
Akeeba Staff
Manager
Admin Tools will either block a request based on the user doing something or it will block an IP either permanently or temporarily.

In the first case there is an entry in the Security Exceptions Log.

In the second case the IP address is either in the IP Blacklist (permanent banning) or in the Auto IP Blocking Administration. However, if you see an IP in the Auto IP Blocking Administration you also see it in the Security Exceptions Log.

Unless you are talking about users being unable to access the backend of the site (/administrator) in which case you should be looking in the IP Whitelist, assuming that you have enabled the IP whitelist feature in the Configure WAF page.

There is no mode of operation which would silently block an IP address with no other information provided anywhere.

Moreover, if an IP address is whitelisted it is not subject to any of the security checks.

At this point I have to assume that you are either missing something or you are miscommunicating what is going on.

Please tell me what happens when an affected user tries to access the site. First of all, is it the public frontend or the administrator backend?

The error message they get, does this message match the "Custom message" or the "Show this message to blocked IPs" in the Configure WAF page and if it does, which one?

What is the setting of "Enable IP workarounds" in the Configure WAF page?

Do you have access to the site?

Are you behind the same virtual desktop server or not?

Have you checked your server's access logs to make sure that the IP address of the virtual desktop server is actually the one your server sees when the user tries to access your site?

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

GuiGuy
Hi Nicholas
Thanks for the detail.

I may have miscommunicated
We have identified five users that get the 403 error.

Each is from a different financial institution in Australia. I should think that their IT administration can best be described as paranoid.

The five institutions each have their own citrix virtualisation software to service their offices and branch offices.

I will follow through on your suggestions next week.

Cheers


GuiGuy
I suppose what I find confounding, at the risk of sounding naive if not stupid, the site has been running for three years, with admin tools. The five affected users have always been able to access the site and log in. Until about a month ago.

Technology is great, but only when it works.

Cheers

nicholas
Akeeba Staff
Manager
Computers work in predictable ways. The problem is that they are what we call a "chaotic system" i.e. a system that's too complex for human minds to fully hold and where a small change can cause a seemingly unrelated, big change. I refuse to throw my hands in the air and declare it's the computer god's will or something like that. I have found that analyzing a problem lets us understand it better and come up with a solution.

The paranoia of the IT department or the timing of this issue is irrelevant. I am not saying this as an aphorism but because I understand the nature of IP blocking.

When a client (another computer) sends a request to your site it sends with it the IP address as part of the TCP/IP packets used to transfer the request data. This information is picked up by Apache and put in the REMOTE_ADDR environment variable. PHP picks that up and puts it in the $_SERVER['REMOTE_ADDR'] super global array element.

However, when you have things like a CDN, a caching proxy, an NginX server set up as a reverse proxy or generally any non-transparent network equipment between the client and the server that address is not the client's address. It's the non-transparent network equipment's address.

In most cases an HTTP header X-Forwarded-For is sent with the request. Look it up in Wikipedia for a fairly simple description of how it works. That header is not picked up automatically by Apache and PHP. That's where Admin Tools' "Enable IP Workarounds" option in the Configure WAF page comes into play. When it's set to Yes it tells Admin Tools to use that information instead of the IP address Apache determined from the TCP/IP information of the incoming request. The reason this is not enabled by default is that when it's not necessary for your server setup it can be used maliciously.

If that option is already enabled and your clients are still being blocked then it's possible that all clients appear to be coming from the same IP address and one of them did something that blocked them.

There's no way they are auto-blocked by Admin Tools without their IP being stored anywhere in Admin Tools. Admin Tools won't block random IP addresses for no reason.

If you work methodically through this issue you CAN fix it. For example, you could try disabling IP auto-blocking in Admin Tools and check if that works. If it does work then the IPs given to you are wrong. Check the Security Exceptions Log to see which IP addresses are really being recognized as triggering malicious requests and ask the IT departments to tell you if any of these belongs to them. Maybe the started doing some common routing in front of their Citrix Remote Desktop servers, effectively changing the IP address your site sees.

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

GuiGuy
A quick update & thanks for the detail.
The WAF IP blocking history and administration show no IP addresses blocked
The IP Workarounds option had already been set to YES.
I have also whitelisted three of the five affected users' IP addresses

None of this has made any difference. So far the only thing that has worked is to disable Admin Tools.

I'll keep at it, but it is frustrating

nicholas
Akeeba Staff
Manager
Here are all the places you need to check:
  • Web Application Firewall, Site IP Blacklist
  • Web Application Firewall, Security Exceptions Log
  • Web Application Firewall, Auto IP Blocking Administration
  • Web Application Firewall, Auto IP Blocking History


Alternatively, use Web Application Firewall, Unblock an IP to delete any records with that IP address.

Go to Web Application Firewall, Configure WAF, Basic Features. Is "Allow administrator access only to IPs in Whitelist" enabled? If so, go to Web Application Firewall, IP whitelisting and make sure the IP addresses are in that list. Otherwise access to the administrator are of the site will be disabled without it being logged.

Do note that if access is disabled if a. IP whitelisting is enabled but the IP address is not in the IP Whitelist; b. IP blacklisting is enabled and the IP is in the Site IP Blacklist; c. the IP has been automatically blacklisted temporarily (Auto IP Blocking Administration) or permanently (Auto IP Blocking History); or d. the IP is in enough Security Exceptions Log entries to trigger automatic temporary or permanent IP blacklisting (case c) THEN there is no log entry generated about the blocked request. This is a reasonable feature to prevent your database crashing under the weight of someone persistently trying to attack your site after they have been IP blocked.
If that still doesn't help repeat all the steps above AND go to Web Application Firewall, Configure WAF, Basic Features and set "Enable IP workarounds" to No. Now have them retry accessing the site.

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

System Task
system
This ticket has been automatically closed. All tickets which have been inactive for a long time are automatically closed. If you believe that this ticket was closed in error, please contact us.

Support Information

Working hours: We are open Monday to Friday, 9am to 7pm Cyprus timezone (EET / EEST). Support is provided by the same developers writing the software, all of which live in Europe. You can still file tickets outside of our working hours, but we cannot respond to them until we're back at the office.

Support policy: We would like to kindly inform you that when using our support you have already agreed to the Support Policy which is part of our Terms of Service. Thank you for your understanding and for helping us help you!