There has been a number of reported issues regarding the installation of Akeeba Backup 8, Admin Tools 6 and Akeeba Ticket System 4. In this article we will tell you how to address them and why they happened (if you are curious).
Akeeba Backup 8, Admin Tools 6 and Akeeba Ticket System 4 were released on March 2021. These are very major upgrades because they change the backend and frontend frameworks used in these extensions.
Due to the nature of this update there has been a higher than usual number of installation issues. In this article we will tell you how to address them.
For most users: do this before the update
If you have ever installed a Professional version of our software you will need to disable (unpublish) the following plugins before installing an update. Please note that your site may not have some or any of these plugins.
- Installer – Akeeba Backup Professional
- Installer – Admin Tools Professional
- Installer – Akeeba Ticket System Professional
If you are a Professional subscriber to any of our extensions make sure that you have entered your up-to-date Download ID in each extension. If not, you will need to enter the Download ID and reset Joomla's update cache. All of these steps are explained in the following documentation pages:
If the first update attempt fails wait a few minutes and retry. Joomla has a hidden cache for plugins which might get in your way.
Most people can stop reading now
The instructions above work for the vast majority of sites. The instructions below are only required for very few sites. Most of the rest of the article is optional technical information for the curious.
If you have lost access to the backend of your site
We saw that a number of sites still had installed some old plugins we discontinued between 1 and 8 years ago. Because the new versions use a new major version of the backend framework and these plugins still rely on the old framework they may cause inability to access your backend. These are already deal with as of Akeeba Backup 8.0.1 and Admin Tools 6.0.2 (this issue does not apply to Akeeba Ticket System).
If you had tried to install Akeeba Backup 8.0.0 and/or Admin Tools 6.0.0 but lost access to your site's backend here's what to do.
First, delete the following folders from your site (some folders may NOT exist on your server):
plugins/actionlog/akeebabackupplugins/actionlog/admintoolsplugins/console/akeebabackupplugins/installer/akeebabackupplugins/installer/admintoolsplugins/quickicon/akeebabackupplugins/system/admintoolsplugins/system/akeebaactionlogplugins/system/akeebaupdatecheckplugins/system/aklazyplugins/system/atoolsjupdatecheckplugins/system/atoolsupdatecheckplugins/system/backuponupdateplugins/system/oneclickactionplugins/system/srp
You now have access to your site's backend. Follow the instructions in "Before the update" and then install the new versions of our software.
Much less common: third party extensions still using FOF 3
This applies only if your site complains that FOF 3 is not installed; or it says that a class whose name starts with FOF30\ is missing.
First follow the advice under “If you have lost access to the backend of your site” further above. If this did not help, you have a problem with a third party (non-Akeeba) extension improperly using an older version of our FOF framework. In this case, do the following:
- Download the latest version of FOF 3.x from our site's Downloads page. Please note that this is a THREE, not a FOUR.
- Extract the ZIP file.
- There is a folder named
fofextracted from the archive. Rename it tofof30. - Upload the
fof30folder into your site'slibrariesfolder so now you have alibraries/fof30folder.
Please note that there might be other folder with similar names in there such as fof, f0f, fof40. DO NOT REPLACE OR DELETE THESE OTHER FOLDERS. The folder fof is part of Joomla 3; if you remove it your site WILL break. The folder f0f is a very old version of our framework used by third party extensions; removing it will most likely break your site. The folder fof40 is FOF 4, used by versions of our software released after March 2nd, 2021; removing it WILL break your site. - Do NOT try to install the FOF 3.x ZIP file you downloaded in a previous step. This would cause the next time a FOF 4 extension is installed to remove the
libraries/fof30folder again and you'd need to repeat these instructions.
This is the end of the troubleshooting instructions.
You do not need to read the rest of the article. It's technical information on the nature of the issues and further insight into our pre-release testing process.
Technical explanation of the installation issues
There are several different issues addressed by the instructions above. We'll go through each one of them.
FOF 3 is uninstalled automatically after upgrading our software
Impact: medium (can't use third party extensions using FOF 3)
Frequency: extremely rare to unlikely
Responsibility: third party developers
Our software released between mid-2015 and February 2021 inclusive was using version 3 of our FOF backend framework. Software released on and after March 2nd, 2021 is using version 4 of the framework.
Since FOF might be used by multiple extensions, either our own or third party, we check if any extension has been marked as dependent on FOF 3. If none is marked as such we uninstall FOF 3.
This might be a problem in the following cases:
- You have used Discover & Install to install an extension depending on FOF 3.
- You are using a third party extension which does not mark itself or its sub-extensions (plugins, modules etc) as dependent on FOF 3.
In these cases the dependency is not marked in any way and there is no way to know that a FOF 3 dependent extension is installed on your site. As a result FOF 3 might be removed when it's actually needed. You will very likely see an error message saying that a class is missing whose name starts with FOF30\.
This is a problem caused by third party developers improperly using our backend framework. Still, we understand that this can be frustrating for you, the user. That's why we documented how to reinstall FOF 3 manually in this case. Furthermore, as of Akeeba Backup 8.0.2 and Admin Tools 6.0.2 we stopped removing FOF 3 automatically to avoid this issue. The corollary is that you need to uninstall it manually if you no longer have any extensions using it.
The installer plugins can get in the way of updates
Impact: high (can't apply updates)
Frequency: all Pro subscribers
Responsibility: ours (Akeeba Ltd)
Joomla has two ways to support the installation of paid extensions. The developer can either enter an additional query parameter in the #__update_sites table or they can provide a plugin in the installer group.
We had been using the additional query parameter method which is the canonical way to support paid extensions since Joomla 3.2 and one which is promoted as the recommended one as of Joomla 4.0. However, it has two ptoblems.
- If you use the Rebuild button in the Manage, Extensions, Update Sites page the additional query will be removed. You end up with a situation where you have entered the Download ID in our extension but Joomla does not see it.
- If you update the additional query (e.g. when you enter the Download ID in our extensions' Options page and click on Save & Close) but Joomla has already determined there are updates for this extension it will not use the updated query, i.e. it will not use the Download ID you just entered.
Both issues are the result of the horrid design choices for the Joomla extensions installer present since Joomla 1.6.0. We proposed a better way to handle updates in 2012, writing a detailed white paper, but the Joomla project decided to keep its semi-broken system. Further to that, the Joomla project decided to make it mandatory that developers use its semi-broken extensions update system in all of their extensions if they want to have any of their extensions listed in the Joomla Extensions Directory. Therefore we have to work with this horribly broken system despite us knowing why it's broken and having made a detailed proposal on how to fix it.
This is why we started shipping the Installer plugins. These plugins are called by Joomla right before an update package is downloaded. The only information provided is the download URL. The plugin does not receive any information about which extension is being updated. This creates a big problem for the implementation of these plugins for software developers like us who have more than one paid extension downloaded from the same server.
As a result, our plugins look if the download URL seems to reference our site (www.akeeba.com) and whether the Download ID is missing from the URL. If it's missing, it tries to load each of Admin Tools, Akeeba Backup and Akeeba Ticket System components and see if they have a Download ID entered in their Options. The first Download ID to be found is added to the URL.
Our software uses our backend framework FOF. The old versions of our software use FOF version 3, the new versions use FOF version 4. These two versions are incompatible with each other. The installer plugins cannot know which version you have installed due to limitations in Joomla itself (no, asking it for an extension's version is NOT guaranteed to return the correct version or any information at all since it reports the information it has cached in the #__extensions table, not the version information from the XML manifest file of the extension currently installed on the server and these two pieces of information might be completely different!). Therefore it's possible that the installer plugin will try to use the wrong version of the FOF framework to load one of our components when Joomla is trying to download an update to any of our extensions but the Download ID is missing. This will cause a PHP error due to the way PHP works, stopping the update. This is why we tell you to disable these plugins.
Given the fact of how Joomla and PHP work there is no viable workaround. Since these plugins run BEFORE the update file is downloaded we don't get the chance to automatically unpublish them for you. This can only happen in the pre-installation code of our extension which is part of the update package. Of course this would require downloading the update package first. Yet the problem occurs before downloading the update package. This a chicken and egg problem which can not be resolved automatically.
The installer plugins will be removed in a future version of our software. They seem to be causing more trouble than the problems they address. In the end of the day you should be following our documentation instructions to enter a Download ID and reset Joomla's update cache.
Old plugins can break your site after installation
Impact: critical (you may be locked out of your site)
Frequency: about 1 in 50 sites
Responsibility: users (we had already removed these plugins or their files from your sites in updates between early 2014 and February 2021!)
We saw a relatively large number of sites which still had installed some plugins we discontinued between one and eight years ago. These were plugins that used to be shipped with Akeeba Backup and Admin Tools.
When we removed these plugins from our software we updated our extension's pre-installation scripts to try and uninstall them or delete their files. However, this could fail, meaning they could remain installed. Moreover, if you restored an older backup the files would re-appear. In this latter case you could accidentally install them again using the Discover method in Extensions, Manage. Likewise, not remembering that they were no longer shipped with our software, you might have accidentally publish them.
These plugins only checked if the extension they were written for was installed and enabled. They did not check the version of the backend framework as there was no reason to do so back when they were created. Therefore updating Akeeba Backup and Admin Tools would make them fail. Since they are system plugins they are loaded in every page of the site. This means that could lose access to your site's frontend or backend.
As of Akeeba Backup 8.0.1 and Admin Tools 6.0.2 we are trying to forcibly delete their files if they are still present. In case this fails e.g. due to file ownership / permissions issues on your server, the update will go through BUT you will lose access to your site. Hence our instructions to delete all these plugins' files to regain access to your site and reinstall the extension update to make sure that only the current, up-to-date system plugins shipped with our software are installed on your site.
The Joomla extension update bug
Impact: high to critical (can't use our extensions to getting locked out of your site)
Frequency: about one in 30 sites
Responsibility: Joomla
Joomla has a really weird bug affecting it since Joomla 1.5.5. In some cases when it tries to update an extension it will fail to copy all of the files included in the update package. This results in the installed files being of mixed versions which might cause things to break. Traditionally, this required applying the update a second time in which case Joomla finally copied the correct files. That's a very frustrating bug and it's very hard for end users to understand it's not the developer's fault since it happens rarely, not on every single update.
We have been addressing this issue in our backend framework (FOF) since 2019 by doing a post-installation check. We go through every file in the package and check if a same name, same size file exists where Joomla should have installed it. If not, we copy it. This workaround is applied to FOF itself and all of our components, plugins and modules.
This workaround, however, was not applied to our frontend framework (FEF). Unsurprisingly, there were many sites where Joomla failed to update FEF correctly. As a result, trying to access a backend or frontend page using the new version of our components failed with an error complaining about a missing method in FEF's loader class.
Even though this is a Joomla bug we did address it by applying the same workaround to FEF. This is included in Akeeba Backup 8.0.1 and Admin Tools 6.0.2. If you are installing Akeeba Ticket System and you get an error just install the package a second time. Joomla will now copy the updated files correctly.
PHP opcache can cause weird issues
Impact: high to critical (can't use our extensions to getting locked out of your site)
Frequency: rare
Responsibility: ours (Akeeba Ltd), Joomla
PHP is an interpreted language. This means that on each page load PHP has to load the source code of each and every .php file involved in loading the page, parse it, compile it into an intermediate byte code (called "op-code") and then execute this byte code. The load, parse and compile steps take quite a lot of time.
PHP 5.5 and later comes with an extension called OPcache. This extension will cache the compiled op-code for each PHP file. This saves time by not having to go through the whole load, parse and compile process. However, this also means that PHP will be unaware when a .php file has been replaced when Joomla is updating itself or an extension.
The PHP OPcache extension tries to address that by checking the last modified time of each .php file before deciding whether to use the cached opcode or do the load, parse and compile steps afresh. Checking the last modified time of each file, however, does take some time. Therefore the PHP OPcache extension allows this check to be disabled or only take place every several seconds.
There is a problem arising from that behaviour. If PHP had encountered a file that is being replaced during an extension update and the same file is being loaded in the post-installation script of the extension being updated it is possible that PHP will use the cached version of the file. The cached version of the file, however, is the one that existed on your site before the update. This means that the post-installation code's expectation that it loads an up-to-date file is broken.
The way we were dealing with that is following the official PHP manual's suggestion to use opcache_reset(), a built-in function which resets the OPcache, therefore forcing PHP to read the newly installed file. However, the PHP manual is wrong. opcache_reset() does not have an immediate effect. It will only clear the OPcache after the page load is complete, i.e. after the post-installation code finishes execution. The fact that this has not been an issue in the past has been pure happenstance. Since Akeeba Backup 8 is based on a new backend framework the disparity between one of the files loaded through OPcache and what is on the server's storage caused a PHP fatal error during the finalisation of the update. This also stopped Joomla from completing the update which could even cause you to lose access to the backend of the site unless you deleted the folders of the system plugins shipped with Akeeba Backup. That's the other reason for the instruction to delete these plugin folders if you got locked out of your site's backend.
Further to that, on servers with a very long revalidation time (how long it takes before PHP checks if a file has been modified instead of serving it from the opcode cache) or servers which have disabled revalidation our software would appear broken after installation since PHP would be serving the cached version of the extension's files, not the ones which has just been installed!
We have addressed this problem by using opcache_invalidate() instead. We use the same code which works around the Joomla bug of not copying all files. As we go through all files in the package we invalidate the OPcache for each and every file which is installed on your site. We are further using some tricks to load critical files from the temporary directory Joomla creates to extract the update package instead of the permanently installed location of the file in our post-installation script, therefore guaranteeing that the OPcache will NOT come into play when loading the file. These workarounds have been applied as of Akeeba Backup 8.0.2 and Admin Tools 6.0.2.
Edit: We should have made it clear that this should be something Joomla does, not something it expects third party developers to do themselves. Joomla is handling the updates itself, copying all update files to their final destination, including .php files. It should anticipate OPcode caching and reset it for the files it copies. The Joomla project seems to disagree, obviously because they do not understand what OPcache does. The only places where Joomla resets the OPcache is the Joomla Update component which we contributed to Joomla back in Joomla 2.5.1. In other words, only our code in Joomla tries to address the OPcache issues. The rest of Joomla lives in blissfull ignorance. We refrained from making that comment originally but seeing how the Joomla project treats us with open hostility for reporting bugs in the extensions updater it forces us to use, on pain of being removed from the Joomla Extensions Directory, it is more than fair to proceed to full disclosure of Joomla's failure to provide a properly working extensions updater even though it forces all third party developers to use it. We accept our responsibilty for not having caught just how broken Joomla's code has been but we do not accept responsibility for the root cause of the OPcache issue. Joomla is 100% responsible for how its own extensions updater works and any claim to the contrary —especially when made by the Joomla core developers— is beyond unreasoanble!
About testing and update quality
Before publishing a new release we make several test installations on development, test and live sites. These are manual tests since we need to not only test whether the software is installed but also if it installed correctly AND the site still works as expected. There's a lot of subjective evaluation going into the latter which cannot be automated using testing tools.
For new major versions (x.0.0) we do a full series of tests that spans several days to weeks. We always install on development sites first, hosted locally on our development machines. These include different versions of PHP and Joomla and they test clean installations, upgrades from the immediately previous version and update from the oldest release appearing in the Compatibility page for the currently supported Joomla versions, even if that old version of our software is no longer supported. We also test updating extensions in at least three different update orders on the latest Joomla 3 and 4 version we support on the last PHP version we support. These are several dozen to hundreds of test installations. When an installation fails we debug it, fix the problem and restart the round of installations. A smaller subset of these tests takes place for minor releases (x.y.0) and an even smaller subset for point / patch releases (x.y.z). The reason for the smaller subsets is economy of time. If, however, we have changed something we believe could probably cause an installation issue we do run the full series of installation tests.
We then proceed with doing test installations on our test sites. These are sites hosted on different commercial hosting services and at least one self-managed server. These are mostly meant to root out any issues which only happen on commercial hosting because its configuration is always stricter than that of our development machines.
When that bar is cleared we test on real world, live sites.
Only then do we make a release.
Right after the release (within 5-10') we update our own business site. This way, if something breaks the site ours will be the first to break, making it impossible for you to download a bad updated. This has happened once, all the way back in 2010.
There is, however, an important point to be made. Despite this extensive testing we cannot possibly test every server environment out there or anticipate all site histories.
The issues discovered with the Akeeba Backup 8.0.0 and to a lesser extent Admin Tools 6.0.0 and Akeeba Ticket System 4.0.0 were not caught because of the following reasons:
- FOF 3 used in third party extensions. We do not use any third party extensions which use FOF on our sites. We use our own extensions. In fact, most of these extensions seem to be bespoke, i.e. made for a specific site, not published in the open market for anyone to download.
- We install the test updates using Upload and Install or Install from URL. There is no other way to do it. If we make an update available in the update site then it's a published release and anyone will see the update and install it. This meant that we did not see the Installer plugin issues during the two weeks we were extensively testing the installation of our software.
- We have set up our Download IDs correctly. We do not have any live sites where the Download IDs are not set up correctly. Therefore the test installation using Joomla's Update page after we published the new release did not hit on the Installer plugin issue.
- Our development sites do not have leftover old plugins due to their nature. Our test and live sites did have some of them but their files had been deleted as part of a previous update. We had never installed an old backup and then a new backup on top of it which would have effectively replaced the obsolete plugin's files. We never used Discover to install plugins we knew they were obsolete, nor had we ever seen any there because of the previous point about not installing new over old backups. Therefore we never saw the problem with leftover obsolete plugins, nor did we consider that this could have been an issue knowing that their files were deleted or the plugins were completely uninstalled in a previous update.
- We didn't hit the Joomla bug about not copying all the files. This issue is quite rare. Based on our observations it happens in around 3% of the updates. That's why we never hit the issue with FEF's files not being copied by Joomla.
- Our servers use rather conservative OPcache settings, revalidating files and having a revalidation time of 2 to 5 seconds. Between the time it takes to click through Joomla's menu items and the time it takes for the update package to upload we never with the OPcache issue during Akeeba Backup's update. In fact it took just over 6 hours to figure out how to reproduce it on a local server. It was only possible after setting an unreasonably low OPcache revalidation frequency (60 seconds instead of 5) and doing the update using automated tools to remove human speed limits and variance from the equation — albeit I'll admit the first successful reproduction took a deep breath and a VERY steady hand.
In retrospect, the only issues we could have reasonably tested against are the missing Download IDs and OPcache settings. We are removing the installer plugins, making the former point moot.
We have added the less than ideal opcache settings to our testing plan.
We think the leftover plugins are not a reasonable thing to test against — this would make it impossible to test upgrading from anything but much older versions of each extension — but a reasonable thing to work around in our pre-installation code. The workaround in the pre-installation code has already been added.
Trying to test for the Joomla update bug is something we would definitely like to do but this requires finding out under which exact circumstances it's triggered. If you ever find out please let us know. It's a bug we've been trying to reproduce reliably since 2007 so we can contribute a fix for it in Joomla itself.
Testing against third party extensions using the no longer in development FOF 3 framework against our documentation is also out of the question. In the end of the day it's a third party doing something against our documentation. All we can do is just leave FOF 3 behind on update.
Why was there no beta version?
Arguably, had there been a beta version we MIGHT have seen some of these problems and fix them before the stable release. Or so common wisdom has it.
In the past we have, indeed, published beta releases. We even had long beta testing periods of 1-3 months. That made no difference whatsoever. Very few people would install the beta releases. Even fewer would do so on real world sites or copies thereof. Installing on a pristine site wouldn't uncover any issues we hadn't already caught with our test installations before the beta release, as described above. From the very few people who would install on anything resembling a real world sites we got no bug reports.
In fact, the same people would only contact us right after the stable release was made to tell us "gee, I had seen XYZ problem in the beta 3 months ago, I thought you'd have fixed it by now". A problem never reported is a problem never addressed. If we magically knew of all possible problems there would be no beta releases and no bug fixes in stable releases. You'd get 1-2 perfect releases every year and that would be all. This lack of communication of issues during the beta period invalidates the reason why betas exist. As a result we are far less likely to publish a beta.
This time we did want to publish a beta because we knew that we are delivering massive changes. However, we were not allowed to.
We had already made a change in the then-unreleased FOF 4 code which replaced our old implementation of a random number generator with PHP's built-in random_bytes. Three weeks before we were going to publish a beta version, the Joomla Security Strike Team contacted us to tell us that they are making the same change in Joomla itself, in the obsolete version of FOF 2 included with Joomla, which they mistakenly considered a security fix. We explained why this is not the case but they wouldn't listen, as you can read in a different news article on our site. They asked us to do a coordinated release on March 2nd, 2021 which meant that we could not release any beta version before that date.
Moreover, since they falsely considered this simple change a security issue it put us in an impossible position. If we chose to release a beta instead of a stable they might just as well have marked the stable versions of our software as "vulnerable" even though it's not really the case. The only version they wouldn't be able to do that with would be the betas we'd release on March 2nd. It's impossible to prove you're not an elephant. Therefore we chose to instead work 12 hour days, 7 days a week between January 27th and March 2nd to deliver stable releases on March 2nd.
So, even though the original planning was to in fact do betas this time we had to not do that because the Joomla Security Strike Team would be smearing our reputation. The resulting stable releases ended up hitting some issues we could have never expected but we pulled an all nighter to address them. So, now, two days later you have a stable release without any problems except the one we can't automatically address: unpublishing the installer plugins before the update.
Edits
Updated March 9th to add the impact, frequency and responsibility of each issue so there are no misunderstandings.