Joomla! 3.7.0 has a bug which prevents CLI scripts from executing correctly. We have identified the root cause of this issue and we are providing a workaround - even though Joomla's production team has acknowledged the issue and will fix it eventually. We recommend all users of the Professional versions of Akeeba Backup for Joomla, Admin Tools and Akeeba Ticket System to upgrade immediately.
What happened
Joomla! 3.7.0 introduced a bug in its session management API. This API is used internally by core Joomla! APIs (e.g. to get user settings) which means that using it is inescapable. Due to the nature of the bug and some code already present in the Joomla! CMS it doesn't affect its public frontend and administrator backend. It does affect, however, every third party script which needs to load the core Joomla! APIs, such as the command line CRON scripts used by Akeeba Backup.
Technical analysis (you can skip over this paragraph if you're not a coder): Joomla! 3.7.0 has a bug in JFactory::getSession(). Instead of returning a fully initialized session object it returns what is essentially an uninitialized object. Trying to use it leads to a PHP Fatal Error. You don't observe that in the CMS itself since JApplicationCms initializes the session manually. You do observe it in other scripts, e.g. anything based on or replicating the functionality of JApplicationWeb or JApplicationCli, where manual initialization of the session is not performed.
This is a bug that's already been acknowledged by the Joomla! leadership. It will eventually be fixed. However, we consider that any day that goes by without our power users being able to automatically backup or scan their sites is a day too many to wait. For this reason we have released new versions of all of the affected Professional versions which we recommend that you install as soon as possible.
Why didn't Joomla! catch this bug before release?
Bugs happen. Since Joomla! is such a big part of many of our lives, it can be easy to forget that it is a volunteer-led and run organization. This means that it can be more prone to process errors than a large company.
This, however, is not an excuse for bugs. Joomla! has a team which is dedicated to overseeing the production and release of code. As part of their process, each code contribution is supposed to require two humans individually testing and accepting the code. In this sense, there are two or more people signing "QA OK" on the code to be included in Joomla!. A third person commits this code to the repository, essentially signing it off.
However, it appears that this bug slipped through the cracks of both the internal and volunteer testers. To his credit, the person who wrote the affected code accepts full responsibility for the bug and is planning to fix it.
Why didn't we catch this bug before release?
We test our software thoroughly with the beta and RC releases, and we even develop on it. In fact we are using Joomla!'s "staging" branch (the code family where beta, RC and stable releases of the upcoming Joomla! version come from) as our secondary development environment, the currently released stable version being always the primary one. That's how we integrated with the new custom administrator menu feature as soon as it was included in the staging branch, being the first third party extension to include support for that feature long before Joomla! 3.7.0 was officially released.
That said, testing and developing has its limits. For starters, we provide support to you with the same few people developing the software. It's the only way to provide meaningful support. Our time has to be shared between support and development. We have no control over the time required for support. Sometimes, like the last two months, it's very busy. Some other times, like mid-July to early September, support is quiet and we can focus more on code. The remaining time has to be fairly shared between developing our software and testing someone else's software (Joomla!).
For this reason we have to prioritize our testing. Some software is easy to test automatically. For example Admin Tools has a predictable outcome for the same input, therefore has automated integration testing. Caveat: it runs on the Joomla! web API, not Joomla! CLI, due to PHP constraints which would otherwise make the testing only work on a limited spectrum of Joomla! releases. Akeeba Backup, however, is a very special case. Not only does most of its code deal with working around server-specific issues and integrating with third party services, but the very nature of time-controlled execution, encryption and compression makes automated tests of very limited utility as we learned after spending an incredible amount of resources towards that goal. Therefore most of its testing is manual.
The priority for us is that you can run a backend backup and a remote (JSON API) backup with the backups stored locally, on S3, Dropbox, Google Drive or OneDrive and restore it from within the component and through Kickstart - that's what the majority of the clients use. Most of the issues we deal with when testing with a new Joomla! version are client-side, i.e. JavaScript and CSS. We typically do not run into core API issues - and we shouldn't, since Joomla! supposedly follows Semantic Versioning. Other features will only be tested if code in their code path has been touched during the last development cycle. That is to say, we won't test CLI backups unless we've modified the CLI code or a part of the backup engine.
And here lies the issue. Joomla! broke the CLI scripts without us having made any change which would justify testing it, also breaking backwards compatibility. At this point we knew that backups were running successfully from the backend of the site and the JSON API. We also believe Joomla's promise of Semantic Versioning which precludes this kind of major changes being included in newer minor versions bumps (3.6 to 3.7). At the same time we were trying to test the entire bunch of extensions we are offering with Joomla! 3.7, given just two to five days per Release Candidate to test everything. As you understand prioritization meant that features we had no reasonable expectation to believe might be broken could not be tested. Finally, during the last two RCs we had to also fly to a Joomla! Day event, limiting the time we could put even further. So, no, we didn't test that feature because we had no reasonable expectation to believe it was broken and because the amount of time given by Joomla! to 3rd Party Developers to test the new release was unrealistically small.
Why did it take so long to come up with a fixed version?
The issue was discovered on Wednesday, April 26th around 8 p.m. in our timezone, shortly after Joomla! was released. We worked until midnight to create a fix. We chose not to release it at that time. We had already seen one major backwards compatibility break in Joomla! 3.7 which affected us and several minor issues which didn't. According to our experience this is a strong indication that more grave issues are to follow. So we chose to sleep on it.
In the morning of Thursday, April 27th we checked our support tickets and found one that was concerning. Someone had spotted an issue about a rather unpopular feature of Admin Tools, the away hours. They seemed to be one hour off. Upon further investigation we discovered a blatant bug in Joomla!'s JDate API. Unfortunately, this API is used everywhere we need to perform date manipulation or format a date and time for display. That three-line-long bug in Joomla! has ripple effects which spread deep into all aspects of every single of the 9 Joomla! extensions we publish.
The only way to deal with that was to fork the JDate API, fix it and put the fixed version in our framework, FOF 3. Then we had to comb through all nine extensions, replace the uses of JDate throughout - just south of 1000 instances - and test every single feature which was touched as a result of that change, making sure nothing is broken. That was an epic 13 hour death march which concluded at 11 pm on the night of Thursday, April 27th. New releases of FOF, Akeeba Backup for Joomla!, Admin Tools and Akeeba Ticket System were released. The rest of our extensions will have to wait for Friday when we've scheduled yet another death march. This is all worth it because that's how we can deliver good quality software.
What can we do?
We will continue to develop and test our software on the latest stable and staging version of Joomla, as always. We're still catching many more Joomla! bugs and backwards incompatible changes than we care to share in public. Our desks have head-shaped indentations as a result.
As always, we advise you to have a copy of your site(s) - you can easily make one with Akeeba Backup - and test the update there before updating your live site. It might be a hassle, but it is the only way to 100% guarantee everything is working as it should before you upgrade.
Thank you for your patience and support.
-Akeeba Backup