Support

Hello,

technically the backup did not fail, since you do have a copy of your website, this is why you can't analyze it with ALICE.

Can you please attach a log of a backup that was not transferred to Dropbox? The issue happens all the time, or it's random?

Are you using OVH as hosting?

We have observed that since around February 20th, 2023 it's not possible to upload backup archives to Dropbox (and possibly other remote storage providers) on sites hosted on OVH. The problem, as you will see, is beyond a mere inconvenience; it's a reliability issue with OVH's hosting services which can lead to completely broken sites.

The log file shows that a lot of chunks of the backup archive are uploaded without a problem, each chunk taking just 1–3 seconds. However, the log file ends trying to upload another chunk with plenty of time left. You see no response from the server and the log file ends abruptly. Here's an excerpt from a real world sample log file, with the site name and backup archive name replaced for privacy reasons:

DEBUG   |20230223 11:24:38|Akeeba\Engine\Core\Domain\Finalization::_run() Resuming finalization object Akeeba\Engine\Core\Domain\Finalizer\PostProcessing
DEBUG   |20230223 11:24:38|Loading post-processing engine object (dropbox2)
INFO    |20230223 11:24:38|Continuing post processing file <root>administrator/components/com_akeebabackup/backup/site-example.com-20230220-121314cet-abcDefghijKl12M3.jpa
DEBUG   |20230223 11:24:38|Akeeba\Engine\Postproc\Dropbox2::processPartChunked - Using chunked upload, part size 20971520
DEBUG   |20230223 11:24:38|Akeeba\Engine\Postproc\Dropbox2::processPartChunked - Uploading chunked part
INFO    |20230223 11:24:40|More post-processing steps required for file <root>administrator/components/com_akeebabackup/backup/site-example.com-20230220-121314cet-abcDefghijKl12M3.jpa
DEBUG   |20230223 11:24:40|Not removing the non-processed file <root>administrator/components/com_akeebabackup/backup/site-example.com-20230220-121314cet-abcDefghijKl12M3.jpa
DEBUG   |20230223 11:24:40|Akeeba\Engine\Core\Domain\Finalization::_run() Resuming finalization object Akeeba\Engine\Core\Domain\Finalizer\PostProcessing
DEBUG   |20230223 11:24:40|Loading post-processing engine object (dropbox2)
INFO    |20230223 11:24:40|Continuing post processing file <root>administrator/components/com_akeebabackup/backup/site-example.com-20230220-121314cet-abcDefghijKl12M3.jpa
DEBUG   |20230223 11:24:40|Akeeba\Engine\Postproc\Dropbox2::processPartChunked - Using chunked upload, part size 20971520
DEBUG   |20230223 11:24:40|Akeeba\Engine\Postproc\Dropbox2::processPartChunked - Uploading chunked part
INFO    |20230223 11:24:43|More post-processing steps required for file <root>administrator/components/com_akeebabackup/backup/site-example.com-20230220-121314cet-abcDefghijKl12M3.jpa
DEBUG   |20230223 11:24:43|Not removing the non-processed file <root>administrator/components/com_akeebabackup/backup/site-example.com-20230220-121314cet-abcDefghijKl12M3.jpa
DEBUG   |20230223 11:24:43|Akeeba\Engine\Core\Domain\Finalization::_run() Resuming finalization object Akeeba\Engine\Core\Domain\Finalizer\PostProcessing
DEBUG   |20230223 11:24:43|Loading post-processing engine object (dropbox2)
INFO    |20230223 11:24:43|Continuing post processing file <root>administrator/components/com_akeebabackup/backup/site-example.com-20230220-121314cet-abcDefghijKl12M3.jpa
DEBUG   |20230223 11:24:43|Akeeba\Engine\Postproc\Dropbox2::processPartChunked - Using chunked upload, part size 20971520
DEBUG   |20230223 11:24:43|Akeeba\Engine\Postproc\Dropbox2::processPartChunked - Uploading chunked part

(end of log file)

The first two pair of line in bold show that archive chunks are being uploaded just fine, within 2–3 seconds each. The last bold line, where the log file ends, shows that the OVH web server killed the backup process without returning a response (neither success, nor failure), and without throwing an error.

We have debugged real world sites where this is happening and we observed that, indeed, the upload works just fine up to the point that the OVH server simply does not return a reply at all. The upload simply times out after the hard limit of 300 seconds we have set up in our software to catch broken servers like that — the kind of which we had not seen for well over 10 years, mind you.

We have tried using different timing settings to no avail. The problem happens randomly and apparently depends solely on the server's load which we cannot determine in any reasonable way. It may happen 5 seconds into a backup upload step, or even within the first second of a backup upload step. This makes it impossible to catch it or work around it in any way. Simply put, the OVH web servers randomly act as a black hole for requests.

On Thursday, February 23rd, 2023 at 12:23 EET a client who had this problem and contacted OVH relayed to us the response from OVH about this issue. They told him this happens, and I quote, “because of the number of sites hosted on the server”. In other words, OVH tacitly admitted that they are overselling their servers, therefore they resort to randomly killing processes and breaking your sites to prevent the web server from failing altogether. The only two solutions they provided were 1. our client manually downloading the backup archives to their computer, and manually transfer them to Dropbox; or 2. upsell our client to a more expensive hosting package where, presumably, they wouldn't intentionally break his site. Both of these so-called "solutions" are completely unacceptable, considering that the problem is caused by OVH itself per their own admission.

At this point we would like to note that this is not just a problem with uploading backup archives. This is a problem which affects your site as a whole. OVH admitting that they are randomly killing processes on their server because it's oversold (it tries to host more sites than it has capacity for) means that your site will randomly stop responding even you are browsing the site, or working on your site's administrator. The effects of this problem can range from inconvenient (your site randomly stops responding when a client is visiting it) to downright catastrophic (the site may stop responding while an update to your CMS or an extension is installed, thereby completely breaking your site to the point it requires restoration from the last known good backup). We consider this kind of hosting service to be beyond unreliable and downright unacceptable.

We very strongly recommend voicing your concerns to OVH which intentionally breaks your sites and demand a solution from them. After all, hosting your sites reliably is what you are paying for. If they cannot provide this service and refuse to address the problems they have intentionally introduced to your sites we advise you to use a different hosting provider.

So far we have only heard this happening with Dropbox which makes me thing that MAYBE the reasoning they gave our client was hogwash and the real reason is that they have screwed up their proxy server.

If you can try using a different remote storage provider, for example Amazon S3, on a couple sites and tell if it works it would help us understand what OVH broke.

Hm, if the problem is only with Dropbox then what OVH told our client was, not to put too fine a point on it, horse crap. I bet the actual problem is a misconfiguration on their proxy server which handles HTTP(S) requests from the server to the outside world.

Can you please tell them to start fixing this? Since you have a ton of sites with them they are far more likely to listen to you.