Support

Akeeba Backup for Joomla!

#24461 Remote Upload Quotas Not Being Honoured...

Posted in ‘Akeeba Backup for Joomla! 4 & 5’
This is a public ticket

Everybody will be able to see its contents. Do not include usernames, passwords or any other sensitive information.

Environment Information

Joomla! version
n/a
PHP version
n/a
Akeeba Backup version
n/a

Latest post by on Monday, 28 March 2016 17:20 CDT

TheEdge2
Hi,

I have a weird one here. The end result I want is for the backup zip to be uploaded to the remote FTP server and the source file deleted on the host. Furthermore I want at most 31 files kept on the remote FTP server.

What I am seeing is that the backup is succeeding OK, it is not being uploaded to the FTP site. It appears to be a timeout issue, but I have played with the settings and cannot get this to work. When I do a manual FTP from the host all is well, so I am guessing it is the fact that the PHP request is timing out before the FTP succeeds?

I attach hereto my Profile, Log and Angie analysis. I am hoping to get back a backup profile that I can load and that will work. Happy to provide any other debug information that is needed.

tampe125
Akeeba Staff
Hello Donovan,

you didn't set any part size for your archive.
This means that Akeeba Backup tries to upload the whole backup (more than 35Mb) in one shot: if such operation takes too long you get a timeout error.
Please try to setup a partsize of 10Mb and try again, that should do the trick.

Davide Tampellini

Developer and Support Staff

🇮🇹Italian: native 🇬🇧English: good • 🕐 My time zone is Europe / Rome (UTC +1)
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

TheEdge2
Hi,

Thanks for that. I now have the backups flowing to the FTP server. However there are no errors in my logs yet the files that are on the FTP server are corrupt.

I have archive integrity turned on and am using ZIP as the file format (I don't want to use JPA as I cannot verify the archives without using Akeeba).

So a couple of questions:

1) Is it possible to have different quotas running - one for local files and one for remote files. That way I can at least have 2 copies of the archive one local and one remote but store more remotely and say only keep 1 or 2 local?
2) How do I diagnose this issue of corrupt remote FTP files when ANGIE reports no errors whatsoever?

tampe125
Akeeba Staff
No, you can't have different quotas.

Regarding the corruption problem, let's do some debugging.
  1. Remove any post-processing engine and try to download and extract the copy saved on your site, is that valid or it's corrupted?
  2. Try another post processing engine, for example Dropbox, is the backup still corrupted?
  3. Try to use JPA format and Dropbox, is the backup still corrupted?
  4. Try to use JPA and FTP post processing engine, is the backup still corrupted?


Let me know the results of your tests, depending on your answers we can narrow down where the problem occurs

Davide Tampellini

Developer and Support Staff

🇮🇹Italian: native 🇬🇧English: good • 🕐 My time zone is Europe / Rome (UTC +1)
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

TheEdge2
Hi Davide,

Did some testing... Basically if the archive is split into parts then it becomes corrupted. The file is corrupt on the server and hence on any remote location. This behaviour happens with the latest version of Akeeba and there are no warnings generated at all. In addition archive integrity checking is turned on - so this is concerning.

As I cannot get FTP to work (with a single archive) what I do have working is sending a ~50MB zip to DropBox and it gets there with no issues whatsover. As I ultimately want it on GoogleDrive (I have an FTP adapter to GoogleDrive and that is why I was using FTP) I will use IFTTT or something else to get it copied across. (Pity there is no GoogleDrive support - I know I know - there is good reason why not)

On a side note... When I got warnings because of issues with the backup there did not seem to be anywhere in Akeeba that I could set up a notification email address so that I got notified when there was a problem. Is Akeeba able to notify me whenever a backup is run and there is any kind of warning or error? The backups are kicked off remotely automatically periodically.

--Donovan

tampe125
Akeeba Staff
Ok, so the problem is in the files included in the backup, not the transfer.
This is caused by a file that shrinks during the backup process. 99% of times they are log files: when the file grows too large, it gets truncated and then new data is appended.
This process is breaking the whole archive, since the recorded size while the backup starts and the real size are different.

You can do this to narrow down the issue: download a backup archive and try to extract it. First try to check the option Ignore most errors inside Kickstart, if you're lucky that could do the trick.
Anyway, we really want to exclude such nasty file from your backup, so do the following steps:
  1. Download your backup archive
  2. Edit the kickstart.php file and uncomment the line that contains define('KSDEBUG', 1);
  3. Run the extraction
  4. Kickstart will produce a massive debug file, logging all the actions it's doing.
  5. Scroll to the end of the log file, the second to last file is the corrupted file (or the last one, I can't recall ATM). Note it down
  6. Go inside your Akeeba Backup installation and exclude such file from the backup

That should do the trick, please let me know if your backups are still corrupted.

Davide Tampellini

Developer and Support Staff

🇮🇹Italian: native 🇬🇧English: good • 🕐 My time zone is Europe / Rome (UTC +1)
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

TheEdge2
Hi,

I am testing the archive itself with both 7-zip and Pea Zip (after downloading to my machine) so not with the kickstart. I will use kickstart with the steps and see what happens and see if I can narrow down the issue.

Am I missing something obvious here? You are aware that the zips can get corrupted with files changing size during the backup. Surely this is a MAJOR problem? Makes me feel less confident about the whole backup process ;-) Is this only when there are split archives or even when it is a single archive file?

And why when archive integrity testing is enabled is Akeeba not reporting the archives as corrupt?

nicholas
Akeeba Staff
Manager
DO NOT USE THIRD PARTY UNARCHIVERS TO EXTRACT THE ZIP FILES. We have already documented this! There are two reasons why you should NOT do that:

1. Standards-compliant multipart ZIP archives ARE NOT supported by most third party unarchivers. Akeeba Backup always generates this kind of backup archives since it's impossible to tell at the start of the backup whether we'll need to create a single or multipart archive. Only PKZIP for Windows and WinZIP –both developed by the inventor of the ZIP format– support this kind of archives. All other third party unarchivers are based on the third party InfoZIP library which erroneously doesn't support this kind of files, reporting them as corrupt.

2. PHP cannot calculate the CRC for larger files (where "larger" for PHP is generally anything over 1Mb long) because that would cause a memory outage error. As a result files of that size will have a CRC of 000000 which is, of course, invalid and makes third party unarchivers to mistakenly report the archive as corrupt.

This is why we tell you to always use Akeeba Kickstart or Akeeba eXtract Wizard to extract the backup archives. Alternatively, you can use PKZIP for Windows or WinZIP always bearing in mind that any reported "corrupt files" are, in fact, not corrupt at all.

Regarding the possibility of having a corrupt backup archive it is extremely rare and will only happen if the file being backed up disappears or shrinks dramatically in size while the backup is in progress. This is NOT something that happens regularly and we DO have workarounds in place to prevent the archive from being corrupt. These workarounds depend on the ability of your server's filesystem to notify PHP about the size change in a way which allows us to enforce said workarounds.

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

TheEdge2
Hi,

I want to automate testing of the multipart archives on a Windows machine. So can you point me at some documentation on what Akeeba PHP code I need to use to do this? I assume I would have to install PHP cli to run either kickstart or the wizard to test the archives. I don't want to install the sites all I want to do is to verify that the backups are good.

nicholas
Akeeba Staff
Manager
If you want to make sure that the multipart archives can be extracted you can use the archive testing feature in Akeeba Backup itself. Go to the Configuration page, check the "Archive integrity check" check box.

If you want to do test restores (recommended!) you can use Akeeba UNiTE. Access to it is included in your subscription. This is a command line PHP solution which allows you to automatically extract and restore a site backup archive. Incidentally this is what we're using at the office server to test our backups every day :)

If you want a command line extraction-only solution you can still use UNiTE with a custom execution chain that only goes through extraction. It is feasible and documented but I don't see why bother doing that when it's equivalent to the archive integrity check in Akeeba Backup, the latter being far easier to use as well.

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

System Task
system
This ticket has been automatically closed. All tickets which have been inactive for a long time are automatically closed. If you believe that this ticket was closed in error, please contact us.

Support Information

Working hours: We are open Monday to Friday, 9am to 7pm Cyprus timezone (EET / EEST). Support is provided by the same developers writing the software, all of which live in Europe. You can still file tickets outside of our working hours, but we cannot respond to them until we're back at the office.

Support policy: We would like to kindly inform you that when using our support you have already agreed to the Support Policy which is part of our Terms of Service. Thank you for your understanding and for helping us help you!