Support

Site Restoration

#14571 Kickstart Pro 3.6 stops importing from S3

Posted in ‘Site restoration’
This is a public ticket

Everybody will be able to see its contents. Do not include usernames, passwords or any other sensitive information.

Environment Information

PHP version
n/a
CMS Type
Other
CMS Version
n/a
Backup Tool Version
n/a
Kickstart version
n/a

Latest post by nicholas on Wednesday, 09 January 2013 02:23 CST

dkdb

Mandatory information about my setup:

Have I read the related troubleshooter articles above before posting (which pages?)? Yes
Have I searched the tickets before posting? Yes
Have I read the documentation before posting (which pages?)? Yes
Joomla! version: 2.5.8
PHP version: 5.3.18
MySQL version: 5.0.96
Host: Blanye
Akeeba Backup version which took the backup: 3.6.10 (I think)
Kickstart version used to extract the backup: 3.6.0

Description of my issue:

Using Firefox 10, tried with Google Chrome, same problem.

One of my profiles sends the backup to S3. Until 26/11 I split it up in 50 MB files, and since then 100 MB files.

When I start kickstart, and asks it to import from S3, it shows the archives nicely.

But no matter if I try to restore the backup in 50 MB parts or the 100 MB parts, it stops.

The backup is around 1.6 GB.

The backup in 50 mb parts gets to around 50% the the backup with 100 mb files stops around 30%.

It never goes beyond the 'import', the ABI is not launched.

If I look at the folder it imports the archive to, I can see a .jps file around 800 GB, which is sort of 50% of the complete backup, and a .jps.tmp at 0 bytes.

If I try and download the files manually from S3, and then upload them manually, I see an error 'Invalid header in archive file, part 0, offset 8'

If I copy the files from my local backup (local to the website), the backup is restored.

Best regards Kenneth

nicholas
Akeeba Staff
Manager

Regarding the import hanging, this is caused either by a network error (but it would have thrown an error message in this case!) or because your host kills the script for some reason (e.g. too much CPU usage).

If you can see a single message of 800Gb when importing a 1.6Gb archive something is not right. Even if you meant 800Mb that would still not be right since you told me that you split the backup archive in 50Mb or 100Mb chunks. Try removing the .jps file and retry the import.

Regarding the problem extracting the files you downloaded and uploaded, I am pretty sure the problem lies in the uploading part. Do the following. Download all backup archive parts from S3 to your local computer. Try extracting them with Akeeba eXtract Wizard. If the extraction does work then your files stored on S3 are just fine and you simply have to make sure that you upload all archive parts by FTP using the Binary transfer mode.

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

dkdb

I tried downloading the files (17 of them), and started the extract wizard, that also fails with 'invalid archive'.

I also tried to 'reimport' the jps file after removing the old one, same result.

 

Best regards Kenneth

nicholas
Akeeba Staff
Manager

There seems to be a problem uploading your backup archives to S3. Can you please ZIP and attach the backup log file?

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

dkdb

As the log is only available for the latest backup, I've downloaded that and made sure it also shows the same problems, it does.

The backup is quite a bit smaller as I've now removed the j4age joomla stats package from the system, and added a seperate piwik backup.

The piwik database is now included in the backup, so there are two databases here, and an extra filespace area.

 

Best regards Kenneth

nicholas
Akeeba Staff
Manager

I have a suspicion. Can you please try clearing the "Process each part immediately" checkbox in the configuration and retry?

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

dkdb

That gave me this warning (this was for test purpose run from backend):

Warnings AEUtilAmazons3::uploadMultipart(): [RequestTimeout] Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed. Failed to process file /home/dkdbdk/public_html/administrator/components/com_akeeba/backup/site-www.dkdb.dk-20130108-143930.j01 Post-processing interrupted -- no more files will be transferred

Best regards Kenneth

nicholas
Akeeba Staff
Manager

My bad. I should have told you to also check the "Disable multipart uploads" checkbox and make sure that if you are going to run this from the back-end of the site you need to make sure that you have a relatively small part size (usually, 10 - 20Mb work best).

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

dkdb

Ok, I set the disable multipart, and set the cron job to run the test as well, that seemed to run better.

But after downloading all the parts (set to 50 mb) it still complains about invalid archive...

 

Best regards Kenneth

dkdb

Ups, forgot the logfile

Best regards Kenneth

nicholas
Akeeba Staff
Manager

I don't see any errors in the log which could explain this and I can certainly not replicate an issue with transfer to S3. On the contrary. I automatically download all backups every night and test extract them on a local server machine.

Let's try something else. Disable the automatic deletion of uploaded backup archives. Take a backup and try downloading the files from your server, not S3. Can you extract them?

If you can't extract them, I'd suspect a problem with the password or the fact that there's a small chance that eXtract can't correctly extract JPS files. Using Kickstart on a local server (e.g. XAMPP) would help test if the problem is, after all, eXtract or even entering the wrong password.

If you can extract them, something's off when downloading the files from S3. Try using an S3 application like S3Fox, CloudBerry Explorer for Amazon S3 or Cyberduck depending on your operating system. Does that allow you to extract the archive you downloaded from S3?

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

dkdb

I just tried again this morning, there must've been something off somewhere, either the downloading failed or transfer problem, because this morning it would start the extraction (dry run), I wonder what is going on here.

This morning I put s3fox on ff, the files have previously been downloaded 'by hand', and I worry that that could somehow have messed it up.

I also tried redownloading the old archives (also running dry run), and that also seems to work, so I guess the backups are ok, even the old ones, before the change to 'process immediately', and 'disable multipart' setting.

Or could it be that I've repeatedly entered the wrong password?

So all in all the backups seems to be valid, sorry for the wild goose chase there.

Now, the original kickstart problem...

I just tried reusing that, and that still creates one very large jps file for the old archives, and stops around 30%

Best regards Kenneth

nicholas
Akeeba Staff
Manager

OK, this is what I suspected: you had a problem downloading the backup, not a corrupt backup. I'm glad we have solved that :)

Regarding the Kickstart import from S3 issue, as I said it largely depends on your server. If the server decides to kill the script because of excessive CPU time usage you can't do much about it. In this case the manual download / upload is the only way.

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

Support Information

Working hours: We are open Monday to Friday, 9am to 7pm Cyprus timezone (EET / EEST). Support is provided by the same developers writing the software, all of which live in Europe. You can still file tickets outside of our working hours, but we cannot respond to them until we're back at the office.

Support policy: We would like to kindly inform you that when using our support you have already agreed to the Support Policy which is part of our Terms of Service. Thank you for your understanding and for helping us help you!