Support

Site Restoration

#35146 Feature request: Add support to Amazon S3 custom endpoints

Posted in ‘Site restoration’
This is a public ticket

Everybody will be able to see its contents. Do not include usernames, passwords or any other sensitive information.

Environment Information

PHP version
n/a
CMS Type
Other
CMS Version
n/a
Backup Tool Version
n/a
Kickstart version
n/a

Latest post by nicholas on Thursday, 29 April 2021 01:08 CDT

carcam

I have started to use Wasabi as my storage provider and akeeba solo configuration is a breeze with the custom endpoint setting so I wonder if it could be possible to add the custom endpoint feature in the Amazon S3 restoration process.

If not possible, ftp import would also be nice...

Right now if I want to use automatic import with Kickstart I have to enable public access to the backup files or use another method to bring them back to my server...

Thanks!!

nicholas
Akeeba Staff
Manager

It's unlikely that the S3 support in Kickstart will be updated. The way Kickstart is generated we can't use namespaced code. This means that the S3 support is stuck to a much older version of our S3 library since before it was namespaces. Bringing it up to speed would require us to essentially rewrite that library just for Kickstart.

Likewise, adding support for custom endpoints is very complicated because Kickstart does not have a session context or configuration file. Handling all the possible permutations of configuration options required for custom endpoints is very complicated in this context.

If anything, it'd make more sense to eventually remove S3 support from Kickstart. This feature dates back to when S3 was relatively straightforward and worked the same regardless the region your bucket was stored in. This is no longer the case.

Regarding FTP import, it's even MORE complicated. FTP and SFTP don't allow staggered (chunked) downloads. You'd have to take a backup with a really small part size (under 10MB) or it would fail to import anything by FTP/SFTP. On top of that there's the problem with regards to having four different (S)FTP connection methods and a dozen or so options on each one of them. This is barely manageable for extraction. In the context of importing files it's extremely complicated to manage.

Moreover, there's a problem with regards to the maximum file size we can have in Kickstart. Once you go over 384Kb things start getting iffy. Some hosts won't allow you to upload a PHP file that big, others will refuse to run it, some might even fail parsing it. We are already at that limit. Adding more features to Kickstart would only make it worse.

All of those features would only make sense if Kickstart was software you install on your site. But if that was the case then why not install, say, Joomla and use Akeeba Backup? Once you get to that point Kickstart has lost its reason of existence.

The feature that makes most sense is the import from URL. No matter which remote storage provide your are using you can create a "share" (direct download) link for your files. Create such a link either for individual part and import them using the Import from URL.

The other thing you can do is of course use Akeeba Backup's and Akeeba Solo's Site Transfer Wizard (STW). You can use FTP or SFTP to upload your backup archive part(s) as small chunks and have a headless Kickstart on the remote server "stitch" them together. This is what the STW does. Since you do not involve your local computer at all it's all much faster than downloading everything locally and uploading stuff to the remote server. Server-to-server transfer is typically one or two orders of magnitude faster.

Finally, you can always use UNiTE on the target server, alone or in combination with other CLI tools such as s3cmd, to import and restore backup archives.

So, there's really minimal chance of adding import features to Kickstart. These made sense back in 2010 but they have been less and less relevant over the years, plus we're hitting a file size limit beyond which Kickstart may not even work at all.

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

carcam

Hi Nicholas,

thank you very much for such a detailed explanation!! as one of my clients usually says: "it's never easy!" ;)

 

I understand it. I actually use the STW feature to quickly move sites to a private development server when I cannot use ssh on the hosting provider and it's a great feature!!!

This request is for when you need to restore an old backup of the site or just want to check your backups can be safely restored (as you usually say: "an untested backup is as good as no backup at all")

I have also used Akeeba UNiTE in the past and it looks like the perfect tool to fullfill my needs right now as it supports custom endpoints as far as I can see.

Thank you very much!!

Best!

 

nicholas
Akeeba Staff
Manager

You're welcome! Have a great day :)

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

Support Information

Working hours: We are open Monday to Friday, 9am to 7pm Cyprus timezone (EET / EEST). Support is provided by the same developers writing the software, all of which live in Europe. You can still file tickets outside of our working hours, but we cannot respond to them until we're back at the office.

Support policy: We would like to kindly inform you that when using our support you have already agreed to the Support Policy which is part of our Terms of Service. Thank you for your understanding and for helping us help you!