Support

UNiTE, Remote CLI, eXtract Wizard

#40246 Akeeba Unite Restoration From Amazon S3

Posted in ‘UNiTE and Remote CLI’
This is a public ticket

Everybody will be able to see its contents. Do not include usernames, passwords or any other sensitive information.

Environment Information

PHP version
8.1
Tool
UNiTE
Tool version
5.1.1

Latest post by nicholas on Monday, 12 February 2024 07:51 CST

JoeJoomla

Hi Nicholas,

I am attempting to set up a Joomla 4 website restoration using Akeeba Unite. Right now I am setting up a test whereby both servers for the Joomla website and the restore location are on external publicly accessed domains. The real case I will need to use this in will be for a Joomla 3 site that where the domains reside on a protected network accessible via VPN. But for now, just getting my head around the setup I need some help.

Here is what I am trying to accomplish:

A nightly backup of the main site by Akeeba Backup Pro post processed to Amazon S3.

A nightly restoration of the most recent backup file from Amazon S3 by Akeeba Unite. This restoration will be to a different server.

In the XML file section for Amazon S3 the field '<filename>path/to/site-backup.jpa</filename>' is set for a specific backup file. I used ChatGPT 4 to question how Unite would pick the most recent backup file to use for restoration. It appears that this is not how Unite works and that I would have to program something in order to have it work this way. This is beyond my knowledge and ability and before I go down this path I'd like to pause here and get your advice on how I need to approach this?

Thank you Nicholas!

Joe Sonne

nicholas
Akeeba Staff
Manager

ChatGPT 🤮 No, glorified autocomplete which needs hundreds of Watt-hours to come up with unreliable bollocks is definitely not the way to go about it. Let's instead use some power-efficient (33W TDP) hardware optimised for complex problem solving. Yeah, that's our brains right in our head.

As per the fine manual you can't really do what you want to do. UNiTE expects a filename. It cannot magically know your archive naming organisation and nomenclature which is necessary to determine what is the latest backup. So, we need to download that latest backup with a name UNiTE can recognise, and then tell UNiTE to restore it.

Therefore, we have these discrete steps:

  1. Find my latest backup.
  2. Download my latest backup.
  3. Restore my latest backup.

Before we go ahead, let's note that since this is a matter of automation we want to use command line tools only. You could use AWS' own CLI but it's rather cumbersome since it's meant to work with all of AWS' services. Instead, we're gonna use s3cmd which is S3-specific, very easy to use, and very fast indeed. If you have not set it up already, go ahead and do it. I will assume it's accessible as s3cmd from your command line.

Find my latest backup

BACKUP_FILENAME=`s3cmd ls s3://my-bucket/site-www.example.com-* | grep .jpa | tail -1 | awk '{print $4}'`

This looks into the bucket my-bucket for any file whose name starts with site-www.example.com-* and prints them alphabetically. Since our naming convention is site-[HOST]-[DATE]-[TIME] alpha sorting is also time sorting.

In case we have multiple parts per backup we use grep .jpa to only fetch the .jpa files from our listing.

We then use tail -1 to only return the latest entry which looks like this:

2024-02-05 01:10    700550664  s3://my-bucket/site-www.example.com-20240204-190002cst.jps

Therefore we use awk to only return the fourth field, which is the filename.

Download my latest backup

s3cmd --skip-existing --no-check-md5 get $BACKUP_FILENAME

As simple as that. Now we get the latest backup file back from S3.

If it's a multipart archive we'll need to do a simple loop to iterate through extensions j01, j02 etc until we get an error.

Restore my latest backup

To restore the backup, we need it to have a predictable filename. So: 

mv $(basename $BACKUP_FILENAME) site.jpa

Now we can have our UNiTE file always try to restore site.jpa.

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

JoeJoomla

Thank you for the detailed instructions Nicholas. I really appreciate it.

I will do my best to absorb this information and implement it. This is me being s t r e t c h e d d d d d d d d d d!

Best regards,

Joe Sonne

JoeJoomla

Hi Nicholas,

I keep getting this error message when I run the cron for Unite:

The referenced package, [packagename] does not exist.

The zip backup is in the Amazon S3 bucket for sure.

I've tried many iterations to get the information in the S3 fields correct and I am missing it somehow. Is there a way I can send you what I have in the Amazon S3 XML file fields in a private message? Or are the attachments in here private?

Joe

 

nicholas
Akeeba Staff
Manager

The attachments here are private. However, when you send me the XML file put it in a ZIP file, and replace real passwords and bucket names with dummy values. I just need to look at the structure of the file.

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

JoeJoomla

Hi Nicholas,

Thank you. File is attached.

Best regards,

Joe Sonne

nicholas
Akeeba Staff
Manager

Your package key is wrong. When restoring from S3 you need to have

<package from="s3"></package>

This tells UNiTE to look at the <s3> section, download the archive part file(s), and use these as the file input.

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

JoeJoomla

Thanks Nicholas, I feel like I am making progress. I have a different error now.

What is Connector.php line 935 referencing? My earlier error was from Validator.php line 64

See attachment

I appreciate your patience with me in this.

Joe

 

nicholas
Akeeba Staff
Manager

This means that the bucket name is wrong.

Please remember that the bucket name is case sensitive. The bucket names FooBar, foobar, FOOBAR, and fooBar are different. Ideally, you should only ever use a bucket whose name is all lowercase.

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

Support Information

Working hours: We are open Monday to Friday, 9am to 7pm Cyprus timezone (EET / EEST). Support is provided by the same developers writing the software, all of which live in Europe. You can still file tickets outside of our working hours, but we cannot respond to them until we're back at the office.

Support policy: We would like to kindly inform you that when using our support you have already agreed to the Support Policy which is part of our Terms of Service. Thank you for your understanding and for helping us help you!