Support

Akeeba Backup for Joomla!

#13463 Problem with Native Cron when performing Incremental backup. It doesn't add new files from second Inc. onwards

Posted in ‘Akeeba Backup for Joomla! 4 & 5’
This is a public ticket

Everybody will be able to see its contents. Do not include usernames, passwords or any other sensitive information.

Environment Information

Joomla! version
n/a
PHP version
n/a
Akeeba Backup version
n/a

Latest post by nicholas on Friday, 07 September 2012 09:33 CDT

user8243
Mandatory information about my setup:

Have I read the related troubleshooter articles above before posting (which pages?)?
Yes. But there is no page about incremental backups. There is one about automating the backups but is not what I need.
https://www.akeebabackup.com/documentation/troubleshooter/abautomation.html
https://www.akeebabackup.com/documentation/troubleshooter/abotherq.html

Have I searched the tickets before posting?
Yes
https://www.akeebabackup.com/support/akeeba-backup-3x/8506-incremental-backup-2.html

Have I read the documentation before posting (which pages?)?
Yes.
https://www.akeebabackup.com/documentation/akeeba-backup-documentation/configuration.html
https://www.akeebabackup.com/documentation/akeeba-backup-documentation/native-cron-script.html
https://www.akeebabackup.com/documentation/walkthroughs/akeeba-backup-cron-cpanel.html

Joomla! version:
1.5.26
PHP version:
5.2.9
MySQL version:
5.0.95-community
Host:
iWeb - Dedicated Server with WHM/Cpanel management.
Akeeba Backup version:
3.4.3 (Because of the Joomla compatibility issue)

EXTREMELY IMPORTANT: Please attach a ZIP file containing your Akeeba Backup log file in order for us to help you with any backup or restoration issue. If the file is over 2Mb, please upload it on your server and post a link to it.

Description of my issue:
I’m trying to configure incremental backups on my server using the Native CRON way. It is really important to do it in this because I’m backing up music (most than 500GB), so, even having a dedicated server, I need to optimize the resources, of course.
Before anything, it worth to mention that:

  • The OS server is Linux

  • The Joomla site is on HDD “X”

  • The music is on HDD “Y”

  • The backups folder is on HDD “Z”

  • I have the needed permissions for all of the HDDs and folders as well.

  • Joomla is 1.5

  • Akeeba Backup Pro is 3.4.3, so I’m not using any “cli” folder since this version doesn’t have one but I’m do using the file named JSite/administrator/components/com_akeeba/backup.php, which is the right one for this version, is that right?


So, guided by what I read on your forums, I’ve did the following (notice that every configuration field that I don’t mention is because is set as default):
  • Created a profile #1 for FULL backup of the music folder. Its cronjob runs once a week.
    [list]
    Output directory: Z/path/to/my/folder

  • Backup archive name: FULL-music-[DATE]

  • Archiver engine: ZIP format

  • Embedded restoration script: No installer

  • Enable count quota: checked

  • Count quota: Custom... 1.00

[/list]
  • Created profile #2 for INCREMENTAL backup of the same music folder. Its cronjob runs every day.
    [list]
    Output directory: Z/path/to/my/folder

  • Backup archive name: INC-music-[DATE]

  • Archiver engine: ZIP format

  • Embedded restoration script: No installer

  • Enable count quota: checked

  • Count quota: Custom... 7.00

[/list]

I want the format to be ZIP because is more easier to handle for me, as in any case, I only should have to unzip the content to the destination folder without any intermediary software or anything like that.

What happens?
For testing purposes, I tried the profile #2 and I add to its name the “[TIME]” string in order to differentiate one backup from another as I do them all in the same day, as I don’t want to wait an entire week to be sure that the component works as expected with the cronjobs, so:

  • For the first incremental run, I’ve set the cronjob to be executed in near time so I test how the cron job works. It does it fine! Obviously, it copies all the content, at it doesn’t exists yet as backup.

  • Then I add a few folders with music and again, set the cron to be executed a few minutes in the future, so I test if it only creates a zip with the new files; which I think is what it should do, isn’t it? Maybe I’m wrong... Well, here is the problem: it creates a file with the expected name but the ZIP only contains the new folders, but without the new music inside.


I look over the akeeba.cli.log file in order to see if, at least, it reads the files and... it does it but skipping them, which is weird as they are new files and they should be included.
Now, I did many things in order to test possible scenarios:
  • I’ve deleted the main backup file and backed up again. Result: it created a file but as on the mentioned point 2, I mean, it reads all files but now it skips all of them, not only the new ones.

  • I waited one hour to test again and “see” what happens (just like if a miracle could help haha) and, weirdly, it created the file with all the songs again. But when I try again, in a closer time for testing purposes, stills happens the same. So I’m little dizzy...

So I wonder if the incremental backup actually works or if it only works once a day or if there is something I’m missing (this is probably what’s actually happening).
I really don’t know how the cron script really works and I don’t want to be doing reverse engineering, less if I have your support as I’ve purchased the Pro version of it.

Well, I wait for your answer. Hope I've been clear enought. But of course, let me know if you have any question.

Hope you can help.

nicholas
Akeeba Staff
Manager
Then I add a few folders with music and again, set the cron to be executed a few minutes in the future, so I test if it only creates a zip with the new files; which I think is what it should do, isn’t it? Maybe I’m wrong... Well, here is the problem: it creates a file with the expected name but the ZIP only contains the new folders, but without the new music inside.

This is the expected and desirable behaviour. OK, it's a little strange, so let me explain it. This feature works by checking the "created on" timestamp of each folder and file. It compares them with the date and time of the last successful backup with this profile. If the file/folder's timestamp is greater then the backup's date and time (i.e. the file was created after the last backup was taken) the file is backed up. There's a catch: the default behaviour of most FTP clients is to preserve the timestamps of files. This means that if you look at when the music file was created you'll see that it's a date/time before the last backup. So Akeeba Backup should, indeed, not back it up. The solution is setting the created date/time of uploaded files to be equal to the upload date/time. You should ask the developers of your FTP application how to do that. It's usually a simple toggle buried deep inside some configuration page.

I waited one hour to test again and “see” what happens (just like if a miracle could help haha) and, weirdly, it created the file with all the songs again. But when I try again, in a closer time for testing purposes, stills happens the same. So I’m little dizzy...

This may also have to do with the timezone difference between GMT and your server's local time. Let me take a wild guess: your server is located in London?

Hope I've been clear enough

Perfectly! It was a welcome change in a day full of trying to guess what people have forgotten or skipped telling me when asking for support :)

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

user8243
Don't worry.

I'm the developer. I'm using my client's account to post the ticket. The better thing to do is to ask directly without intermediates, right?

About your answer, you've been so clear.
So, if I understood, the script takes the created time as parameter to know if the file needs to be backed up or not, right?

If it's that, then there is no problem at all, 'cause my client uploads the new music every day.

All the time I've been testing with existent files, copying them from one place to another, so surely happened what you say: they preserved their created timestamp.

I'll keep the ticket opened until tomorrow. Why? Because today he uploaded new music. So it's supposed to work fine because the music is new: their created and modified time are exactly the same and they are both major than the last backup time.

If tomorrow it's all okay, then I'll close the ticket.

Thank you very much for the very detailed answer.

I'll let you know if anything tomorrow.

PS: Maybe this could help for developers debug: in the log file, would be nice that, if is an incremental backup, it figures the last backup date and the file date which you compare. So if the file is older than the last backup, the script could say something like "is is the older backup" or something like that. Maybe you think is doesn't worth it, but it's up to you, of course. I think would be nice! Just saying...

Regards!

nicholas
Akeeba Staff
Manager
So, if I understood, the script takes the created time as parameter to know if the file needs to be backed up or not, right?

Precisely! As I've written in the documentation: "The only comparison made is between the file's modification time and the last successful backup's time. The "last successful backup" refers to the last backup made using this backup Profile and which has a status of "OK", "Remote" or "Obsolete"."

All the time I've been testing with existent files, copying them from one place to another, so surely happened what you say: they preserved their created timestamp.

Seems so.

I'll keep the ticket opened until tomorrow. Why? Because today he uploaded new music. So it's supposed to work fine because the music is new: their created and modified time are exactly the same and they are both major than the last backup time.

Fair enough :)

Thank you very much for the very detailed answer.

You're welcome!

Maybe this could help for developers debug: in the log file, would be nice that, if is an incremental backup, it figures the last backup date and the file date which you compare. So if the file is older than the last backup, the script could say something like "is is the older backup" or something like that. Maybe you think is doesn't worth it, but it's up to you, of course. I think would be nice! Just saying...

It would bloat the log file. The comparison is made in a Filter file. OK, let me explain this. Akeeba Backup always starts scanning all of your site's folders recursively. As it scans each folder it gets a list of files and folders. As soon as it gets the listing it puts it through the filters. The filters determine if this file/folder should be backed up. The cleared files are added to the files list, the cleared folders are added to the to-be-scanned folders list (it's actually a tad more complicated, but this simplified explanation is very close to what actually happens). As a result the last backup date is first calculated during the first call the the incremental file inclusion filter for each backup Step. Implementing your feature request would end with this timestamp output on each backup step at seemingly random spots within the backup file. It would be very hard to figure out where to find it. Moreover, it would have to list the timestamps of each file and folder. It gets really messy really fast.

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

user8243
I totally understand.

I did many exporting modules that work by reading directories and files. I totally understand that can be very tricky trying to implement something like that. As I said: "just saying...".

Besides, doing echoes for all the timestamps, could slow the timing of the script, now I'm deeply thinking about it.

So... leave my comment as it was: just a comment haha

Thanks again!

nicholas
Akeeba Staff
Manager
You're welcome :)

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

user8243
Just wanted to let you know that the backups worked perfectly.

Thank you very much for your attention.

And congratulations for such a very nice component!

Regards!
Nahuel Sotelo
Cloudlight.me

nicholas
Akeeba Staff
Manager
You're welcome! I'm glad it's all working now :)

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

Support Information

Working hours: We are open Monday to Friday, 9am to 7pm Cyprus timezone (EET / EEST). Support is provided by the same developers writing the software, all of which live in Europe. You can still file tickets outside of our working hours, but we cannot respond to them until we're back at the office.

Support policy: We would like to kindly inform you that when using our support you have already agreed to the Support Policy which is part of our Terms of Service. Thank you for your understanding and for helping us help you!