Support

Akeeba Backup for Joomla!

#35224 are there any known issues regarding dropbox connectivity? (dropbox upload doesnt work)

Posted in ‘Akeeba Backup for Joomla! 4 & 5’
This is a public ticket

Everybody will be able to see its contents. Do not include usernames, passwords or any other sensitive information.

Environment Information

Joomla! version
n/a
PHP version
n/a
Akeeba Backup version
n/a

Latest post by on Thursday, 01 July 2021 20:17 CDT

pthron

Description of my issue:

- Since some days  akeeba backup  doens't upload backups to dropbox.

- "normal" backup (no postprocessing) works well and finishes quickly.

We can see these problems at some of our sites only. At other sites  dropbox upload works well.same configuration.

Logfile doesnt report any errors.  it's cycling in step 13 (upload dropbox) 3-times.

- always 250 secs. then ends.

we have tried some minimum/maximum execution times and bias. nothing changed.

we made new "authentication - step 1" with dropbox. Entered new acchess token. no change.

any ideas?

what do we miss?

regards :) klaus

 

 

 

nicholas
Akeeba Staff
Manager

I have not seen any issues and I just completed a backup to Dropbox.

First, try doing the authentication again for each failing backup profile.

If this doesn't help, we will need a backup log file. I have a suspicion that your server has a firewall for outgoing connections and your host would need to allow connections to Dropbox. Do note that hosts do not announce which additional security measures they implement and when, or even to which servers in their fleet. This is why you might experience things "suddenly" not working or working inconsistently across different sites on the same hosting provider.

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

pthron

Thx for your quick reply.

I've changed dropbox authorization some more times already.

FW at hosting provider server could be an idea.

the provider is strato. one of the bigger german providers, i think.

It looks somehow anusual to secure some servers with FW and others not.

And not allowing dropbox connections from inside...  

anyway...  I put them the question...  waiting for answer...

----   back to akeeba backup....

now I could catch an error message :)

It is difficult because it dissappears again very quickly, and it doesnt find the way into the logfile, too.

maybe it is of some help?

here is the message, which can be seen after 250 seconds upload trial... and disappears after 9 seconds ..

then next upload trial starts.... this procedure ist repeated 3-times.

--

The backup operation has been halted because an error was detected. However, Akeeba Backup will attempt to resume the backup. If you do not want to resume the backup please click the Cancel button below.

The backup will resume in 1 seconds

For your information, the last error message was:

<strong>AJAX Loading Error</strong><br/>HTTP Status: 502 (Bad Gateway)<br/>Internal status: error<br/>XHR ReadyState: 4<br/>Raw server response:<br/><!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>502 Proxy Error</title> </head><body> <h1>Proxy Error</h1> <p>The proxy server received an invalid response from an upstream server.<br /> The proxy server could not handle the request<p>Reason: <strong>Error reading from remote server</strong></p></p> </body></html>

--

i'll send logfile in private channel.

regards :) klaus

pthron

2 questions....

- regarding possible FW at hoster server...   what should be allowed? how will transfer to dropbox be started?...

 

and....

how to send logfile in private to you ??   new "private" ticket?

regards :) klaus

 

nicholas
Akeeba Staff
Manager

Please note that all attachments are private, even on public tickets. You don't have to file a private ticket.

It does look like the problem might be a proxy server for outbound connections on your host. I see that the first request to create a new upload session goes through to Dropbox. The second request to actually upload the data seems to hang waiting for a network response until the server gives up (that's the 502 error reported).

This is consistent with the experience other clients of ours have had on Strato, insofar as inconsistency of server setup and occasionally encountering server proxies which effectively kill PUT requests can be called consistent.

What most likely happens is that their proxy "eats" the body (the chunk of binary file data) of the PUT request we send to Dropbox. Dropbox patiently waits for this data to arrive, leading to an impasse: your server sends nothing, Dropbox waits and sends nothing. At some point your web server decides that it has been waiting long enough and kills the PHP execution with a 502 error.

You need to ask your host to look into it and reconfigure their proxy to allow PUT requests to go through correctly.

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

pthron

thx a lot.

yes, we can see...

however we are astonished a little bit, too.

We have other sites hosted by strato, were we see none of these problems.  all upload to dropbox works fine.

I did just set up a new site there, where all works fine, too...

hope we can solve this soon :)   

have a nice weekend :)

regards Klaus

nicholas
Akeeba Staff
Manager

I understand :) The first time you get two servers on the same host yielding radically different results it's unnerving. What usually happens is that servers may belong to different "generations" of configuration or a server might be in a server group which has exhibited an increased rate of problems for the host (getting its IP blacklisted by spam filters, they received too many abuse reports etc) which might justify stricter measures across all sites hosted on it. Sometimes servers may be part of a group that undergoes a trial of new features / configuration options the host intends to apply. And so on and so forth. When in doubt, ask the host. You may get an evasive reply but they'll do their best to address the situation.

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

pthron

Nicholas, thx.

I think it could be of some help, to know how  akeeba backup gets the contact with dropbox, and how the transfer of data is planned...

Which port is used? which protocol? and so on....

You can speak technical.  Some years ago - in an earlier life - I administered a lot of linux servers and FWs too. in a big company.

I think when I can handle this information to webhosting support, they can easier look up what is going on...

-----

And... back to now....

I made some tests this weekend. :)

I created a new and very small and simple website.  In a hosting environment where the data transfer to dropbox failed (from other bigger websites).

With some interesting results....

In the new website I installed joomla only. Nothing else. and akeeba backup.

local backup worked like a charme.  I didnt wonder :)

Backup with postprocessing to dropbox  worked well too,  with "db only" - very small DB - but without any problem.

Backup with full website ... I tried this with different amount of data...    (uploaded some large files before that) 

backupped all first ..  and then excluded some of the bigger files.

 

As long long as the amount of data was upto 15 MB  all worked well too. 

Sometimes it lasted a little bit longer...

however, amount of data over 20 MB  ran into looping always... 

-----

In akeeba backup configuration - dropbox connection configuration -  I recognized a parameter  "chunk size"  beneath some other configuration, where i didnt understand exactly what they do...

chunk size is 20 MB configured by default.

After this experience it seemed to me, that connection will be disturbed or  interrupted  when more then 1 chunk has to be transferred...

This is somewhat annoying...  It is not that easy to think of a configuration in proxy or FW which works in this way.

I tried with other chunk size.-  higher or lower -  nothing changed....

I can give you an account in this environment.  If you want...  You can look by yourself and perhaps do some debugging...

please let me know :)

regards klaus

 

 

 

 

 

 

nicholas
Akeeba Staff
Manager

We use the official Dropbox API v2. It uses two different URLs, both using H.

https://api.dropboxapi.com/2/ is used to communicate commands and https://content.dropboxapi.com/2 is used to upload data.

There are two way to upload to Dropbox. One is straight up upload the entire file, the other one is chunked upload. I did check chunked upload before sending you my first reply.

In the straight up uploads things are straightforward. We make a single POST request to the content URL. However, since you're trying to upload a lot of data it might take too long and either PHP or your web server times out. Hence the chunked mode.

In the chunked upload each archive part is uploaded in smaller chunks. Each chunk is as big as the chunk size except for the last chunk which is the remained. So, if you want to upload a 93.4MB file with a 10MB chunk you have ten chunks: nine chunks at 10MB each and a final chunk of 3.4MB. We can pause the upload after each chunk and do a new page load, sidestepping the timeout issue as long as uploading each chunk takes less time than your PHP and web server timeout limit.

For chunked uploads we need to do n+2 requests to Dropbox. First a POST request to the api URL to get a chunked upload key, Then n POST requests (where n is the number of chunks) to the content URL to upload each chunk. Finally a POST request to the api URL to tell Dropbox we're done uploading.

Dropbox uses time limited OAuth2 access tokens for authentication. These last for one hour. When you are linking Akeeba Backup to Dropbox you get a pair of tokens, an access token which expires in one hour and a refresh token which does not expire. If a request fails, Akeeba Backup exchanges the refresh token for a new access token. Since this requires signing with our secret key we cannot distribute with our software this happens through a mediator service on our site. The mediator service requires you to have entered a valid Download ID in the component's options. The Download ID must correspond to a user account on our site with an active subscription which gives access to any of our backup products: AKEEBABACKUP, ESSENTIALS, BACKUPWP, WPESSENTIALS or SOLOPHP.

Your test probably failed for one of two reasons. You either didn't enter a Download ID or the chunk size was too big for the server.

However, that test is not the same problem I see in your real site's log file. What I see there tells me that the token exchange works perfectly fine. The first POST request to the api URL (which is of course authenticated!) works and we do get an upload session ID. The second POST request, to the content URL, seems to never complete. Even if you're using a chunk size of 20MB, at 250 seconds we'd be talking about a transfer rate of under 82 Kb/sec. This is WAY too low to be true, unless your server is on a PSTN dial-up connection. Since the calendar reads 2021, not 1991, I think that's more than unlikely. What is more likely and we have seen before is what I told you: there is a firewall or proxy on the server which "eats up" the POST request to the content URL so that it never returns.

You really need to contact your host and tell them that your site will be uploading stuff to Dropbox therefore requests whose URL begins with the two URLs I shared above must go through and they must be uncached (the HTTP protocol already specifies POST requests must go through uncached but I've seen plenty of hosts doing things they should not be doing, so yeah...).

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

pthron

Ok and thx a lot :)

akeeba-backup --> dropbox   works now fine, too.

Also with websites  residing at Strato :)

seems they have done something right now :)

best regards

:) klaus

 

 

nicholas
Akeeba Staff
Manager

It must have been a problem with their proxy server, as we suspected :)

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

System Task
system
This ticket has been automatically closed. All tickets which have been inactive for a long time are automatically closed. If you believe that this ticket was closed in error, please contact us.

Support Information

Working hours: We are open Monday to Friday, 9am to 7pm Cyprus timezone (EET / EEST). Support is provided by the same developers writing the software, all of which live in Europe. You can still file tickets outside of our working hours, but we cannot respond to them until we're back at the office.

Support policy: We would like to kindly inform you that when using our support you have already agreed to the Support Policy which is part of our Terms of Service. Thank you for your understanding and for helping us help you!