Duplicity

Threading with Backblaze B2?

Asked by Hans

In spite of my fast uplink, my upload speed to backblaze B2 is rather low: less than a tenth of what I could get (300mbps on their speed test, 15mbps upload in reality). Backblaze states that this is due to the fact that the ping delay I have to backblaze is very high (180ms: EU->US). Backblaze does 1 upload at a time, hence the sensitivity to delay. They suggest using multiple connections through threading.
I have found no option to enable threading in the B2 backend in duplicity.
Is that a feature that is
* available?
* foreseen?
* possible?
* not wanted?

I could help implementing it, if I manage to free up some time.
(latest duplicity by the way, from tarball, on debian 9)

Question information

Language:
English Edit question
Status:
Solved
For:
Duplicity Edit question
Assignee:
No assignee Edit question
Solved by:
edso
Solved:
Last query:
Last reply:
Revision history for this message
Kenneth Loafman (kenneth-loafman) said : 		#1

Multi-threading the backends is not available, yet. It is on the wish
list, but there are about 20 backends, so the task will be quite involved.

On Sun, Dec 23, 2018 at 6:17 AM Hans Boot <
<email address hidden>> wrote:

> New question #676995 on Duplicity:
> https://answers.launchpad.net/duplicity/+question/676995
>
> In spite of my fast uplink, my upload speed to backblaze B2 is rather low:
> less than a tenth of what I could get (300mbps on their speed test, 15mbps
> upload in reality). Backblaze states that this is due to the fact that the
> ping delay I have to backblaze is very high (180ms: EU->US). Backblaze does
> 1 upload at a time, hence the sensitivity to delay. They suggest using
> multiple connections through threading.
> I have found no option to enable threading in the B2 backend in duplicity.
> Is that a feature that is
> * available?
> * foreseen?
> * possible?
> * not wanted?
>
> I could help implementing it, if I manage to free up some time.
> (latest duplicity by the way, from tarball, on debian 9)
>
> --
> You received this question notification because your team duplicity-team
> is an answer contact for Duplicity.
>
> _______________________________________________
> Mailing list: https://launchpad.net/~duplicity-team
> Post to : <email address hidden>
> Unsubscribe : https://launchpad.net/~duplicity-team
> More help : https://help.launchpad.net/ListHelp
>

Revision history for this message
Best edso (ed.so) said : 		#2

and it's not only the backend infrastructure.

currently backup file creation (volumes, manifest ...) is also running in the same thread, meaning a volume is created and uploaded and only after that the next is created etc.. exception being the '--asynchronous-upload' where the next volume is prepared during the upload of the previous volume already, but only during upload and limited to one.

to conclude, duplicity currently lacks a threading architecture that utilizes multiple cores or optimizes bandwidth.

..ede/duply.net

Revision history for this message
Hans (hb020) said : 		#3

Thanks edso, that solved my question.