Multi-Thread FTP Uploads?

Asked by Martin Klefas

I've been using Duplicity for a while now, and let me first say that it's absolutely fantastic.

Having asked elsewhere I know it doesn't currently support concurrent ftp uploads of two or more volumes of the backup at once, so I was wondering if anyone could point me towards the parts of the source code that I would need to look at to perhaps contribute that option.

Thanks in advance,

Martin

Question information

Language:
English Edit question
Status:
Answered
For:
Duplicity Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Kenneth Loafman (kenneth-loafman) said :
#1

Thanks for the kind words.

Duplicity does indeed the --asynchronous-upload option, but currently only
for one thread at a time. You are more than welcome to enhance the current
code to allow more threads of execution. You'll need to look into
asyncscheduler.py for the code and how it's used. I'm sure there will be
plenty of challenges.

...Ken

On Mon, Jul 6, 2015 at 8:07 AM, Martin Klefas <
<email address hidden>> wrote:

> New question #268927 on Duplicity:
> https://answers.launchpad.net/duplicity/+question/268927
>
> I've been using Duplicity for a while now, and let me first say that it's
> absolutely fantastic.
>
> Having asked elsewhere I know it doesn't currently support concurrent ftp
> uploads of two or more volumes of the backup at once, so I was wondering if
> anyone could point me towards the parts of the source code that I would
> need to look at to perhaps contribute that option.
>
> Thanks in advance,
>
> Martin
>
> --
> You received this question notification because you are a member of
> duplicity-team, which is an answer contact for Duplicity.
>
> _______________________________________________
> Mailing list: https://launchpad.net/~duplicity-team
> Post to : <email address hidden>
> Unsubscribe : https://launchpad.net/~duplicity-team
> More help : https://help.launchpad.net/ListHelp
>

Revision history for this message
Martin Klefas (cleverbum) said :
#2

Apologies, I know this will be in the remainder of the code, which seems very well documented, but I'm struggling to find all the places that the scheduler is called.

Once you've created the first volume Are you then doing volume creation at level 1 and upload at level 1 followed by a call to insert_barrier?

If so, would a workaround (for 2 upload threads) be:
 Volume level 1
 upload level 2
 volume level 1
 upload level 2
 insert_barrier

Or have I got the wrong end of the stick entirely?

Can you help with this problem?

Provide an answer of your own, or ask Martin Klefas for more information if necessary.

To post a message you must log in.