Duplicity

Parallelizing compression and gpg

Asked by alex

Dear duplicity developers,

I am using duplicity for a while now and do backup very large files (>500GB) on really fast machines (XEON CPUs, SSDs, and RAID10).

Wondering about the speed of duplicity I saw that duplicity only utilizes one CPU core and the storage is bored at about 20-30 MB/s.

What about parallelizing duplicity's compression and gpg implementation? I had a look at https://code.google.com/p/threadzip/ and it doesn't seem to be very complex. I am a programmer but do not dare to implement it myself because of lack in experience in python.

So, is someone willing to implement this? I can test it. GZIP parallelization is a good start for me.

Regards
Alex

Question information

Language:
English Edit question
Status:
Answered
For:
Duplicity Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Kenneth Loafman (kenneth-loafman) said : 		#1

The ThreadZip package would not be what we would use. We need to
parallelize gpg usage since that is the one that does the compression as
well as encryption. I don't have time to work on this at the moment, but
it is on the plan for someday.

At the moment, the best we can do is to use --asynchronous-upload to make
the upload and gpg steps go at the same time.

...Ken

On Sun, Sep 13, 2015 at 4:57 AM, alex <email address hidden>
wrote:

> New question #271336 on Duplicity:
> https://answers.launchpad.net/duplicity/+question/271336
>
> Dear duplicity developers,
>
> I am using duplicity for a while now and do backup very large files
> (>500GB) on really fast machines (XEON CPUs, SSDs, and RAID10).
>
> Wondering about the speed of duplicity I saw that duplicity only utilizes
> one CPU core and the storage is bored at about 20-30 MB/s.
>
> What about parallelizing duplicity's compression and gpg implementation? I
> had a look at https://code.google.com/p/threadzip/ and it doesn't seem to
> be very complex. I am a programmer but do not dare to implement it myself
> because of lack in experience in python.
>
> So, is someone willing to implement this? I can test it. GZIP
> parallelization is a good start for me.
>
> Regards
> Alex
>
> --
> You received this question notification because your team duplicity-team
> is an answer contact for Duplicity.
>
> _______________________________________________
> Mailing list: https://launchpad.net/~duplicity-team
> Post to : <email address hidden>
> Unsubscribe : https://launchpad.net/~duplicity-team
> More help : https://help.launchpad.net/ListHelp
>

Can you help with this problem?

Provide an answer of your own, or ask alex for more information if necessary.

To post a message you must log in.