8TB in 10 MB chunks? This can't be a good idea

Asked by Darren Obbard

[I tagged this on to the end of a previous question - but realize it should be a thread in itself]

Is there any way to change teh 10Mb chunk size for remote backups?

I am backing up ~8TB in total. I suspect this is not going to go well when it is broken into 800,000 files in a single folder. Surely people must have run in to this a lot? is it just an option I cannot find?

Thanks,

Darren

Question information

Language:
English Edit question
Status:
Answered
For:
Déjà Dup Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Michael Terry (mterry) said :
#1

This is not customizable. Though modern Deja Dup versions use 50MB locally and 25MB remotely. As to why we picked the values we did, I'll quote some comments from the source code:

    // Advantages of a smaller value:
    // * takes less temp space
    // * retries of a volume take less time
    // * quicker restore of a particular file (less excess baggage to download)
    // * we get feedback more frequently (duplicity only gives us a progress
    // report at the end of a volume) -- mostly fixed by reporting when we're uploading
    // Downsides:
    // * less throughput:
    // * some protocols have large per-file overhead (like sftp)
    // * the network doesn't have time to ramp up to max tcp transfer speed per
    // file
    // * lots of files looks ugly to users

Revision history for this message
Darren Obbard (darren-obbard) said :
#2

Hmmmmm - how did I end up with an old version? Where do I go to get the newest version? This one (14.2) was the only one offered to me via synaptic package manager on under Ubuntu 10.04.

Re: file sizes - Yes, all seems reasonable - except

 // * lots of files looks ugly to users

800,000 files in a single folder can cause grief if you ls (or similar) when not expecting it.

Revision history for this message
Michael Terry (mterry) said :
#3

14.2 is the expected version to have with Ubuntu 10.04 LTS. But that is over two years old.

Note that 10.04 LTS is now end-of-life on the desktop. I'd recommend upgrading to 12.04 LTS. You'll get a newer Deja Dup as well as ongoing security updates.

Not only security updates, but data corruption updates to duplicity and deja-dup. I know there is one possible bug that got fixed in 12.04 LTS, but not 10.04 LTS. (Bug 1091269 -- actually, it's just waiting for confirmation on 10.04 LTS, maybe if you're intent on using it, you could confirm that bug so it can reach all users)

Revision history for this message
Michael Terry (mterry) said :
#4

Over three years old! My math is bad today. :)

Can you help with this problem?

Provide an answer of your own, or ask Darren Obbard for more information if necessary.

To post a message you must log in.