Duplicity

Volume size: Pros and Cons of big Volumes?

Asked by Rodrigo Alvarez

I'm backing up about 350GB with duplicity, and at 25MB per volume that is 14000 volumes at the very least. I'm considering running --volsize 250 but wanted a sanity check.

The only unofficial info about volume size I could find is at:
http://blog.gmane.org/gmane.comp.sysutils.backup.duplicity.general/month=20090101

In my case, I have no shortage of temp space (32GB) and I'm uploading to a mounted cifs volume that lives in the same LAN.

Am I Go/No Go for --volsize 250?

Question information

Language:
English Edit question
Status:
Solved
For:
Duplicity Edit question
Assignee:
No assignee Edit question
Solved by:
Peter Schuller
Solved:
Last query:
Last reply:
Revision history for this message
Best Peter Schuller (scode) said : 		#1

As far as I know, what I stated in that mailing list thread pretty much still applies. If you're fine with the temp space, and your volume sizes are not so large that individual transfers are too unreliable, and your storage backend does not impose limits, I see no reason not to go for 250 mb.

When I did that the first time (backing up a few hundreds of gigs), I did run into a performance issue but that was fixed as a result (duplicity no longer allocates huge buffers just because the volume size is huge).

If I recall correctly I was doing 250 mb volume sizes for a long time; and that was towards s3/rsync.net, rather than a local network-mounted filesystem.

Note though that if the 'storage backend' is the fie system, is does depend on how it behaves. I'm not aware of issues with CIFS, but for example some versions of AFS/Coda might have issues with large file efficiency.

Revision history for this message
Rodrigo Alvarez (rodrigo-alvarez-i) said : 		#2

Thanks Peter Schuller, that solved my question.

Revision history for this message
Rodrigo Alvarez (rodrigo-alvarez-i) said : 		#3

I'd like to follow up on this.

I just compressed ~300GB using 250 MB volume sizes.

In short, on a small white box running Ubuntu 10 and duplicity 0.6.08b, it took about 25 hrs @ 4MBps and compressed to 97% of original. Mostly files are photos in RAW (.CR2). The subsequent incremental backups will be small and quick so this very long first full backup is bearable. I had no expectations for compression, nevertheless, context aware compression would be nice to make compression efficient.

Possible bug:
Noteworthy is the output of a very large signatures file: 2147536063 bytes (~2GB)
$ ls -l *signatures*
-rw------- 1 Owner Group 2147536063 2010-07-04 23:56 duplicity-full-signatures.20100704T055652Z.sigtar.gpg

This is not a problem for me but it could create conflicts for other users with file transfer limits as stated before, and therefore this signatures file should not exceed the volume size.

--------------[ Backup Statistics ]--------------
StartTime 1278223013.20 (Sat Jul 3 22:56:53 2010)
EndTime 1278312373.73 (Sun Jul 4 23:46:13 2010)
ElapsedTime 89360.53 (24 hours 49 minutes 20.53 seconds)
SourceFiles 98281
SourceFileSize 355038288199 (331 GB)
NewFiles 98281
NewFileSize 355038288199 (331 GB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 98281
RawDeltaSize 355027466567 (331 GB)
TotalDestinationSizeChange 344039855481 (320 GB)
Errors 0
-------------------------------------------------

Revision history for this message
edso (ed.so) said : 		#4

You might want to try asking gpg to use another compressor e.g.

"--compress-algo=bzip2 --bzip2-compress-level=9"

..ede

On 05.07.2010 09:19, Rodrigo Alvarez wrote:
> Question #115570 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/115570
>
> Rodrigo Alvarez posted a new comment:
> I'd like to follow up on this.
>
> I just compressed ~300GB using 250 MB volume sizes.
>
> In short, on a small white box running Ubuntu 10 and duplicity 0.6.08b,
> it took about 25 hrs @ 4MBps and compressed to 97% of original. Mostly
> files are photos in RAW (.CR2). The subsequent incremental backups will
> be small and quick so this very long first full backup is bearable. I
> had no expectations for compression, nevertheless, context aware
> compression would be nice to make compression efficient.
>
> Possible bug:
> Noteworthy is the output of a very large signatures file: 2147536063 bytes (~2GB)
> $ ls -l *signatures*
> -rw------- 1 Owner Group 2147536063 2010-07-04 23:56 duplicity-full-signatures.20100704T055652Z.sigtar.gpg
>
> This is not a problem for me but it could create conflicts for other
> users with file transfer limits as stated before, and therefore this
> signatures file should not exceed the volume size.
>
> --------------[ Backup Statistics ]--------------
> StartTime 1278223013.20 (Sat Jul 3 22:56:53 2010)
> EndTime 1278312373.73 (Sun Jul 4 23:46:13 2010)
> ElapsedTime 89360.53 (24 hours 49 minutes 20.53 seconds)
> SourceFiles 98281
> SourceFileSize 355038288199 (331 GB)
> NewFiles 98281
> NewFileSize 355038288199 (331 GB)
> DeletedFiles 0
> ChangedFiles 0
> ChangedFileSize 0 (0 bytes)
> ChangedDeltaSize 0 (0 bytes)
> DeltaEntries 98281
> RawDeltaSize 355027466567 (331 GB)
> TotalDestinationSizeChange 344039855481 (320 GB)
> Errors 0
> -------------------------------------------------
>
> You received this question notification because you are a member of
> duplicity-team, which is an answer contact for Duplicity.
>
> _______________________________________________
> Mailing list: https://launchpad.net/~duplicity-team
> Post to : <email address hidden>
> Unsubscribe : https://launchpad.net/~duplicity-team
> More help : https://help.launchpad.net/ListHelp

--
public class WhoDidIt{ // A comment. I love comments
   private static Person sender;

   public static void main (String[] foo){

   sender = new Person();
   sender.setName(new String[]{"Edgar", "Soldin"});

   Address address = new Address();
   address.setStreet("Stadtweg 119");
   address.setZip(39116);
   address.setCity("Magdeburg");
   address.setCountry("Germany");

   sender.setAddress(address);

   sender.setMobilePhone(" +49(0)171-2782880 ");
   sender.setWebSiteUrl(" http://www.soldin.de ");
   sender.setEmail(" <email address hidden> ");
   sender.setPGPPublicKey(" http://www.soldin.de/edgar_soldin.asc ");
   sender.setGender(true);

   System.out.println(sender.toString());
   }
}