5MB pieces

Thanks for the quick answer.

I would go for at least 100M pieces. With the 5M split, only a couple of files fit into each and I couldn't find an easy way of getting a specific file other than looking into one at a time. For large backups, this would make for a myriad of 5M chunks all over the place.

I haven't tried the incremental yet, but I imagine that this would add more little pieces all over the place?

Steve

Thanks Michael Terry, that solved my question.

Yup, incremental is more 5M files. Once you start doing incrementals, though, you really need to use a program (i.e. Deja Dup or duplicity) to cleanly get back the recent version of the file (since the incrementals contain patches to the original files), so the relevance of a file being wholly contained in a tarball is diminished.

But yeah, no reason not to go big or go home with local tarballs (as long as they still fit reasonably on common backup devices).

It's possible to add a simple option on general preferences to set the single amount of the pieces?

I think too, also for network backup or incremental, 5 MB is too small, now i have millions of gpg files on my local server :-)

Maybe if i can set 50 or 200 MB for Chunck all is more human readable.

Thank you for you attention.

+1 for BArrYZ
I would like to set volsize for backups. 5MB is tool small when you have some GB to backup locally (or local network). It could be useful to set this in gconf ;-)

Is it also possible to manage a list of tasks instead of only one?

thanks, and congrats for the awesome work :-)
a.

With 11.0, default volsize depends on whether the backup is remote or not. Remote volsize is 10MB, local volsize is 30MB.

I don't understand the 'list of tasks' question.

Hi Michael, thanks for your quick answer ;)

sorry, my bad english strikes again, I'll try to explain: It could be useful to manage a list of backup configs, each one has different options.

Anyway, I've installed 11.0, but for my needs I think that 30MB is still too low. If I would like to backup 5GB to external/local HD, then I will have more than 160 *little* files. It's not human readable :P

Integrating this feature (just as gconf key, hidden from GUI?) will permit every (experienced) user to use deja-dup with a custom volsize.

thanks in advance for you attention :)
a.

I've historically called lists of configurations "profiles," and the request for it is bug 324631. I encourage you to subscribe to that bug if interested.

The list of files being human readable isn't a big concern to me. The file list is in some sense supposed to be opaque. Users can't meaningful use those files themselves.

I am resistant to adding a gconf key for it, just because I consider it a minor detail, and I don't like gconf sprawl, but I'll keep it in mind.

I've just subscribed to bug :)

I've agree with your opinion about files, it's just a matter of *visual* convenience: if I would like to burn backups to fit a DVD, I don't need to view hundreds of files, even if they aren't usable themselves :)

about gconf key, I don't like it too. but using it, involves few changes to get implemented :P

cheers,
a.

I would like to be able so select the volume size for my backup chunks, too.

It is really nasty for a backup with more GiB to have hundreds of files in one directory, if it could be less.

I do not think that going from 5MiB to 30MiB is a step forward, why not just make it configurable?

I agree with BeowulfOF. Configurable chunk size will be useful. I have more than hundreds backup file and it is not easy to burn CD/DVD. My backup files on a server and periodically the server's hard disk backing up to another hdd or DVD. And the copying process is slow because of the number of files.

I've got another reason for a user-defined volume size:

I ran out of diskspace during a backup, writing to a 500GB portable USB drive. I don't know the filesystem, but it's just that there are way too much 30MB files in the subdirectory. The filesystem can't handle more. Even if the drive has still +400GB free space.

So please, don't tell me I can't make a backup because the filesystem I'm writing to is wrong. :-)

I agree for a user-defined volume size too... in my case I tell deja-dup to backup into a local folder but the directory is a remote filesystem mounted via fuse (deja-dup doesn't support that filesystem); it's a free box.net account and so it has file size limit to 25MB

I don't even need an option in the GUI but how about making it configurable via gconf, dconf or whatever deja-dup uses?

I backup to a tape robot with thousands of cartridges. For them, the biggest constraint is the number of files to manage, so the admins want the slices to be 500MB or more. I am backing up via ssh. at home, using my NAS box, I have a couple of terabytes available, and a single backup directory has 15 thousand files. It is not useful to slice it so finely. Again I´m using ssh target. I would like here again, to set --volsize of duplicity to oh... 500 mb.

Prefer if it were in the GUI and configurable as part of the storage location.

https://code.launchpad.net/~peter-bsqt/deja-dup/volsize

I created a version that has a volsize setting one can set with dconf-editor. no change to GUI.

Well, actually I had done this also a couple of weeks ago, the code is but still in my local repo, with GUI fields for changing the volsize according to the media.

BUT testing it completely changed my mind about this topic. Using 500MiB chunks or even just 100MiB chunks to save the backup on a network attached storage slows down backup process enormously. It is really much more fluent using 5/25MiB chunks, since duplicity blocks compressing until the transport is done.

After seeing how that slows down the backup, I can no longer stand the point, that it is useful to have bigger chunks.

Have you tried also adding --asynchronous-upload ?

Asynchronous upload sounds good. I'll update my patch and add that option as well, can take a day or two, but I'll let you know when I'm done.

Having a bit of a problem, maybe someone here can help me out. I'm trying to include a new widget with a SpinButton for the volsize setting. The File ConfigSpin.vala is in the deja-dup/widgets directory and is appended to the list in CMakeLists.c. But when I compile the whole thing I do get the following output:

make -f deja-dup/widgets/CMakeFiles/widgets.dir/build.make deja-dup/widgets/CMakeFiles/widgets.dir/depend
make[3]: Entering directory '/tmp/buildd/deja-dup-33.ubuntu3/obj-x86_64-linux-gnu'
make[3]: *** No rule to make target '../deja-dup/widgets/ConfigSpin.vala', needed by 'deja-dup/widgets/ConfigBool.c'. Stop.

I don't get it, why there is no rule, since rules get dynamically created as I understand it. Also I don't get why ConfigSpin.vala is needed by ConfigBool.c. Maybe someone can help me figure this out?

If you want to try it yourself, you can use my branch lp:~beowulfof/+junk/deja-dup

There were all kinds of errors when I tried to build it, get_integer doesn't exist, changed them to get_int,
the var = is_uri? line was complained about, no idea why, but I made it only fill it in when it was an integer.
SpinButton had no prototype, because you were calling it with no arguments. Built an ''Adjustment" that it needs
as the first argument, and filled in some integers for the other two. Now it matches the prototype and builds.

I see it in the GUI, but the value never gets saved.... one step fwd anyways.

blacklab% bzr diff
=== modified file 'deja-dup/widgets/ConfigSpin.vala'
--- deja-dup/widgets/ConfigSpin.vala 2015-05-07 19:54:37 +0000
+++ deja-dup/widgets/ConfigSpin.vala 2015-05-08 00:25:59 +0000
@@ -44,7 +44,8 @@

   protected Gtk.SpinButton spin;
   construct {
- spin = new Gtk.SpinButton();
+ var adj = new Gtk.Adjustment(0,0,0,0,0,0);
+ spin = new Gtk.SpinButton(adj,10,1);
     add(spin);
     mnemonic_widget = spin;

@@ -54,13 +55,19 @@

   protected override async void set_from_config()
   {
- var val = is_uri ? settings.get_uri(key) : settings.get_integer(key);
- spin.set_value(val);
+ if (is_uri) {
+ //spin.set_value( settings.get_uri(key))
+ ;
+
+ } else {
+ spin.set_value( settings.get_int(key));
+ }
+ //spin.set_value(val);
   }

   public virtual void write_to_config()
   {
- settings.set_integer(key, spin.get_value_as_int());
+ settings.set_int(key, spin.get_value_as_int());
   }

   bool handle_focus_out()

Thanks, Peter, that was not my best day I think.

Anyway, got it working with your hints. The value was not saved, because we didn't connect to the event value_changed yet. Additionally, the adjustment must be made somewhat different, because it always returns to zero upon leaving the focus. I'm currently initializing the Spin with a range.

But got it running, range is now between 0 and 1000MiB, using the Gtk.SpinButton will take steps of 25 (to avoid unnecessary clicking). If volsize is set to 0, the defaults that are currently active will be used.

The Async setting will just append the --asynchronous-upload option to duplicity. You can find the code in my repo: lp:~beowulfof/+junk/deja-dup

Currently I cannot backup, by the way, I filed a bug for that: https://bugs.launchpad.net/ubuntu/+source/deja-dup/+bug/1453121

Ok, forget that last thing, somehow I messed up my old password and it won't start a backup, if the password differs from that one used in the former backup.

Orr... the settings are there and work, the GUI lets you set the settings. but the backup runs without the new options, although in DuplicityJob the settings are to be recognized. Well, at least one step further.

So, update on that. The code in my repo does work, but it will only be applied, if you start a new backup, otherwise the last used options are used and volsize as well as asynchronous upload are ignored.

I currently hazle with the versioning thing for building a ppa package, see: https://askubuntu.com/questions/622692/how-to-correctly-version-a-modified-package

So, ok, the patch works, it makes the volsize changeable on backend basis and allows for choosing asynchronous upload.

The patch can be found here: https://bazaar.launchpad.net/~beowulfof/+junk/deja-dup/view/head:/debian/patches/0002-volsize.patch

@mterry, maybe you could have a look at it and include it?