Déjà Dup

Is there a way to speed up retrieval of the most recent backups?

Asked by Daniel Krajnik

It would be great if there was a way to speed up restoration of the most recent backups at a cost of the older ones. Most of the time I'm using Deja Dup to get the most recent version of backups stored in the cloud (google drive), but it may take a few hours to restore it and a few minutes just to display its folders. Is there a strategy that could allow user to have an option to increase performance of just the most recent backups?

EDIT: after a whole day of restoring the backup it's still at barely 10%... I'm so confused rigth now how is this supposed to work. Any help would be grately appreciated. Is there a size limit? Should they be "rotated"? Anyone?

Question information

Language:
English Edit question
Status:
Answered
For:
Déjà Dup Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Michael Terry (mterry) said : 		#1

Hello! Yeah, I agree that Deja Dup can be slow. It's not something I've spent a whole lot of effort optimizing. I will say that I am experimenting with Restic support, which should be faster than the duplicity backend. But that's a way out from being fully stable (though you can opt-into it today).

But to actually address about your issue:
- There might be a sizable time cost up front as we download a bunch of initial metadata (you can see this local cache in ~/.deja-dup/cache). This might be optimizable, but I'm not sure how tweakable that is on the duplicity side, I haven't investigated.
- I don't think this would be any faster, but you can also download from Google Drive and restore from the now-local folder. But that would involve downloading all the data rather than just what you need. (Though... you would only need to download just the most recent backup chain and its incrementals.)
- One thing we've done in recent releases is change the default volume size to 200MB, from 25MB. I'm not 100% the performance implications of that, but if you are doing a full restore, it's probably faster to download a few larger files than a lot of small ones.

But ultimately, I don't have much positive news - I think we're a little slow these days. Hopefully it will get better in the future if Restic support pans out, but sorry.

Revision history for this message
Daniel Krajnik (dankray) said : 		#2

Thanks, I've started looking into restic as well. Could you please let me know how to recognize which files belong to the latest chain and it incrementals? I've tried downloading everything, as well, but it amounted to 40GB - if it was possible to select only the right portion of that, it would be already a big inprovement for me.

To be fair it may be also possible that Google is throttling my account, I will test it on AWS GLacier and B2.

Revision history for this message
Michael Terry (mterry) said : 		#3

> Could you please let me know how to recognize which files belong to the latest chain and it incrementals?

Duplicity files start with either "duplicity-full-" for full snapshots or "duplicity-inc-"/"duplicity-new-" for incremental snapshots. And each file also has a timestamp in its name. Find the most recent set of duplicity-full-* files and you only need those and newer files.

Can you help with this problem?

Provide an answer of your own, or ask Daniel Krajnik for more information if necessary.

To post a message you must log in.