Duplicity

single backedup file - restore problem

Asked by John

Some time ago I've configured duplicity to backup my files. what it does backup is single veracrypt container (around 27gbs). Splits it and sends to S3. Though, it turns out I have problem to restore it. Duply (or duplicity for that matter) seems to have a problem to merge these files into one single file. My output is just an empty folder with container's name.
Anyone had such problem before?
For now I have just downloaded from Amazon's S3 all files to my local disk and tried to restore it from local directory but it didn't work.
While backing up I didn't use gpg encryption (since vc's container is itself encrypted..).

I guess that worth mentioning is fact that it seemed to back it up earlier without problems. Both full and incr backups went just fine. All diff.tar files together have around the same size as backed-up container itself.
Command I'm using to restore:

    duplicity restore --no-encryption file:///home/johnniedoe/backup/ /home/johnniedoe/restored

And duply's config if it will be of any help.

    GPG_KEY='disabled'
    TARGET='s3://s3-eu-west-1.amazonaws.com/johnniedoe/backup'
    export AWS_ACCESS_KEY_ID='MYACCESSKEY'
    export AWS_SECRET_ACCESS_KEY='MYSECRETKEY'

    SOURCE='Private'

    PYTHON="python2"

    MAX_AGE=6M
    MAX_FULL_BACKUPS=3

    # verbosity of output (error 0, warning 1-2, notice 3-4, info 5-8, debug 9)
    # default is 4, if not set
    VERBOSITY=9

And version:

    ➜ ~ duply --version
      duply version 1.11.3
      (http://duply.net)

      Using installed duplicity version 0.7.09, python 2.7.12, gpg 2.1.15 (Home: /home/johnniedoe/.gnupg), awk 'GNU Awk 4.1.4, API: 1.1 (GNU MPFR 3.1.5, GNU MP 6.1.1)', grep 'grep (GNU grep) 2.25', bash '4.3.46(1)-release (x86_64-unknown-linux-gnu)'.
    ➜ ~ uname -a
    Linux 1337 4.7.6-1-ARCH #1 SMP PREEMPT Fri Sep 30 19:28:42 CEST 2016 x86_64 GNU/Linux

Anyone seen this before?

Question information

Language:
English Edit question
Status:
Solved
For:
Duplicity Edit question
Assignee:
No assignee Edit question
Solved by:
Aaron Whitehouse
Solved:
Last query:
Last reply:
Revision history for this message
Launchpad Janitor (janitor) said : 		#1

This question was expired because it remained in the 'Open' state without activity for the last 15 days.

Revision history for this message
John (johnniedoe) said : 		#2

Unfortunately it isn't "expired" because I'm still having this problem..

-------- Original Message --------
Subject: Re: [Question #402919]: single backedup file - restore problem
Local Time: October 26, 2016 12:04 PM
UTC Time: October 26, 2016 10:04 AM
From: <email address hidden>
To: <email address hidden>

Your question #402919 on Duplicity changed:
https://answers.launchpad.net/duplicity/+question/402919

Status: Open => Expired

Launchpad Janitor expired the question:
This question was expired because it remained in the 'Open' state
without activity for the last 15 days.

--
If you're still having this problem, you can reopen your question either
by replying to this email or by going to the following page and
entering more information about your problem:
https://answers.launchpad.net/duplicity/+question/402919

You received this question notification because you asked the question.

Revision history for this message
edso (ed.so) said : 		#3

On 03.11.2016 19:58, John wrote:
> Question #402919 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/402919
>
> Status: Expired => Open
>
> John is still having a problem:
> Unfortunately it isn't "expired" because I'm still having this problem..
>

can you run the restore command again w/ max verbosity and redirect the whole output (STDERR/OUT) into a file and pastebin that? check it beforehand for infos you deem private and obfuscate those.

also please run a
  'duplicity list-current-files ...'
and
  'duplicity collection-status ...'
and send these outputs as well.

..ede/duply.net

Revision history for this message
John (johnniedoe) said : 		#4

collection-status - http://pastebin.com/aT2m5HdY
restore - http://pastebin.com/1Vsh2BKF
list-files - http://pastebin.com/C7Mk808s
Am I to understand, that even though all volumes together are 27gb large (equals container's size), they don't actually contain any data? : /

Revision history for this message
Launchpad Janitor (janitor) said : 		#5

This question was expired because it remained in the 'Open' state without activity for the last 15 days.

Revision history for this message
John (johnniedoe) said : 		#6

I'm updating because it expired but I'm still having this problem.

-------- Original Message --------
Subject: Re: [Question #402919]: single backedup file - restore problem
Local Time: December 12, 2016 10:33 AM
UTC Time: December 12, 2016 9:33 AM
From: <email address hidden>
To: <email address hidden>

Your question #402919 on Duplicity changed:
https://answers.launchpad.net/duplicity/+question/402919

Status: Open => Expired

Launchpad Janitor expired the question:
This question was expired because it remained in the 'Open' state
without activity for the last 15 days.

--
If you're still having this problem, you can reopen your question either
by replying to this email or by going to the following page and
entering more information about your problem:
https://answers.launchpad.net/duplicity/+question/402919

You received this question notification because you asked the question.

Revision history for this message
Best Aaron Whitehouse (aaron-whitehouse) said : 		#7

John,

Very sorry to hear that you have had this problem. I have not seen it before.

I am wondering if the file could have been deleted before the most recent duply/duplicity backup. If that is the case then it would make sense that it is restoring an empty directory.

I would try restoring the initial backup with option:
-t 2016-06-10
and see if anything restores. If it does, you can try later dates until you find the last backed up version.

Revision history for this message
John (johnniedoe) said : 		#8

Thanks Aaron Whitehouse, that solved my question.

Revision history for this message
John (johnniedoe) said : 		#9

Hey Aaron,
Thank you, that was a brilliant idea. I just never thought about tying that.
As for removing a destination file before backup - I doubt it. I was doing backups manually because this vc container was rarely being changed.
Nevertheless, I've got access to this data now so thank you again.

Revision history for this message
Aaron Whitehouse (aaron-whitehouse) said : 		#10

No problem at all. Glad that you recovered most of your data.

Obviously the latest version should have restored correctly unless you deleted the source file in the meantime. It would be good to get to the bottom of this if it was a problem with duplicity, but it will likely be difficult to debug unless it happens again.

I would *strongly* recommend running frequent verify checks on duplicity backups. It gives a much greater level of confidence that the backup is working correctly by checking that what duplicity thinks should be in the archives (the index and hash values) is restorable and matches what is in the archives. Combine this with “list-current-files” to ensure that what you expect is being backed up is in the index/signature files.

The summary at the end of a backup operation also gives some useful information about the number and size of files that have been backed up. I always check these after I have done a backup to ensure that they are broadly what I would expect.

If any of these checks are not giving the results that you expect, please let us know, as we can help you debug the issue and that is normally far easier than restoring a backup that is not correct.

Kind regards,

Aaron