SHA1 hash mismatch while recovering backup

On 22.03.2016 06:02, Jon Beyer wrote:
> New question #289074 on Duplicity:
> https://answers.launchpad.net/duplicity/+question/289074
>
> I am trying to recover a sizable backup that is stored in S3. There are 1,200 volumes, and the restore process has failed at various points in the restoration process with an SHA1 hash mismatch (i.e. one time it fails on volume 512, the next time on volume 692). If I manually download the problematic volume, the SHA1 hash is what it should be. That implies to me that the download from S3 is terminating prematurely and causing the invalid SHA1 hash.
>
> I have --num-retries set at 10, although I am working through duply, so I'm not 100% certain that that parameter is actually getting passed through to duplicity.
>
> Any suggestions as to how to overcome this issue?
>
> Thanks in advance!
>

hmmm, you can check if the parameter is propagated properly to duplicity w/ the --preview option for duply.

but my guess would be that the backend fails to recognize that the download failed (that's the part that would be retried) and simply returns the broken file to duplicity to work with. once it is there, there is no logic to say hey, maybe the download was faulty let's retry it currently (afaik).

what's your duplicity version?

..ede/duply.net

Ed,

Thanks for your assistance. The "--ignore-errors" flag is indeed getting passed through to duplicity. Full anonymized command to duplicity at the bottom. Duplicity version is 0.7.02.

It would seem that "--ignore-errors" does not ignore this type of error.

Because I know that the S3 volumes are intact, and are simply failing occasionally, is there a recommended way for me to download all of the S3 volumes, and then have duplicity pull them locally? If you could point me in the right direction as to how to achieve this, I would appreciate it immensely.

Thanks,
Jon

TMPDIR='/mnt/sdf/tmp' PASSPHRASE=0612sensations AWS_ACCESS_KEY_ID='AKIAIFOO' AWS_SECRET_ACCESS_KEY='dtqLKw4u' duplicity --archive-dir '/mnt/sdf/.cache' --name duply_backup --verbosity '9' --volsize 200 --ignore-errors --num-retries 10 --asynchronous-upload -t now 's3://s3.amazonaws.com/duplicity-backups-100mb/' '/mnt/sdf/backup/'

Trying to mark this as "I Still Need an Answer", it seems that it needs an additional message to do so.

On 22.03.2016 17:42, Jon Beyer wrote:
> Question #289074 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/289074
>
> Status: Answered => Solved
>
> Jon Beyer confirmed that the question is solved:
> Ed,
>
> Thanks for your assistance. The "--ignore-errors" flag is indeed
> getting passed through to duplicity. Full anonymized command to
> duplicity at the bottom. Duplicity version is 0.7.02.
>
> It would seem that "--ignore-errors" does not ignore this type of error.

yeah, it's only implemented for some errors so far

> Because I know that the S3 volumes are intact, and are simply failing
> occasionally, is there a recommended way for me to download all of the
> S3 volumes, and then have duplicity pull them locally? If you could
> point me in the right direction as to how to achieve this, I would
> appreciate it immensely.

not using s3 personally. i read on this list that the boto cmd line tools do the trick
  http://boto.cloudhackers.com/en/latest/commandline.html
.

download everything to a local folder and use file:// for restoration.

updating boto to the latest stable might help you with the issue, if possible.

..ede/duply.net