Duplicity

Invalid data - SHA1 hash mishmatch during restore

Asked by Chris Bero

I'm excited to be using duplicity with some backups, but have run into some trouble and would like to know if it's recoverable.

Using Duplicity version 0.7.06 I backed up a 1.8TB set of files to a local NAS via an NFS mount. This took many attempts, and a week of tinkering to accomplish. I had to restart the process due to issues such as gpg-agent timing out the passphrase. Eventually the backup completed successfully, but I don't remember if it was fresh (all NAS-side files removed first) or had been resumed. This will be something to watch better next time.

I went on to immediately test the restore process by bringing the files back to a spare 2TB partition. Each volume is 100M.

# duplicity --asynchronous-upload --force --num-retries 10 -v9 file:///mnt/bkup /mnt/tmp/berocs

Which made it to volume 7485 of 13821.

Processed volume 7484 of 13821
Registering (mktemp) temporary file /tmp/duplicity-UKeud7-tempdir/mktemp-bdrFu_-7494
Deleting /tmp/duplicity-UKeud7-tempdir/mktemp-bdrFu_-7494
Forgetting temporary file /tmp/duplicity-UKeud7-tempdir/mktemp-bdrFu_-7494
Processed volume 7485 of 13821
Registering (mktemp) temporary file /tmp/duplicity-UKeud7-tempdir/mktemp-61HOHl-7495
Invalid data - SHA1 hash mismatch for file:
 duplicity-full.20160205T033010Z.vol7485.difftar.gpg
 Calculated hash: 9b905a77192131637bc6ae172a602f0c11896d73
 Manifest hash: 35ffeea04fe95d30c153d841fe445c4c8a87d30e

Releasing lockfile /root/.cache/duplicity/2dc20957bea0e51ed1d54f05c291ab69/lockfile.lock
Removing still remembered temporary file /tmp/duplicity-UKeud7-tempdir/mktemp-61HOHl-7495
Removing still remembered temporary file /tmp/duplicity-UKeud7-tempdir/mkstemp-B8zpTT-1

I'd rather not go volume by volume to recover partial files, as is suggested in other threads concerning this error. Instead, since I still have the original files, is there a way to repair the backup without having to delete and re-transfer all 2TB again?

Question information

Language:
English Edit question
Status:
Answered
For:
Duplicity Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Kenneth Loafman (kenneth-loafman) said : 		#1

"sha1sum duplicity-full.20160205T033010Z.vol7485.difftar.gpg" will give you
the correct hash. Save the manifest file and edit the original to insert
the correct hash. That should get you past this problem. Next time
consider breaking up the backup into more reasonable sized chunks. It'll
be much more reliable that way.

On Mon, Feb 8, 2016 at 10:02 AM, Chris Bero <
<email address hidden>> wrote:

> New question #284779 on Duplicity:
> https://answers.launchpad.net/duplicity/+question/284779
>
> I'm excited to be using duplicity with some backups, but have run into
> some trouble and would like to know if it's recoverable.
>
> Using Duplicity version 0.7.06 I backed up a 1.8TB set of files to a local
> NAS via an NFS mount. This took many attempts, and a week of tinkering to
> accomplish. I had to restart the process due to issues such as gpg-agent
> timing out the passphrase. Eventually the backup completed successfully,
> but I don't remember if it was fresh (all NAS-side files removed first) or
> had been resumed. This will be something to watch better next time.
>
> I went on to immediately test the restore process by bringing the files
> back to a spare 2TB partition. Each volume is 100M.
>
> # duplicity --asynchronous-upload --force --num-retries 10 -v9
> file:///mnt/bkup /mnt/tmp/berocs
>
> Which made it to volume 7485 of 13821.
>
> Processed volume 7484 of 13821
> Registering (mktemp) temporary file
> /tmp/duplicity-UKeud7-tempdir/mktemp-bdrFu_-7494
> Deleting /tmp/duplicity-UKeud7-tempdir/mktemp-bdrFu_-7494
> Forgetting temporary file /tmp/duplicity-UKeud7-tempdir/mktemp-bdrFu_-7494
> Processed volume 7485 of 13821
> Registering (mktemp) temporary file
> /tmp/duplicity-UKeud7-tempdir/mktemp-61HOHl-7495
> Invalid data - SHA1 hash mismatch for file:
> duplicity-full.20160205T033010Z.vol7485.difftar.gpg
> Calculated hash: 9b905a77192131637bc6ae172a602f0c11896d73
> Manifest hash: 35ffeea04fe95d30c153d841fe445c4c8a87d30e
>
> Releasing lockfile
> /root/.cache/duplicity/2dc20957bea0e51ed1d54f05c291ab69/lockfile.lock
> Removing still remembered temporary file
> /tmp/duplicity-UKeud7-tempdir/mktemp-61HOHl-7495
> Removing still remembered temporary file
> /tmp/duplicity-UKeud7-tempdir/mkstemp-B8zpTT-1
>
> I'd rather not go volume by volume to recover partial files, as is
> suggested in other threads concerning this error. Instead, since I still
> have the original files, is there a way to repair the backup without having
> to delete and re-transfer all 2TB again?
>
> --
> You received this question notification because your team duplicity-team
> is an answer contact for Duplicity.
>
> _______________________________________________
> Mailing list: https://launchpad.net/~duplicity-team
> Post to : <email address hidden>
> Unsubscribe : https://launchpad.net/~duplicity-team
> More help : https://help.launchpad.net/ListHelp
>

Revision history for this message
edso (ed.so) said : 		#2

hey Ken,

correct in this case meaning, correct for the file actually on the backend. if it was uploaded incompletely, gpg will still choke and duplicity will still be unable to restore anything from that volume, right?

..ede/duply.net

On 08.02.2016 22:46, Kenneth Loafman wrote:
> Question #284779 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/284779
>
> Status: Open => Answered
>
> Kenneth Loafman proposed the following answer:
> "sha1sum duplicity-full.20160205T033010Z.vol7485.difftar.gpg" will give you
> the correct hash. Save the manifest file and edit the original to insert
> the correct hash. That should get you past this problem. Next time
> consider breaking up the backup into more reasonable sized chunks. It'll
> be much more reliable that way.
>
>
>
> On Mon, Feb 8, 2016 at 10:02 AM, Chris Bero <
> <email address hidden>> wrote:
>
>> New question #284779 on Duplicity:
>> https://answers.launchpad.net/duplicity/+question/284779
>>
>> I'm excited to be using duplicity with some backups, but have run into
>> some trouble and would like to know if it's recoverable.
>>
>> Using Duplicity version 0.7.06 I backed up a 1.8TB set of files to a local
>> NAS via an NFS mount. This took many attempts, and a week of tinkering to
>> accomplish. I had to restart the process due to issues such as gpg-agent
>> timing out the passphrase. Eventually the backup completed successfully,
>> but I don't remember if it was fresh (all NAS-side files removed first) or
>> had been resumed. This will be something to watch better next time.
>>
>> I went on to immediately test the restore process by bringing the files
>> back to a spare 2TB partition. Each volume is 100M.
>>
>> # duplicity --asynchronous-upload --force --num-retries 10 -v9
>> file:///mnt/bkup /mnt/tmp/berocs
>>
>> Which made it to volume 7485 of 13821.
>>
>> Processed volume 7484 of 13821
>> Registering (mktemp) temporary file
>> /tmp/duplicity-UKeud7-tempdir/mktemp-bdrFu_-7494
>> Deleting /tmp/duplicity-UKeud7-tempdir/mktemp-bdrFu_-7494
>> Forgetting temporary file /tmp/duplicity-UKeud7-tempdir/mktemp-bdrFu_-7494
>> Processed volume 7485 of 13821
>> Registering (mktemp) temporary file
>> /tmp/duplicity-UKeud7-tempdir/mktemp-61HOHl-7495
>> Invalid data - SHA1 hash mismatch for file:
>> duplicity-full.20160205T033010Z.vol7485.difftar.gpg
>> Calculated hash: 9b905a77192131637bc6ae172a602f0c11896d73
>> Manifest hash: 35ffeea04fe95d30c153d841fe445c4c8a87d30e
>>
>> Releasing lockfile
>> /root/.cache/duplicity/2dc20957bea0e51ed1d54f05c291ab69/lockfile.lock
>> Removing still remembered temporary file
>> /tmp/duplicity-UKeud7-tempdir/mktemp-61HOHl-7495
>> Removing still remembered temporary file
>> /tmp/duplicity-UKeud7-tempdir/mkstemp-B8zpTT-1
>>
>> I'd rather not go volume by volume to recover partial files, as is
>> suggested in other threads concerning this error. Instead, since I still
>> have the original files, is there a way to repair the backup without having
>> to delete and re-transfer all 2TB again?
>>
>> --
>> You received this question notification because your team duplicity-team
>> is an answer contact for Duplicity.
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~duplicity-team
>> Post to : <email address hidden>
>> Unsubscribe : https://launchpad.net/~duplicity-team
>> More help : https://help.launchpad.net/ListHelp
>>
>

Revision history for this message
Kenneth Loafman (kenneth-loafman) said : 		#3

Absolutely. All volume sizes should be roughly the same with the last
volume possibly smaller.

He did mention he had a bunch of restarts, so all bets are off.

On Tue, Feb 9, 2016 at 4:22 AM, edso <email address hidden>
wrote:

> Question #284779 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/284779
>
> edso proposed the following answer:
> hey Ken,
>
> correct in this case meaning, correct for the file actually on the
> backend. if it was uploaded incompletely, gpg will still choke and
> duplicity will still be unable to restore anything from that volume,
> right?
>
> ..ede/duply.net
>
> On 08.02.2016 22:46, Kenneth Loafman wrote:
> > Question #284779 on Duplicity changed:
> > https://answers.launchpad.net/duplicity/+question/284779
> >
> > Status: Open => Answered
> >
> > Kenneth Loafman proposed the following answer:
> > "sha1sum duplicity-full.20160205T033010Z.vol7485.difftar.gpg" will give
> you
> > the correct hash. Save the manifest file and edit the original to insert
> > the correct hash. That should get you past this problem. Next time
> > consider breaking up the backup into more reasonable sized chunks. It'll
> > be much more reliable that way.
> >
> >
> >
> > On Mon, Feb 8, 2016 at 10:02 AM, Chris Bero <
> > <email address hidden>> wrote:
> >
> >> New question #284779 on Duplicity:
> >> https://answers.launchpad.net/duplicity/+question/284779
> >>
> >> I'm excited to be using duplicity with some backups, but have run into
> >> some trouble and would like to know if it's recoverable.
> >>
> >> Using Duplicity version 0.7.06 I backed up a 1.8TB set of files to a
> local
> >> NAS via an NFS mount. This took many attempts, and a week of tinkering
> to
> >> accomplish. I had to restart the process due to issues such as gpg-agent
> >> timing out the passphrase. Eventually the backup completed successfully,
> >> but I don't remember if it was fresh (all NAS-side files removed first)
> or
> >> had been resumed. This will be something to watch better next time.
> >>
> >> I went on to immediately test the restore process by bringing the files
> >> back to a spare 2TB partition. Each volume is 100M.
> >>
> >> # duplicity --asynchronous-upload --force --num-retries 10 -v9
> >> file:///mnt/bkup /mnt/tmp/berocs
> >>
> >> Which made it to volume 7485 of 13821.
> >>
> >> Processed volume 7484 of 13821
> >> Registering (mktemp) temporary file
> >> /tmp/duplicity-UKeud7-tempdir/mktemp-bdrFu_-7494
> >> Deleting /tmp/duplicity-UKeud7-tempdir/mktemp-bdrFu_-7494
> >> Forgetting temporary file
> /tmp/duplicity-UKeud7-tempdir/mktemp-bdrFu_-7494
> >> Processed volume 7485 of 13821
> >> Registering (mktemp) temporary file
> >> /tmp/duplicity-UKeud7-tempdir/mktemp-61HOHl-7495
> >> Invalid data - SHA1 hash mismatch for file:
> >> duplicity-full.20160205T033010Z.vol7485.difftar.gpg
> >> Calculated hash: 9b905a77192131637bc6ae172a602f0c11896d73
> >> Manifest hash: 35ffeea04fe95d30c153d841fe445c4c8a87d30e
> >>
> >> Releasing lockfile
> >> /root/.cache/duplicity/2dc20957bea0e51ed1d54f05c291ab69/lockfile.lock
> >> Removing still remembered temporary file
> >> /tmp/duplicity-UKeud7-tempdir/mktemp-61HOHl-7495
> >> Removing still remembered temporary file
> >> /tmp/duplicity-UKeud7-tempdir/mkstemp-B8zpTT-1
> >>
> >> I'd rather not go volume by volume to recover partial files, as is
> >> suggested in other threads concerning this error. Instead, since I still
> >> have the original files, is there a way to repair the backup without
> having
> >> to delete and re-transfer all 2TB again?
> >>
> >> --
> >> You received this question notification because your team duplicity-team
> >> is an answer contact for Duplicity.
> >>
> >> _______________________________________________
> >> Mailing list: https://launchpad.net/~duplicity-team
> >> Post to : <email address hidden>
> >> Unsubscribe : https://launchpad.net/~duplicity-team
> >> More help : https://help.launchpad.net/ListHelp
> >>
> >
>
> --
> You received this question notification because your team duplicity-team
> is an answer contact for Duplicity.
>
> _______________________________________________
> Mailing list: https://launchpad.net/~duplicity-team
> Post to : <email address hidden>
> Unsubscribe : https://launchpad.net/~duplicity-team
> More help : https://help.launchpad.net/ListHelp
>

Can you help with this problem?

Provide an answer of your own, or ask Chris Bero for more information if necessary.

To post a message you must log in.