duplicity: Invalid data - SHA1 hash mismatch for file

On 03.04.2015 18:16, Domoradov Alex wrote:
> New question #264557 on Duplicity:
> https://answers.launchpad.net/duplicity/+question/264557
>
> Hello,
>
> I'm using the following system
>
> OS: CentOS-6, x64. duplicity-0.6.22-3.el6.x86_64. Today I have encountered with issue during restore process
>
> # duplicity -t 3D --force restore file:///backup/vhosts/ /home/restore/
>
> Last full backup date: Sun Mar 22 22:07:29 2015
> Invalid data - SHA1 hash mismatch for file:
> duplicity-full.20150322T220729Z.vol16.difftar.gz
> Calculated hash: f225519ed7c773b418cd1697e3010e343bb79fad
> Manifest hash: d2b34a36ff6c80c5dbbc9aaf27d927439f86bcd5
>
>
> the file looks correct
>
> # gzip -v -t duplicity-full.20150322T220729Z.vol16.difftar.gz
> duplicity-full.20150322T220729Z.vol16.difftar.gz: OK
>
> Could anyone point me in the right direction?
>

duplicity thinks it is corrupt. that's what it saves the checksums for.

first update to latest stable 0.6.25 to make sure you have the latest greatest most bugfixed version.

then create a new full and remember that your backup after the timestamp given in the filename is probably broken in case you have to restore something from your old backup. you can give the parameter --ignore-errors to have the restoration run through but read it's description on the manpage
 http://duplicity.nongnu.org/duplicity.1.html

finally, it's possible while unlikely that you ran into a bug there. try to run a verify with --ignore-errors and check the output as described on the manpage doc for the parameter.

..ede/duply.net

> duplicity thinks it is corrupt. that's what it saves the checksums for.
May be I need to add some flags to my ssh transport?

P.S.
Can I verify integrity of all archives on the destination host?

On 07.04.2015 11:21, Domoradov Alex wrote:
> Question #264557 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/264557
>
> Domoradov Alex posted a new comment:
>> duplicity thinks it is corrupt. that's what it saves the checksums for.
> May be I need to add some flags to my ssh transport?

the difference can have many causes, ranging from duplicity bugs to storage inconsistencies on the backend.

> P.S.
> Can I verify integrity of all archives on the destination host?
>

you want to circumvent downloading the whole backup i guess?

if you can run duplicity there, yes, kind of. the 'verify' action is a bastard of verify and compare against source. as you do not have the source on the backend, you might rather go with a full restore using the ignore error switch and redirect the whole cmd line output eg. '1>log.txt 2>&1' to a log file to sift for errors later.

..ede/duply.net

> storage inconsistencies on the backend.
it's very strange, because the archive is fine

# gzip -v -t duplicity-full.20150322T220729Z.vol16.difftar.gz
duplicity-full.20150322T220729Z.vol16.difftar.gz: OK

That does not mean the enclosed tar file is OK though. To test that, run:
$ tar tzvf duplicity-full.20150322T220729Z.vol16.difftar.gz

If you run:
$ sha1sum duplicity-full.20150322T220729Z.vol16.difftar.gz
do you get the value in the manifest or the value calculated in the error
message?

Have you tried downloading the file again?

On Wed, Apr 8, 2015 at 8:26 AM, Domoradov Alex <
<email address hidden>> wrote:

> Question #264557 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/264557
>
> Domoradov Alex posted a new comment:
> > storage inconsistencies on the backend.
> it's very strange, because the archive is fine
>
> # gzip -v -t duplicity-full.20150322T220729Z.vol16.difftar.gz
> duplicity-full.20150322T220729Z.vol16.difftar.gz: OK
>
> --
> You received this question notification because you are a member of
> duplicity-team, which is an answer contact for Duplicity.
>
> _______________________________________________
> Mailing list: https://launchpad.net/~duplicity-team
> Post to : <email address hidden>
> Unsubscribe : https://launchpad.net/~duplicity-team
> More help : https://help.launchpad.net/ListHelp
>

> $ tar tzvf duplicity-full.20150322T220729Z.vol16.difftar.gz
completed successfully without any errors

> do you get the value in the manifest or the value calculated in the error message?
I got the value calculated in the error message

> first update to latest stable 0.6.25 to make sure you have the latest greatest most bugfixed version.
there is some problems with librsync-1.0+. I can't build it

gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python2.6 -c duplicity/_librsyncmodule.c -o build/temp.linux-x86_64-2.6/duplicity/_librsyncmodule.o
duplicity/_librsyncmodule.c: In function '_librsync_new_sigmaker':
duplicity/_librsyncmodule.c:71: error: 'RS_DEFAULT_STRONG_LEN' undeclared (first use in this function)
duplicity/_librsyncmodule.c:71: error: (Each undeclared identifier is reported only once
duplicity/_librsyncmodule.c:71: error: for each function it appears in.)
duplicity/_librsyncmodule.c:71: error: too few arguments to function 'rs_sig_begin'
error: command 'gcc' failed with exit status 1
error: Bad exit status from /var/tmp/rpm-tmp.V3BaQQ (%build)

RPM build errors:
    line 14: buildprereq is deprecated: BuildPrereq: python-devel >= 2.4, librsync-devel >= 0.9.6
    Bad exit status from /var/tmp/rpm-tmp.V3BaQQ (%build)

0.6.25 does not support the new librsync. You'll need to downgrade to
librsync 0.9.6, or upgrade to duplicity 0.7.02.

An easier alternative might be to pick up the new _librsyncmodule.c from
the 0.7 series at:
https://bazaar.launchpad.net/~duplicity-team/duplicity/0.7-series/view/head:/duplicity/_librsyncmodule.c
and replace 0.6.25's version with that one before running setup.py. It
will work with librsync 1+

On Wed, Apr 8, 2015 at 10:51 AM, Domoradov Alex <
<email address hidden>> wrote:

> Question #264557 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/264557
>
> Domoradov Alex posted a new comment:
> > $ tar tzvf duplicity-full.20150322T220729Z.vol16.difftar.gz
> completed successfully without any errors
>
> > do you get the value in the manifest or the value calculated in the
> error message?
> I got the value calculated in the error message
>
> > first update to latest stable 0.6.25 to make sure you have the latest
> greatest most bugfixed version.
> there is some problems with librsync-1.0+. I can't build it
>
> gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall
> -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector
> --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv
> -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
> -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic
> -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python2.6 -c
> duplicity/_librsyncmodule.c -o
> build/temp.linux-x86_64-2.6/duplicity/_librsyncmodule.o
> duplicity/_librsyncmodule.c: In function '_librsync_new_sigmaker':
> duplicity/_librsyncmodule.c:71: error: 'RS_DEFAULT_STRONG_LEN' undeclared
> (first use in this function)
> duplicity/_librsyncmodule.c:71: error: (Each undeclared identifier is
> reported only once
> duplicity/_librsyncmodule.c:71: error: for each function it appears in.)
> duplicity/_librsyncmodule.c:71: error: too few arguments to function
> 'rs_sig_begin'
> error: command 'gcc' failed with exit status 1
> error: Bad exit status from /var/tmp/rpm-tmp.V3BaQQ (%build)
>
> RPM build errors:
> line 14: buildprereq is deprecated: BuildPrereq: python-devel >= 2.4,
> librsync-devel >= 0.9.6
> Bad exit status from /var/tmp/rpm-tmp.V3BaQQ (%build)
>
> --
> You received this question notification because you are a member of
> duplicity-team, which is an answer contact for Duplicity.
>
> _______________________________________________
> Mailing list: https://launchpad.net/~duplicity-team
> Post to : <email address hidden>
> Unsubscribe : https://launchpad.net/~duplicity-team
> More help : https://help.launchpad.net/ListHelp
>

I have the same issue (SHA1 mismatch) with local backend. Now I'm testing
0.7 series

Do I need to notify EPEL maintainer of duplicity?

On Wed, Apr 8, 2015 at 7:16 PM, Kenneth Loafman <
<email address hidden>> wrote:

> Your question #264557 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/264557
>
> Kenneth Loafman proposed the following answer:
> 0.6.25 does not support the new librsync. You'll need to downgrade to
> librsync 0.9.6, or upgrade to duplicity 0.7.02.
>
> An easier alternative might be to pick up the new _librsyncmodule.c from
> the 0.7 series at:
>
> https://bazaar.launchpad.net/~duplicity-team/duplicity/0.7-series/view/head:/duplicity/_librsyncmodule.c
> and replace 0.6.25's version with that one before running setup.py. It
> will work with librsync 1+
>
> On Wed, Apr 8, 2015 at 10:51 AM, Domoradov Alex <
> <email address hidden>> wrote:
>
> > Question #264557 on Duplicity changed:
> > https://answers.launchpad.net/duplicity/+question/264557
> >
> > Domoradov Alex posted a new comment:
> > > $ tar tzvf duplicity-full.20150322T220729Z.vol16.difftar.gz
> > completed successfully without any errors
> >
> > > do you get the value in the manifest or the value calculated in the
> > error message?
> > I got the value calculated in the error message
> >
> > > first update to latest stable 0.6.25 to make sure you have the latest
> > greatest most bugfixed version.
> > there is some problems with librsync-1.0+. I can't build it
> >
> > gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall
> > -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector
> > --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv
> > -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
> > -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic
> > -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python2.6 -c
> > duplicity/_librsyncmodule.c -o
> > build/temp.linux-x86_64-2.6/duplicity/_librsyncmodule.o
> > duplicity/_librsyncmodule.c: In function '_librsync_new_sigmaker':
> > duplicity/_librsyncmodule.c:71: error: 'RS_DEFAULT_STRONG_LEN' undeclared
> > (first use in this function)
> > duplicity/_librsyncmodule.c:71: error: (Each undeclared identifier is
> > reported only once
> > duplicity/_librsyncmodule.c:71: error: for each function it appears in.)
> > duplicity/_librsyncmodule.c:71: error: too few arguments to function
> > 'rs_sig_begin'
> > error: command 'gcc' failed with exit status 1
> > error: Bad exit status from /var/tmp/rpm-tmp.V3BaQQ (%build)
> >
> > RPM build errors:
> > line 14: buildprereq is deprecated: BuildPrereq: python-devel >= 2.4,
> > librsync-devel >= 0.9.6
> > Bad exit status from /var/tmp/rpm-tmp.V3BaQQ (%build)
> >
> > --
> > You received this question notification because you are a member of
> > duplicity-team, which is an answer contact for Duplicity.
> >
> > _______________________________________________
> > Mailing list: https://launchpad.net/~duplicity-team
> > Post to : <email address hidden>
> > Unsubscribe : https://launchpad.net/~duplicity-team
> > More help : https://help.launchpad.net/ListHelp
> >
>
> --
> If this answers your question, please go to the following page to let us
> know that it is solved:
>
> https://answers.launchpad.net/duplicity/+question/264557/+confirm?answer_id=6
>
> If you still need help, you can reply to this email or go to the
> following page to enter your feedback:
> https://answers.launchpad.net/duplicity/+question/264557
>
> You received this question notification because you asked the question.
>

On 15.04.2015 09:12, Domoradov Alex wrote:
> Question #264557 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/264557
>
> Status: Answered => Open
>
> Domoradov Alex is still having a problem:
> I have the same issue (SHA1 mismatch) with local backend. Now I'm testing
> 0.7 series

what was your version that created the backup?

> Do I need to notify EPEL maintainer of duplicity?
>

if they promote anything older than 0.6.25 it'd make sense.. ede/duply.net

# cat /etc/redhat-release
CentOS release 6.6 (Final)

# yum info duplicity
Available Packages
Name : duplicity
Arch : x86_64
Version : 0.6.22
Release : 4.el6
Size : 533 k
Repo : epel
Summary : Encrypted bandwidth-efficient backup using rsync algorithm
URL : http://www.nongnu.org/duplicity/
License : GPLv2+

On Wed, Apr 15, 2015 at 12:07 PM, edso <<email address hidden>
> wrote:

> Your question #264557 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/264557
>
> Status: Open => Answered
>
> edso proposed the following answer:
> On 15.04.2015 09:12, Domoradov Alex wrote:
> > Question #264557 on Duplicity changed:
> > https://answers.launchpad.net/duplicity/+question/264557
> >
> > Status: Answered => Open
> >
> > Domoradov Alex is still having a problem:
> > I have the same issue (SHA1 mismatch) with local backend. Now I'm testing
> > 0.7 series
>
> what was your version that created the backup?
>
> > Do I need to notify EPEL maintainer of duplicity?
> >
>
> if they promote anything older than 0.6.25 it'd make sense..
> ede/duply.net
>
> --
> If this answers your question, please go to the following page to let us
> know that it is solved:
>
> https://answers.launchpad.net/duplicity/+question/264557/+confirm?answer_id=8
>
> If you still need help, you can reply to this email or go to the
> following page to enter your feedback:
> https://answers.launchpad.net/duplicity/+question/264557
>
> You received this question notification because you asked the question.
>

the corruption bug was fixed in 0.6.23
 http://duplicity.nongnu.org/CHANGELOG
"
* Merged in lp:~mterry/duplicity/disappearing-source
  - When restarting a backup, we may accidentally skip the first chunk of one of
    the source files. To reproduce this,:
    1) interrupt a backup
    2) delete the source file it was in the middle of
    3) restart the backup
  - When replaying the source iterator to find where to resume from, we can't
    notice that the file is gone until we've already iterated past where it
    would be!
  - The solution I came up with is to just let duplicity stuff the data we
    accidentally read back into the source iterator.
  - This is actually a data loss bug, because it's possible to back up
    corrupted files (that are missing their first chunk).
"

..ede/duply.net

On 15.04.2015 11:46, Domoradov Alex wrote:
> Question #264557 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/264557
>
> Status: Answered => Open
>
> Domoradov Alex is still having a problem:
> # cat /etc/redhat-release
> CentOS release 6.6 (Final)
>
> # yum info duplicity
> Available Packages
> Name : duplicity
> Arch : x86_64
> Version : 0.6.22
> Release : 4.el6
> Size : 533 k
> Repo : epel
> Summary : Encrypted bandwidth-efficient backup using rsync algorithm
> URL : http://www.nongnu.org/duplicity/
> License : GPLv2+
>
> On Wed, Apr 15, 2015 at 12:07 PM, edso <<email address hidden>
>> wrote:
>
>> Your question #264557 on Duplicity changed:
>> https://answers.launchpad.net/duplicity/+question/264557
>>
>> Status: Open => Answered
>>
>> edso proposed the following answer:
>> On 15.04.2015 09:12, Domoradov Alex wrote:
>>> Question #264557 on Duplicity changed:
>>> https://answers.launchpad.net/duplicity/+question/264557
>>>
>>> Status: Answered => Open
>>>
>>> Domoradov Alex is still having a problem:
>>> I have the same issue (SHA1 mismatch) with local backend. Now I'm testing
>>> 0.7 series
>>
>> what was your version that created the backup?
>>
>>> Do I need to notify EPEL maintainer of duplicity?
>>>
>>
>> if they promote anything older than 0.6.25 it'd make sense..
>> ede/duply.net
>>
>> --
>> If this answers your question, please go to the following page to let us
>> know that it is solved:
>>
>> https://answers.launchpad.net/duplicity/+question/264557/+confirm?answer_id=8
>>
>> If you still need help, you can reply to this email or go to the
>> following page to enter your feedback:
>> https://answers.launchpad.net/duplicity/+question/264557
>>
>> You received this question notification because you asked the question.
>>
>

Thanks for the info. So as I understood, all I need is to upgrade to 0.6.25?

As far as I remember 0.6.25 is not compatible with librsync-1.0.0 from the
EPEL. So what would be the best way:

1. Downgrade to librsync-0.9.x
2. Replace librsyncmodule.c from 0.7 series

I don't need any new features, all I need is - STABLE :)

Thanks in advance.

On Wed, Apr 15, 2015 at 12:56 PM, edso <<email address hidden>
> wrote:

> Your question #264557 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/264557
>
> Status: Open => Answered
>
> edso proposed the following answer:
> the corruption bug was fixed in 0.6.23
> http://duplicity.nongnu.org/CHANGELOG
> "
> * Merged in lp:~mterry/duplicity/disappearing-source
> - When restarting a backup, we may accidentally skip the first chunk of
> one of
> the source files. To reproduce this,:
> 1) interrupt a backup
> 2) delete the source file it was in the middle of
> 3) restart the backup
> - When replaying the source iterator to find where to resume from, we
> can't
> notice that the file is gone until we've already iterated past where it
> would be!
> - The solution I came up with is to just let duplicity stuff the data we
> accidentally read back into the source iterator.
> - This is actually a data loss bug, because it's possible to back up
> corrupted files (that are missing their first chunk).
> "
>
> ..ede/duply.net
>
> On 15.04.2015 11:46, Domoradov Alex wrote:
> > Question #264557 on Duplicity changed:
> > https://answers.launchpad.net/duplicity/+question/264557
> >
> > Status: Answered => Open
> >
> > Domoradov Alex is still having a problem:
> > # cat /etc/redhat-release
> > CentOS release 6.6 (Final)
> >
> > # yum info duplicity
> > Available Packages
> > Name : duplicity
> > Arch : x86_64
> > Version : 0.6.22
> > Release : 4.el6
> > Size : 533 k
> > Repo : epel
> > Summary : Encrypted bandwidth-efficient backup using rsync algorithm
> > URL : http://www.nongnu.org/duplicity/
> > License : GPLv2+
> >
> > On Wed, Apr 15, 2015 at 12:07 PM, edso <
> <email address hidden>
> >> wrote:
> >
> >> Your question #264557 on Duplicity changed:
> >> https://answers.launchpad.net/duplicity/+question/264557
> >>
> >> Status: Open => Answered
> >>
> >> edso proposed the following answer:
> >> On 15.04.2015 09:12, Domoradov Alex wrote:
> >>> Question #264557 on Duplicity changed:
> >>> https://answers.launchpad.net/duplicity/+question/264557
> >>>
> >>> Status: Answered => Open
> >>>
> >>> Domoradov Alex is still having a problem:
> >>> I have the same issue (SHA1 mismatch) with local backend. Now I'm
> testing
> >>> 0.7 series
> >>
> >> what was your version that created the backup?
> >>
> >>> Do I need to notify EPEL maintainer of duplicity?
> >>>
> >>
> >> if they promote anything older than 0.6.25 it'd make sense..
> >> ede/duply.net
> >>
> >> --
> >> If this answers your question, please go to the following page to let us
> >> know that it is solved:
> >>
> >>
> https://answers.launchpad.net/duplicity/+question/264557/+confirm?answer_id=8
> >>
> >> If you still need help, you can reply to this email or go to the
> >> following page to enter your feedback:
> >> https://answers.launchpad.net/duplicity/+question/264557
> >>
> >> You received this question notification because you asked the question.
> >>
> >
>
> --
> If this answers your question, please go to the following page to let us
> know that it is solved:
>
> https://answers.launchpad.net/duplicity/+question/264557/+confirm?answer_id=10
>
> If you still need help, you can reply to this email or go to the
> following page to enter your feedback:
> https://answers.launchpad.net/duplicity/+question/264557
>
> You received this question notification because you asked the question.
>

On 15.04.2015 12:26, Domoradov Alex wrote:
> Question #264557 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/264557
>
> Status: Answered => Open
>
> Domoradov Alex is still having a problem:
> Thanks for the info. So as I understood, all I need is to upgrade to
> 0.6.25?

yes.. your backup is and will stay broken though. the buggy version wrote corrupt vol. files on resumes. this can be detected via verify runs btw.

> As far as I remember 0.6.25 is not compatible with librsync-1.0.0 from the
> EPEL. So what would be the best way:
>
> 1. Downgrade to librsync-0.9.x
> 2. Replace librsyncmodule.c from 0.7 series

both ways work

> I don't need any new features, all I need is - STABLE :)
>

we are too few to _guarantee_ anything in this regard. simply make sure to
- retest all your whole backup plan routines
when switching versions. and generally to
- verify regularly
.

duplicity is quite stable and heavily used, but that is in no way reflected on the developer side unfortunately.

..ede/duply.net

Thanks, now I'm testing 0.6.25 with _librsyncmodule.c from 0.7 series.

> verify regularly
One more thing. If I restored backup to some specific date, for .e.g

# duplicity -t "2015-04-15T08:00:01" --no-encryption restore file:///backup/vhosts/ /restore/vhosts/

And now I want to get the most recent version of files (Thu Apr 16 08:05:01 2015), according to output of the collection-status
...
...
...
         Incremental Thu Apr 16 06:05:01 2015 1
         Incremental Thu Apr 16 07:05:02 2015 1
         Incremental Thu Apr 16 08:05:01 2015 1
-------------------------
No orphaned or incomplete backup sets found.

Do I need to restore the whole folder vhosts or it is possible just to restore files which has been changed since 2015-04-15T08:00:01 up to 2015-04-16T08:05:01 ?

On 16.04.2015 11:36, Domoradov Alex wrote:
> Question #264557 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/264557
>
> Domoradov Alex posted a new comment:
> Thanks, now I'm testing 0.6.25 with _librsyncmodule.c from 0.7 series.
>
>> verify regularly
> One more thing. If I restored backup to some specific date, for .e.g
>
> # duplicity -t "2015-04-15T08:00:01" --no-encryption restore
> file:///backup/vhosts/ /restore/vhosts/
>
> And now I want to get the most recent version of files (Thu Apr 16 08:05:01 2015), according to output of the collection-status
> ...
> ...
> ...
> Incremental Thu Apr 16 06:05:01 2015 1
> Incremental Thu Apr 16 07:05:02 2015 1
> Incremental Thu Apr 16 08:05:01 2015 1
> -------------------------
> No orphaned or incomplete backup sets found.
>
> Do I need to restore the whole folder vhosts or it is possible just to
> restore files which has been changed since 2015-04-15T08:00:01 up to
> 2015-04-16T08:05:01 ?
>

you can pick files/folders via '--file-to-restore', see manpage
 http://duplicity.nongnu.org/duplicity.1.html

be aware, as a design choice you cannot overwrite anything with duplicity. always restore to a temp location and consciously copy/move to target from there.

no, you cannot select only changed files, you can select a point in time for the selection you want to restore however.

..ede/duply.net

It's very uncomfortable. I want just to check that all incremental backups
are valid and I can restore it. My full backup ~50Gb, incremental
~100Mb-500Mb per hour. So to check additional 100-500Mb every time I would
need to restore the whole full + all increments?!

full+Incremental Thu Apr 16 06:05:01 2015
full+Incremental Thu Apr 16 06:05:01 2015 + Thu Apr 16 07:05:02 2015
full+Incremental Thu Apr 16 06:05:01 2015 + Thu Apr 16 07:05:02 2015 + Thu
Apr 16 08:05:01 2015

and so on

On Thu, Apr 16, 2015 at 12:51 PM, edso <<email address hidden>
> wrote:

> Your question #264557 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/264557
>
> edso proposed the following answer:
> On 16.04.2015 11:36, Domoradov Alex wrote:
> > Question #264557 on Duplicity changed:
> > https://answers.launchpad.net/duplicity/+question/264557
> >
> > Domoradov Alex posted a new comment:
> > Thanks, now I'm testing 0.6.25 with _librsyncmodule.c from 0.7 series.
> >
> >> verify regularly
> > One more thing. If I restored backup to some specific date, for .e.g
> >
> > # duplicity -t "2015-04-15T08:00:01" --no-encryption restore
> > file:///backup/vhosts/ /restore/vhosts/
> >
> > And now I want to get the most recent version of files (Thu Apr 16
> 08:05:01 2015), according to output of the collection-status
> > ...
> > ...
> > ...
> > Incremental Thu Apr 16 06:05:01 2015 1
> > Incremental Thu Apr 16 07:05:02 2015 1
> > Incremental Thu Apr 16 08:05:01 2015 1
> > -------------------------
> > No orphaned or incomplete backup sets found.
> >
> > Do I need to restore the whole folder vhosts or it is possible just to
> > restore files which has been changed since 2015-04-15T08:00:01 up to
> > 2015-04-16T08:05:01 ?
> >
>
> you can pick files/folders via '--file-to-restore', see manpage
> http://duplicity.nongnu.org/duplicity.1.html
>
> be aware, as a design choice you cannot overwrite anything with
> duplicity. always restore to a temp location and consciously copy/move
> to target from there.
>
> no, you cannot select only changed files, you can select a point in time
> for the selection you want to restore however.
>
> ..ede/duply.net
>
> --
> If this answers your question, please go to the following page to let us
> know that it is solved:
>
> https://answers.launchpad.net/duplicity/+question/264557/+confirm?answer_id=14
>
> If you still need help, you can reply to this email or go to the
> following page to enter your feedback:
> https://answers.launchpad.net/duplicity/+question/264557
>
> You received this question notification because you asked the question.
>

well, yes that's the disadvantage of a remote repository. how about backing up to a local file:// repository, verifying that and keeping it in sync with remote?

alternatively you might wanna verify only from week to week or such, as much as your budget allows.

..ede/duply.net

On 16.04.2015 12:16, Domoradov Alex wrote:
> Question #264557 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/264557
>
> Status: Answered => Open
>
> Domoradov Alex is still having a problem:
> It's very uncomfortable. I want just to check that all incremental backups
> are valid and I can restore it. My full backup ~50Gb, incremental
> ~100Mb-500Mb per hour. So to check additional 100-500Mb every time I would
> need to restore the whole full + all increments?!
>
> full+Incremental Thu Apr 16 06:05:01 2015
> full+Incremental Thu Apr 16 06:05:01 2015 + Thu Apr 16 07:05:02 2015
> full+Incremental Thu Apr 16 06:05:01 2015 + Thu Apr 16 07:05:02 2015 + Thu
> Apr 16 08:05:01 2015
>
> and so on
>
> On Thu, Apr 16, 2015 at 12:51 PM, edso <<email address hidden>
>> wrote:
>
>> Your question #264557 on Duplicity changed:
>> https://answers.launchpad.net/duplicity/+question/264557
>>
>> edso proposed the following answer:
>> On 16.04.2015 11:36, Domoradov Alex wrote:
>>> Question #264557 on Duplicity changed:
>>> https://answers.launchpad.net/duplicity/+question/264557
>>>
>>> Domoradov Alex posted a new comment:
>>> Thanks, now I'm testing 0.6.25 with _librsyncmodule.c from 0.7 series.
>>>
>>>> verify regularly
>>> One more thing. If I restored backup to some specific date, for .e.g
>>>
>>> # duplicity -t "2015-04-15T08:00:01" --no-encryption restore
>>> file:///backup/vhosts/ /restore/vhosts/
>>>
>>> And now I want to get the most recent version of files (Thu Apr 16
>> 08:05:01 2015), according to output of the collection-status
>>> ...
>>> ...
>>> ...
>>> Incremental Thu Apr 16 06:05:01 2015 1
>>> Incremental Thu Apr 16 07:05:02 2015 1
>>> Incremental Thu Apr 16 08:05:01 2015 1
>>> -------------------------
>>> No orphaned or incomplete backup sets found.
>>>
>>> Do I need to restore the whole folder vhosts or it is possible just to
>>> restore files which has been changed since 2015-04-15T08:00:01 up to
>>> 2015-04-16T08:05:01 ?
>>>
>>
>> you can pick files/folders via '--file-to-restore', see manpage
>> http://duplicity.nongnu.org/duplicity.1.html
>>
>> be aware, as a design choice you cannot overwrite anything with
>> duplicity. always restore to a temp location and consciously copy/move
>> to target from there.
>>
>> no, you cannot select only changed files, you can select a point in time
>> for the selection you want to restore however.
>>
>> ..ede/duply.net
>>
>> --
>> If this answers your question, please go to the following page to let us
>> know that it is solved:
>>
>> https://answers.launchpad.net/duplicity/+question/264557/+confirm?answer_id=14
>>
>> If you still need help, you can reply to this email or go to the
>> following page to enter your feedback:
>> https://answers.launchpad.net/duplicity/+question/264557
>>
>> You received this question notification because you asked the question.
>>
>

> how about backing up to a local file:// repository
at the moment I'm doing exactly what you have said

On the web server

# duplicity --no-encryption --volsize 1024 incr /data/vhosts/
file:///backup/vhosts/

On the backup server

# /usr/bin/rsync -rza --stats --rsh='ssh -p2222 -c arcfour' --inplace
root@148.251.xxx.xxx:/backup/vhosts/ /data/backup/148.251.xxx.xxx/vhosts/

> verifying that and keeping it in sync with remote?
could you explain more detailed how I can verify a single incremental
backup?

On Thu, Apr 16, 2015 at 10:06 PM, edso <<email address hidden>
> wrote:

> Your question #264557 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/264557
>
> Status: Open => Answered
>
> edso proposed the following answer:
> well, yes that's the disadvantage of a remote repository. how about
> backing up to a local file:// repository, verifying that and keeping it
> in sync with remote?
>
> alternatively you might wanna verify only from week to week or such, as
> much as your budget allows.
>
> ..ede/duply.net
>
> On 16.04.2015 12:16, Domoradov Alex wrote:
> > Question #264557 on Duplicity changed:
> > https://answers.launchpad.net/duplicity/+question/264557
> >
> > Status: Answered => Open
> >
> > Domoradov Alex is still having a problem:
> > It's very uncomfortable. I want just to check that all incremental
> backups
> > are valid and I can restore it. My full backup ~50Gb, incremental
> > ~100Mb-500Mb per hour. So to check additional 100-500Mb every time I
> would
> > need to restore the whole full + all increments?!
> >
> > full+Incremental Thu Apr 16 06:05:01 2015
> > full+Incremental Thu Apr 16 06:05:01 2015 + Thu Apr 16 07:05:02 2015
> > full+Incremental Thu Apr 16 06:05:01 2015 + Thu Apr 16 07:05:02 2015 +
> Thu
> > Apr 16 08:05:01 2015
> >
> > and so on
> >
> > On Thu, Apr 16, 2015 at 12:51 PM, edso <
> <email address hidden>
> >> wrote:
> >
> >> Your question #264557 on Duplicity changed:
> >> https://answers.launchpad.net/duplicity/+question/264557
> >>
> >> edso proposed the following answer:
> >> On 16.04.2015 11:36, Domoradov Alex wrote:
> >>> Question #264557 on Duplicity changed:
> >>> https://answers.launchpad.net/duplicity/+question/264557
> >>>
> >>> Domoradov Alex posted a new comment:
> >>> Thanks, now I'm testing 0.6.25 with _librsyncmodule.c from 0.7 series.
> >>>
> >>>> verify regularly
> >>> One more thing. If I restored backup to some specific date, for .e.g
> >>>
> >>> # duplicity -t "2015-04-15T08:00:01" --no-encryption restore
> >>> file:///backup/vhosts/ /restore/vhosts/
> >>>
> >>> And now I want to get the most recent version of files (Thu Apr 16
> >> 08:05:01 2015), according to output of the collection-status
> >>> ...
> >>> ...
> >>> ...
> >>> Incremental Thu Apr 16 06:05:01 2015 1
> >>> Incremental Thu Apr 16 07:05:02 2015 1
> >>> Incremental Thu Apr 16 08:05:01 2015 1
> >>> -------------------------
> >>> No orphaned or incomplete backup sets found.
> >>>
> >>> Do I need to restore the whole folder vhosts or it is possible just to
> >>> restore files which has been changed since 2015-04-15T08:00:01 up to
> >>> 2015-04-16T08:05:01 ?
> >>>
> >>
> >> you can pick files/folders via '--file-to-restore', see manpage
> >> http://duplicity.nongnu.org/duplicity.1.html
> >>
> >> be aware, as a design choice you cannot overwrite anything with
> >> duplicity. always restore to a temp location and consciously copy/move
> >> to target from there.
> >>
> >> no, you cannot select only changed files, you can select a point in time
> >> for the selection you want to restore however.
> >>
> >> ..ede/duply.net
> >>
> >> --
> >> If this answers your question, please go to the following page to let us
> >> know that it is solved:
> >>
> >>
> https://answers.launchpad.net/duplicity/+question/264557/+confirm?answer_id=14
> >>
> >> If you still need help, you can reply to this email or go to the
> >> following page to enter your feedback:
> >> https://answers.launchpad.net/duplicity/+question/264557
> >>
> >> You received this question notification because you asked the question.
> >>
> >
>
> --
> If this answers your question, please go to the following page to let us
> know that it is solved:
>
> https://answers.launchpad.net/duplicity/+question/264557/+confirm?answer_id=16
>
> If you still need help, you can reply to this email or go to the
> following page to enter your feedback:
> https://answers.launchpad.net/duplicity/+question/264557
>
> You received this question notification because you asked the question.
>

Any updates?

>> verifying that and keeping it in sync with remote?
>could you explain more detailed how I can verify a single incremental backup?

verifying is essentially a restore without keeping the restored files. so you will always restore the full up to the incremental you wanted to test. this is how duplicity works. restore the file from the full, get the files changes from next incr, apply changes via librsync, repeat with next incr etc.

..ede/duply.net

I see, thanks for the info. I'm using 0.7 for a month without any problem.

One more question. I hope the last one. I have 3 full backups

Type of backup set: Time: Num volumes:
Full Tue Apr 14 20:06:06 2015 34
Full Sat Apr 18 13:28:14 2015 34
Full Sat Apr 25 19:22:15 2015 35

And I want to delete the old ones and keep only the last

# duplicity remove-all-but-n-full 1 --dry-run file:///data/backup/duplicity/
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Sat Apr 25 19:22:15 2015
Found old backup chains at the following times:
Sat Apr 18 13:05:01 2015
Sat Apr 25 19:05:01 2015
Rerun command with --force option to actually delete.

According to the man page - A value of 1 means that only the single most recent backup chain will be kept

So my question is - why duplicity want to delete the last available full backup? Did I miss something?