incremental adding unchanged files

first and foremost, please state your duplicity, python and os version.

secondly, you can run duplicity with '--dry-run' and maximum verbosity '-v9' and somewhere it should tell you why it thinks a specific file needs to be backed up again.

wrt. to the filecount:
i have no clue.. maybe it lists only files but counts folders as well? you may want investigate with a small test set and compare list-current-files with the actual restored content. usually duplicity does not backup more than necessary or forgets data. would be bad mo for a backup software anyway.

..ede/duply.net

Oops.
duplicity 0.6.19
Python 2.7.3
ubuntu server 12.04

With --dry-run it has no problem. I tried it with -v9 and again with -v4. As long as --dry-run is there it tries to only back up a few files that have changed. If I remove --dry-run it adds 1000s of files that haven't changed.

It's only on this large backup that this problem occurs. I have 3 smaller backups (~7GB) that have no problems. 2 to the same external drive and one to S3.

Keith

On 05.11.2012 00:11, Keith Cornwell wrote:
> Question #212772 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/212772
>
> Status: Needs information => Open
>
> Keith Cornwell gave more information on the question:
> Oops.
> duplicity 0.6.19
> Python 2.7.3
> ubuntu server 12.04
>
> With --dry-run it has no problem. I tried it with -v9 and again with
> -v4. As long as --dry-run is there it tries to only back up a few files
> that have changed. If I remove --dry-run it adds 1000s of files that
> haven't changed.
>
> It's only on this large backup that this problem occurs. I have 3
> smaller backups (~7GB) that have no problems. 2 to the same external
> drive and one to S3.
>

first.. try the new 0.6.20 it's much more performant on ubuntu.

secondly..
never heard of an issue like that. please do both runs with/without '--dry-run' in debug verbosity and copy

- the complete output
- the command line used on top

into a log file per run and pastebin those. obfuscate private data in it that you do not want other to see first!

post the links here so i can have a look at those.

..ede/duply.net

Upgraded to 0.6.20

Before running another incremental. I ran a cleanup.
duplicity cleanup --force file:///mnt/external/backup-Shared

Then I ran this:
duplicity --dry-run --verbosity debug --name opus-shared --volsize 100 --asynchronous-upload /filestore01/Shared file:///mnt/external/backup-Shared

But this time it gets confused about the full that it's working against and decides to do another full.
No backup chains with active signatures found
No orphaned or incomplete backup sets found.
No signatures found, switching to full backup.

Here's the pastebin of the log file:
http://paste.ubuntu.com/1337731/

Thank you for looking at this.
Keith

On 06.11.2012 18:06, Keith Cornwell wrote:
> Question #212772 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/212772
>
> Keith Cornwell posted a new comment:
> Upgraded to 0.6.20
>
> Before running another incremental. I ran a cleanup.
> duplicity cleanup --force file:///mnt/external/backup-Shared
>
> Then I ran this:
> duplicity --dry-run --verbosity debug --name opus-shared --volsize 100 --asynchronous-upload /filestore01/Shared file:///mnt/external/backup-Shared
>
> But this time it gets confused about the full that it's working against and decides to do another full.
> No backup chains with active signatures found
> No orphaned or incomplete backup sets found.
> No signatures found, switching to full backup.
>
> Here's the pastebin of the log file:
> http://paste.ubuntu.com/1337731/
>

the below could explain why it ignores your previous full...

6183 File duplicity-full-signatures.20121026T211319Z.sigtar.gpg is not part of a known set; creating new set
6184 Ignoring file (rejected by backup set) 'duplicity-full-signatures.20121026T211319Z.sigtar.gpg'

anyway, it is not guaranteed that newer version /always/ work perfectly with backup sets created with previous versions. the easiest way out of it is probably:

- install the duplicity version used previously in parallel (see here under TIP http://duply.net/?title=Duply-documentation )
- check that your old backup chain verify's ok with it
- do a new full with 0.6.20 to a new repository folder
- do incrementals on it and observe if the error persists

if it does, sent the two outputs i mentioned above... ede/duply.net

duplicity support

duplicity 0.6.20

Deleted contents of target, /tmp and .cache/duplicity/backup-name

Ran full with no errors.
duplicity --verbosity info --name opus-shared --volsize 200 --asynchronous-upload /filestore01/Shared file:///mnt/external/backup-Shared
logfile: http://paste.ubuntu.com/1347967/

Ran duplicity verify -v9 file:///mnt/external/backup-Shared /filestore01/Shared
Verify complete: 211964 files compared, 0 differences found.
logfile: http://paste.ubuntu.com/1347975/

Ran duplicity list-current-files -v9 file:///mnt/external/backup-Shared/
logfile: http://paste.ubuntu.com/1347894/
These lines show up again:
28 File duplicity-full.20121107T172254Z.vol163.difftar.gpg is not part of a known set; creating new set
182 File duplicity-full-signatures.20121107T172254Z.sigtar.gpg is not part of a known set; creating new set
183 Ignoring file (rejected by backup set) 'duplicity-full-signatures.20121107T172254Z.sigtar.gpg'

As before the list-current-files only lists 75770 files instead of 211964.

Ran duplicity --dry-run --verbosity debug --name opus-shared --volsize 200 --asynchronous-upload /filestore01/Shared file:///mnt/external/backup-Shared
logfile: http://paste.ubuntu.com/1348014/ I included the lines where it started to Add the unchanged files.
As before it wanted to add unchanged files to the archive.
 egrep '^A ' logfile | wc -l
yielded 135872 files it wanted to add.

thanks for the pastings...

the output of the full suggests that all files ended in the backup..

DeltaEntries 211964
RawDeltaSize 432621374933 (403 GB)

listing and incremental though seem to unable to find/list them

Comparing ('BACKUP_PHOTOS_ALL_MACHINES', 'MB10 - Done', 'Game 4 at Newton Co Done', 'Sarah', 'No - Not Now', 'SSGm4_0279.JPG') and None
Getting delta of (('BACKUP_PHOTOS_ALL_MACHINES', 'MB10 - Done', 'Game 4 at Newton Co Done', 'Sarah', 'No - Not Now', 'SSGm4_0279.JPG') /filestore01/Shared/BACKUP_PHOTOS_ALL_MACHINES/MB10 - Done/Game 4 at Newton Co Done/Sarah/No - Not Now/SSGm4_0279.JPG reg) and None
A BACKUP_PHOTOS_ALL_MACHINES/MB10 - Done/Game 4 at Newton Co Done/Sarah/No - Not Now/SSGm4_0279.JPG

a compare to _None_ means it is not in the backup. may i ask you to post an unencrypted sigtar of your backup?
you can either manually decrypt 'duplicity-full-signatures.20121107T172254Z.sigtar.gpg' or simply do a full backup w/o any encryption '--no-encryption'.

so far i'd assume duplicity fails somehow because of all the spaces in the path.

..ede/duply.net

The sigtar is 2.1GB, so I'm not sure how to post it.

I did try to untar it and have a look. The tar failed part way with:
tar: Skipping to next header
tar: Exiting with failure status due to previous errors

is it a sigtar.gpg or gzipped one?

in case it is unencrypted/unzipped already .. it should be a plain text file afair .. you could simply compress it and it should be much smaller.. then upload it to mediafire or some other filehoster

..ede/duply.net

Ok. I'll compress it first.

But the one I have won't untar. It's damaged somehow.
Is there a way to regen a sigtar without doing a backup? It's takes 17 hours for the backup.

On 12.11.2012 19:35, Keith Cornwell wrote:
> Question #212772 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/212772
>
> Keith Cornwell posted a new comment:
> Ok. I'll compress it first.
>
> But the one I have won't untar. It's damaged somehow.

where did you get it from. what's the files name? is it unencrypted?

> Is there a way to regen a sigtar without doing a backup? It's takes 17 hours for the backup.

not to my knowledge.. sigtars are generated during backup only.

..ede/duply.net

What filesystem types are the source and destination. Is it possible that
one of them does not have largefile support, e.g. files over 2GB?

On Mon, Nov 12, 2012 at 1:26 PM, edso
<email address hidden>wrote:

> Question #212772 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/212772
>
> edso proposed the following answer:
> On 12.11.2012 19:35, Keith Cornwell wrote:
> > Question #212772 on Duplicity changed:
> > https://answers.launchpad.net/duplicity/+question/212772
> >
> > Keith Cornwell posted a new comment:
> > Ok. I'll compress it first.
> >
> > But the one I have won't untar. It's damaged somehow.
>
> where did you get it from. what's the files name? is it unencrypted?
>
> > Is there a way to regen a sigtar without doing a backup? It's takes 17
> hours for the backup.
>
> not to my knowledge.. sigtars are generated during backup only.
>
> ..ede/duply.net
>
> --
> You received this question notification because you are a member of
> duplicity-team, which is an answer contact for Duplicity.
>
> _______________________________________________
> Mailing list: https://launchpad.net/~duplicity-team
> Post to : <email address hidden>
> Unsubscribe : https://launchpad.net/~duplicity-team
> More help : https://help.launchpad.net/ListHelp
>

duplicity-full-signatures.20121107T172254Z.sigtar generated by:
gpg -d /mnt/external/backup-Shared/duplicity-full-signatures.20121107T172254Z.sigtar.gpg >duplicity-full-signatures.20121107T172254Z.sigtar

Both source and target are ext4 with mount options noatime,nodiratime.
Source fs is on a RAID1(md) pair of internal drives and the target is a 1TB external USB drive.

ok, it's not plain text, but created with python tarfile somehow. never dug into this area of duplicity so far.

https://gist.github.com/2001367
has a routine to list it's contents and sort it by size.. check for yourself or put the sigtar somewhere for me to check it's entries..

..ede