High usage of /tmp when downloading signatures file for verification

Normally duplicity will download to TMP then rename to the desired
location, so no copying is needed. If TMP is on a different filesystem,
then duplicity cannot simply move and has to copy to the the new location.
When you specified TMP on the same filesystem, duplicity did not copy, it
moved, so only one instance was on the machine. With TMP on a different
filesystem, duplicity had to copy, then delete, so two instances were on
the machine for a while.

On Sat, Mar 21, 2015 at 8:01 PM, Marco B. <
<email address hidden>> wrote:

> New question #263976 on Duplicity:
> https://answers.launchpad.net/duplicity/+question/263976
>
> I made a backup of an NTFS partition to an external hard drive, which
> resulted in a ~60GB archive, and I tried to verify it. After a first
> successful verification, I tried deleting the backup's local cache folder
> and verifying again: duplicity tried to download the signatures file back
> to the local cache. The problem is that this file is apparently generated
> in /tmp, in a process which requires about twice as much space as the size
> of the signatures file itself (~1.7GB compared to 835MB). After the
> process, the signatures file is apparently copied to the local cache and
> the temporary directory is deleted. The same happens if I run a 'duplicity
> collection-status'.
>
> Is this intended behaviour? If so, what is the reason for it? Why can't
> duplicity just copy the signatures file as it is?
>
> This behaviour prevents me from using the fast volatile storage at /tmp,
> which is too small on my machine, and forces me to indicate an actual
> folder on a persistent storage, which is then also used for the rest of the
> verification process.
>
> Here is the output from duplicity (it is translated from Italian, my
> apologies) and the command I ran.
>
> -----------------
>
> # duplicity verify --no-encryption
> file:///run/media/marco/SAMSUNG/backups/ /run/media/marco/AA988E87988E522D
> --name winxp_c
>
> Synchronizing remote metadata into local cache...
> Copying duplicity-full-signatures.20150318T105452Z.sigtar.gz into local
> cache.
> Copying duplicity-full.20150318T105452Z.manifest into local cache.
> Warning, incomplete backup sets found, probably left from aborted session
> Date of last complete backup: Wed Mar 18 11:54:52 2015
> Verification completed: 420626 files compared, 0 differences found
>
> -----------------
>
> My system:
> duplicity 0.6.24, Fedora 20, kernel 3.17.8
>
> Thank you.
>
>
> --
> You received this question notification because you are a member of
> duplicity-team, which is an answer contact for Duplicity.
>
> _______________________________________________
> Mailing list: https://launchpad.net/~duplicity-team
> Post to : <email address hidden>
> Unsubscribe : https://launchpad.net/~duplicity-team
> More help : https://help.launchpad.net/ListHelp
>

Thank you very much for the explanation. That makes sense, but still, why would it need 1.7GB in /tmp? I would expect it to need 835MB in /tmp and the same in the local cache...

These are the temporary files created in whatever temp dir is specified:

# ls /tmp/duplicity-q_os9X-tempdir/ -lrth
totale 1,3G
-rw-rw-r--. 1 marco marco 834M 22 mar 13.18 mktemp-8k2YcZ-1
-rw-rw-r--. 1 marco marco 834M 22 mar 13.19 mktemp-1b9tpR-2

They look identical (I couldn't check because they are deleted too fast). The first one is then deleted, and the second one seems to be copied to the local cache, then deleted too.
I also cleaned up the target dir from incomplete sets, but this didn't change.

Thank you,
Marco

On 22.03.2015 13:36, Marco B. wrote:
> Question #263976 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/263976
>
> Status: Answered => Open
>
> Marco B. is still having a problem:
> Thank you very much for the explanation. That makes sense, but still,
> why would it need 1.7GB in /tmp? I would expect it to need 835MB in /tmp
> and the same in the local cache...
>
> These are the temporary files created in whatever temp dir is specified:
>
> # ls /tmp/duplicity-q_os9X-tempdir/ -lrth
> totale 1,3G
> -rw-rw-r--. 1 marco marco 834M 22 mar 13.18 mktemp-8k2YcZ-1
> -rw-rw-r--. 1 marco marco 834M 22 mar 13.19 mktemp-1b9tpR-2
>
> They look identical (I couldn't check because they are deleted too fast). The first one is then deleted, and the second one seems to be copied to the local cache, then deleted too.
> I also cleaned up the target dir from incomplete sets, but this didn't change.
>
> Thank you,
> Marco
>

hey Marco,

please run duplicity in max. verbose mode. it'll tell you what files are created in the context of what it is doing. please attach post the appropriate parts with some margin above and below here for us to see what is going on.
make sure to obfuscate parts you deem private in the log.

generally, as signatures and manifests are not run through librsync there is no need for a second copy in temp.

..ede/duply.net

Dear all,

thank you for your help. I ran a collection-status with verbose level 9, using an external drive as a temporary dir so that the operation could complete. I didn't run a verify because the behaviour seems to be the same, and a verify would take hours.

Here is a shortened version of the output log; you can find the full log here: http://pastebin.com/XUQBZ5da . (I managed to make it speak English, so this is untranslated.)

Thank you,
Marco

------------------------------------------------------------------------------------

# duplicity collection-status --no-encryption file:///run/media/marco/SAMSUNG/backups/Portatile_Acer/Windows_XP_C/duplicity/ --name winxp_c -v9 --tempdir=/run/media/marco/SAMSUNG/backups/.duplicity_tmp

Using archive dir: /home/marco/.cache/duplicity/winxp_c
Using backup name: winxp_c
Import of duplicity.backends.botobackend Succeeded
Import of duplicity.backends.cfbackend Succeeded
Import of duplicity.backends.dpbxbackend Succeeded
Import of duplicity.backends.ftpbackend Succeeded
Import of duplicity.backends.ftpsbackend Succeeded
Import of duplicity.backends.gdocsbackend Succeeded
Import of duplicity.backends.hsibackend Succeeded
Import of duplicity.backends.imapbackend Succeeded
Import of duplicity.backends.localbackend Succeeded
Import of duplicity.backends.megabackend Succeeded
Import of duplicity.backends.rsyncbackend Succeeded
Import of duplicity.backends.sshbackend Succeeded
Import of duplicity.backends.swiftbackend Succeeded
Import of duplicity.backends.tahoebackend Succeeded
Import of duplicity.backends.webdavbackend Succeeded
Import of duplicity.backends.~par2wrapperbackend Succeeded
Main action: collection-status
================================================================================
duplicity 0.6.24 (May 09, 2014)
Args: /usr/bin/duplicity collection-status --no-encryption file:///run/media/marco/SAMSUNG/backups/Portatile_Acer/Windows_XP_C/duplicity/ --name winxp_c -v9 --tempdir=/run/media/marco/SAMSUNG/backups/.duplicity_tmp
Linux localhost.localdomain 3.17.8-200.fc20.i686 #1 SMP Fri Jan 9 00:01:03 UTC 2015 i686 i686
/usr/bin/python 2.7.5 (default, Nov 3 2014, 14:26:46)
[GCC 4.8.3 20140911 (Red Hat 4.8.3-7)]
================================================================================
Synchronizing remote metadata to local cache...
Copying duplicity-full-signatures.20150318T105452Z.sigtar.gz to local cache.
Using temporary directory /run/media/marco/SAMSUNG/backups/.duplicity_tmp/duplicity-qzA6Mh-tempdir
Registering (mktemp) temporary file /run/media/marco/SAMSUNG/backups/.duplicity_tmp/duplicity-qzA6Mh-tempdir/mktemp-C_atCK-1
Registering (mktemp) temporary file /run/media/marco/SAMSUNG/backups/.duplicity_tmp/duplicity-qzA6Mh-tempdir/mktemp-xIGBNe-2
Deleting /run/media/marco/SAMSUNG/backups/.duplicity_tmp/duplicity-qzA6Mh-tempdir/mktemp-C_atCK-1
Forgetting temporary file /run/media/marco/SAMSUNG/backups/.duplicity_tmp/duplicity-qzA6Mh-tempdir/mktemp-C_atCK-1
Deleting /run/media/marco/SAMSUNG/backups/.duplicity_tmp/duplicity-qzA6Mh-tempdir/mktemp-xIGBNe-2
Forgetting temporary file /run/media/marco/SAMSUNG/backups/.duplicity_tmp/duplicity-qzA6Mh-tempdir/mktemp-xIGBNe-2
Copying duplicity-full.20150318T105452Z.manifest to local cache.
Registering (mktemp) temporary file /run/media/marco/SAMSUNG/backups/.duplicity_tmp/duplicity-qzA6Mh-tempdir/mktemp-YiSvp6-3
Registering (mktemp) temporary file /run/media/marco/SAMSUNG/backups/.duplicity_tmp/duplicity-qzA6Mh-tempdir/mktemp-ACUqOL-4
Deleting /run/media/marco/SAMSUNG/backups/.duplicity_tmp/duplicity-qzA6Mh-tempdir/mktemp-YiSvp6-3
Forgetting temporary file /run/media/marco/SAMSUNG/backups/.duplicity_tmp/duplicity-qzA6Mh-tempdir/mktemp-YiSvp6-3
Deleting /run/media/marco/SAMSUNG/backups/.duplicity_tmp/duplicity-qzA6Mh-tempdir/mktemp-ACUqOL-4
Forgetting temporary file /run/media/marco/SAMSUNG/backups/.duplicity_tmp/duplicity-qzA6Mh-tempdir/mktemp-ACUqOL-4
2392 files exist on backend
4 files exist in cache
Extracting backup chains from list of files: [u'duplicity-full-signatures.20150318T105452Z.sigtar.gz', u'duplicity-full.20150318T105452Z.manifest', u'duplicity-full.20150318T105452Z.vol1.difftar.gz', u'duplicity-full.20150318T105452Z.vol10.difftar.gz', u'duplicity-full.20150318T105452Z.vol100.difftar.gz', [......SNIP......], u'duplicity-full.20150318T105452Z.vol2388.difftar.gz', u'duplicity-full.20150318T105452Z.vol2389.difftar.gz']
File duplicity-full-signatures.20150318T105452Z.sigtar.gz is not part of a known set; creating new set
Ignoring file (rejected by backup set) 'duplicity-full-signatures.20150318T105452Z.sigtar.gz'
File duplicity-full.20150318T105452Z.manifest is not part of a known set; creating new set
File duplicity-full.20150318T105452Z.vol1.difftar.gz is part of known set
File duplicity-full.20150318T105452Z.vol10.difftar.gz is part of known set
File duplicity-full.20150318T105452Z.vol100.difftar.gz is part of known set
File duplicity-full.20150318T105452Z.vol1000.difftar.gz is part of known set
[......SNIP......]
File duplicity-full.20150318T105452Z.vol2387.difftar.gz is part of known set
File duplicity-full.20150318T105452Z.vol2388.difftar.gz is part of known set
File duplicity-full.20150318T105452Z.vol2389.difftar.gz is part of known set
Found backup chain [Wed Mar 18 11:54:52 2015]-[Wed Mar 18 11:54:52 2015]
Last full backup date: Wed Mar 18 11:54:52 2015
Collection Status
-----------------
Connecting with backend: LocalBackend
Archive dir: /home/marco/.cache/duplicity/winxp_c

Found 0 secondary backup chains.

Found primary backup chain with matching signature chain:
-------------------------
Chain start time: Wed Mar 18 11:54:52 2015
Chain end time: Wed Mar 18 11:54:52 2015
Number of contained backup sets: 1
Total number of contained volumes: 2390
 Type of backup set: Time: Num volumes:
                Full Wed Mar 18 11:54:52 2015 2390
-------------------------
No orphaned or incomplete backup sets found.
Collection Status
-----------------
Connecting with backend: LocalBackend
Archive dir: /home/marco/.cache/duplicity/winxp_c

Found 0 secondary backup chains.

Found primary backup chain with matching signature chain:
-------------------------
Chain start time: Wed Mar 18 11:54:52 2015
Chain end time: Wed Mar 18 11:54:52 2015
Number of contained backup sets: 1
Total number of contained volumes: 2390
 Type of backup set: Time: Num volumes:
                Full Wed Mar 18 11:54:52 2015 2390
-------------------------
No orphaned or incomplete backup sets found.
Releasing lockfile <lockfile.linklockfile.LinkLockFile instance at 0x9fcda0c>

Oh, and this is the content of the local cache after the operation (it was purged just before running duplicity):

-------------------------------

# ls -lrth ~/.cache/duplicity/winxp_c

total 835M
-rw-------. 1 marco marco 834M Mar 22 15:14 duplicity-full-signatures.20150318T105452Z.sigtar.gz
-rw-------. 1 marco marco 688K Mar 22 15:15 duplicity-full.20150318T105452Z.manifest

-------------------------------

Thank you,
Marco

On 22.03.2015 15:41, Marco B. wrote:
> Question #263976 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/263976
>
> Marco B. posted a new comment:
> Dear all,
>
> thank you for your help. I ran a collection-status with verbose level 9,
> using an external drive as a temporary dir so that the operation could
> complete. I didn't run a verify because the behaviour seems to be the
> same, and a verify would take hours.
>
> Here is a shortened version of the output log; you can find the full log
> here: http://pastebin.com/XUQBZ5da . (I managed to make it speak
> English, so this is untranslated.)
>

nice catch, looks like the fetch routines really create two temp files.
one in
 backend.get_fileobj_read(...)
the other in it's caller a little bit later under
 duplicity.sync_archive(...).copy_to_local(...)

will have to have a closer look to decide the necessity and come up with a fix as probably necessary. please create a bug ticket for this and link it to your question.

thanks ede/duply.net

There is a button on the page to convert the question to a bug.

On Sun, Mar 22, 2015 at 10:21 AM, edso <<email address hidden>
> wrote:

> Question #263976 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/263976
>
> edso proposed the following answer:
> On 22.03.2015 15:41, Marco B. wrote:
> > Question #263976 on Duplicity changed:
> > https://answers.launchpad.net/duplicity/+question/263976
> >
> > Marco B. posted a new comment:
> > Dear all,
> >
> > thank you for your help. I ran a collection-status with verbose level 9,
> > using an external drive as a temporary dir so that the operation could
> > complete. I didn't run a verify because the behaviour seems to be the
> > same, and a verify would take hours.
> >
> > Here is a shortened version of the output log; you can find the full log
> > here: http://pastebin.com/XUQBZ5da . (I managed to make it speak
> > English, so this is untranslated.)
> >
>
> nice catch, looks like the fetch routines really create two temp files.
> one in
> backend.get_fileobj_read(...)
> the other in it's caller a little bit later under
> duplicity.sync_archive(...).copy_to_local(...)
>
> will have to have a closer look to decide the necessity and come up with
> a fix as probably necessary. please create a bug ticket for this and
> link it to your question.
>
> thanks ede/duply.net
>
> --
> You received this question notification because you are a member of
> duplicity-team, which is an answer contact for Duplicity.
>
> _______________________________________________
> Mailing list: https://launchpad.net/~duplicity-team
> Post to : <email address hidden>
> Unsubscribe : https://launchpad.net/~duplicity-team
> More help : https://help.launchpad.net/ListHelp
>

Thank you Kenneth and edso for the help!

I created a bug report: https://bugs.launchpad.net/duplicity/+bug/1435027. I hope I did it right.

Should I close this question as solved? I think I got my answer...

Thanks,
Marco