Swift is working but duplicity is unable to upload a file

On 27.09.2014 22:56, stephane wrote:
> New question #255082 on Duplicity:
> https://answers.launchpad.net/duplicity/+question/255082
>
> Swift seems to work well :
>
> stephane@marlysa:~$ swift --auth-version $SWIFT_AUTHVERSION --os-auth-url $SWIFT_AUTHURL --os-username $SWIFT_USERNAME --os-password $SWIFT_PASSWORD upload Test7 /home/stephane/Documents/2.Velo/Mensuration.ods
> home/stephane/Documents/2.Velo/Mensuration.ods
>
> But, duplicity is unable to upload a file :
>
> stephane@marlysa:~$ duplicity --verbosity 8 --num-retries 1 full /home/stephane/Documents/2.Velo/ swift://Test8/
> Using archive dir: /home/stephane/.cache/duplicity/1e81562248ef66ee3b7330f52dfc0be5
> Using backup name: 1e81562248ef66ee3b7330f52dfc0be5
> Import of duplicity.backends.botobackend Succeeded
> Import of duplicity.backends.cfbackend Succeeded
> Import of duplicity.backends.dpbxbackend Succeeded
> Import of duplicity.backends.ftpbackend Succeeded
> Import of duplicity.backends.ftpsbackend Succeeded
> Import of duplicity.backends.gdocsbackend Succeeded
> Import of duplicity.backends.hsibackend Succeeded
> Import of duplicity.backends.imapbackend Succeeded
> Import of duplicity.backends.localbackend Succeeded
> Import of duplicity.backends.megabackend Succeeded
> Import of duplicity.backends.rsyncbackend Succeeded
> Import of duplicity.backends.sshbackend Succeeded
> Import of duplicity.backends.swiftbackend Succeeded
> Import of duplicity.backends.tahoebackend Succeeded
> Import of duplicity.backends.webdavbackend Succeeded
> Import of duplicity.backends.~par2wrapperbackend Succeeded
> Main action: full
> ================================================================================
> duplicity 0.6.24 (May 09, 2014)
> Args: /usr/bin/duplicity --verbosity 8 --num-retries 1 full /home/stephane/Documents/2.Velo/ swift://Test8/
> Linux marlysa 3.14-1-amd64 #1 SMP Debian 3.14.9-1 (2014-06-30) x86_64
> /usr/bin/python 2.7.8 (default, Sep 9 2014, 22:08:43)
> [GCC 4.9.1]
> ================================================================================
> Using temporary directory /tmp/duplicity-n47fGv-tempdir
> Temp has 67567054848 available, backup will use approx 34078720.
> Listing 'Test8/'
> Local and Remote metadata are synchronized, no sync needed.
> Listing 'Test8/'
> Last full backup left a partial set, restarting.
> Last full backup date: Sat Sep 27 22:44:02 2014
> Collection Status
> -----------------
> Connecting with backend: SwiftBackend
> Archive dir: /home/stephane/.cache/duplicity/1e81562248ef66ee3b7330f52dfc0be5
>
> Found 0 secondary backup chains.
>
> Found primary backup chain with matching signature chain:
> -------------------------
> Chain start time: Sat Sep 27 22:44:02 2014
> Chain end time: Sat Sep 27 22:44:02 2014
> Number of contained backup sets: 1
> Total number of contained volumes: 0
> Type of backup set: Time: Num volumes:
> -------------------------
> No orphaned or incomplete backup sets found.
> RESTART: The first volume failed to upload before termination.
> Restart is impossible...starting backup from beginning.
> Deleting /home/stephane/.cache/duplicity/1e81562248ef66ee3b7330f52dfc0be5/duplicity-full.20140927T204402Z.manifest.part
> Deleting /home/stephane/.cache/duplicity/1e81562248ef66ee3b7330f52dfc0be5/duplicity-full-signatures.20140927T204402Z.sigtar.part
> Using archive dir: /home/stephane/.cache/duplicity/1e81562248ef66ee3b7330f52dfc0be5
> Using backup name: 1e81562248ef66ee3b7330f52dfc0be5
> Import of duplicity.backends.botobackend Succeeded
> Import of duplicity.backends.cfbackend Succeeded
> Import of duplicity.backends.dpbxbackend Succeeded
> Import of duplicity.backends.ftpbackend Succeeded
> Import of duplicity.backends.ftpsbackend Succeeded
> Import of duplicity.backends.gdocsbackend Succeeded
> Import of duplicity.backends.hsibackend Succeeded
> Import of duplicity.backends.imapbackend Succeeded
> Import of duplicity.backends.localbackend Succeeded
> Import of duplicity.backends.megabackend Succeeded
> Import of duplicity.backends.rsyncbackend Succeeded
> Import of duplicity.backends.sshbackend Succeeded
> Import of duplicity.backends.swiftbackend Succeeded
> Import of duplicity.backends.tahoebackend Succeeded
> Import of duplicity.backends.webdavbackend Succeeded
> Import of duplicity.backends.~par2wrapperbackend Succeeded
> Main action: full
> ================================================================================
> duplicity 0.6.24 (May 09, 2014)
> Args: /usr/bin/duplicity --verbosity 8 --num-retries 1 full /home/stephane/Documents/2.Velo/ swift://Test8/
> Linux marlysa 3.14-1-amd64 #1 SMP Debian 3.14.9-1 (2014-06-30) x86_64
> /usr/bin/python 2.7.8 (default, Sep 9 2014, 22:08:43)
> [GCC 4.9.1]
> ================================================================================
> Using temporary directory /tmp/duplicity-rjXDfA-tempdir
> Temp has 67567050752 available, backup will use approx 34078720.
> Listing 'Test8/'
> Local and Remote metadata are synchronized, no sync needed.
> Listing 'Test8/'
> Last full backup date: none
> Collection Status
> -----------------
> Connecting with backend: SwiftBackend
> Archive dir: /home/stephane/.cache/duplicity/1e81562248ef66ee3b7330f52dfc0be5
>
> Found 0 secondary backup chains.
> No backup chains with active signatures found
> No orphaned or incomplete backup sets found.
> Using temporary directory /home/stephane/.cache/duplicity/1e81562248ef66ee3b7330f52dfc0be5/duplicity-um9rkr-tempdir
> Using temporary directory /home/stephane/.cache/duplicity/1e81562248ef66ee3b7330f52dfc0be5/duplicity-dMbhFi-tempdir
> AsyncScheduler: instantiating at concurrency 0
> A .
> A .directory
> A Mensuration.ods
> A Velo2010.ods
> A Velo2011.ods
> A Velo2012.ods
> A Velo2013.ods
> AsyncScheduler: running task synchronously (asynchronicity disabled)
> Uploading 'Test8//duplicity-full.20140927T204429Z.vol1.difftar.gpg'
> Upload of 'duplicity-full.20140927T204429Z.vol1.difftar.gpg' failed (attempt 1): Error: []
> Giving up uploading 'duplicity-full.20140927T204429Z.vol1.difftar.gpg' after 1 attempts
> Backend error detail: Traceback (most recent call last):
> File "/usr/bin/duplicity", line 1509, in <module>
> with_tempdir(main)
> File "/usr/bin/duplicity", line 1503, in with_tempdir
> fn()
> File "/usr/bin/duplicity", line 1352, in main
> do_backup(action)
> File "/usr/bin/duplicity", line 1473, in do_backup
> full_backup(col_stats)
> File "/usr/bin/duplicity", line 545, in full_backup
> globals.backend)
> File "/usr/bin/duplicity", line 427, in write_multivol
> (tdp, dest_filename, vol_num)))
> File "/usr/lib/python2.7/dist-packages/duplicity/asyncscheduler.py", line 145, in schedule_task
> return self.__run_synchronously(fn, params)
> File "/usr/lib/python2.7/dist-packages/duplicity/asyncscheduler.py", line 171, in __run_synchronously
> ret = fn(*params)
> File "/usr/bin/duplicity", line 426, in <lambda>
> async_waiters.append(io_scheduler.schedule_task(lambda tdp, dest_filename, vol_num: put(tdp, dest_filename, vol_num),
> File "/usr/bin/duplicity", line 315, in put
> backend.put(tdp, dest_filename)
> File "/usr/lib/python2.7/dist-packages/duplicity/backends/swiftbackend.py", line 107, in put
> raise BackendException("Error uploading '%s'" % remote_filename)
> BackendException: Error uploading 'duplicity-full.20140927T204429Z.vol1.difftar.gpg'
>
> BackendException: Error uploading 'duplicity-full.20140927T204429Z.vol1.difftar.gpg'
>
> The container Test8 has been correctly created, but no file can be uploaded.
>
> The cloud used is cloudwatt. I have a debian (Jessie).
>
> I do not know where the problem is coming from.
>
> Does somebody have any idea ?
>

did you set the required env vars as described on the man page?
http://duplicity.nongnu.org/duplicity.1.html#sect19

..ede/duply.net

Hi Edso,

Yes I do. And I assume that this is correctly set. Indeed, the swift command worked well :.
stephane@marlysa:~$ swift --auth-version $SWIFT_AUTHVERSION --os-auth-url $SWIFT_AUTHURL --os-username $SWIFT_USERNAME --os-password $SWIFT_PASSWORD upload Test7 /home/stephane/Documents/2.Velo/Mensuration.ods

Futhermore, duplicity can correctly connect the cloud to create a new container.

Any other idea ?

Best regards,
Stéphane.

On 29.09.2014 13:29, stephane wrote:
> Question #255082 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/255082
>
> Status: Answered => Open
>
> stephane is still having a problem:
> Hi Edso,
>
> Yes I do. And I assume that this is correctly set. Indeed, the swift command worked well :.
> stephane@marlysa:~$ swift --auth-version $SWIFT_AUTHVERSION --os-auth-url $SWIFT_AUTHURL --os-username $SWIFT_USERNAME --os-password $SWIFT_PASSWORD upload Test7 /home/stephane/Documents/2.Velo/Mensuration.ods
>
> Futhermore, duplicity can correctly connect the cloud to create a new
> container.
>
> Any other idea ?
>

no.. your verbose output does not really say what's wrong. you can try to expose the specific error thrown by swift within duplicity. have to dive into python for that though.

or google? .. good luck .. ede

Unfortunately, google does not help.

It comes from the putrequest function.
the bug does not appear for all backups. It depends of the files or documents .

stephane@marlysa:~$ duplicity --num-retries 1 ~stephane/Téléchargement/test.wav swift://Test3
<...>
Last full backup date: none
No signatures found, switching to full backup.
> /usr/lib/python2.7/dist-packages/swiftclient/client.py(199)putrequest()
-> return self.request('PUT', full_path, data, headers, files)
(Pdb) print (full_path)
/v1/AUTH_fce*********************************/Test3/duplicity-full.20141004T124110Z.vol1.difftar.gpg
(Pdb) print(data)
<generator object chunk_reader at 0x7ff443d4fe60>
(Pdb) print(headers)
{'X-Auth-Token': u'f**********************************'}
(Pdb) print(files)
None
(Pdb) next
Error: Error([],)
> /usr/lib/python2.7/dist-packages/swiftclient/client.py(199)putrequest()
-> return self.request('PUT', full_path, data, headers, files)
(Pdb)

well, the failure seems to come from the putrequest.
Especially, when the upload is split in several segments and / or when the put request is done in chunk mode.

Indeed,

swift --auth-version $SWIFT_AUTHVERSION --os-auth-url $SWIFT_AUTHURL --os-username $SWIFT_USERNAME --os-password $SWIFT_PASSWORD upload Test7 /home/stephane/Documents/2.Velo/Velo2010.ods
Succeed.

swift --auth-version $SWIFT_AUTHVERSION --os-auth-url $SWIFT_AUTHURL --os-username $SWIFT_USERNAME --os-password $SWIFT_PASSWORD upload -S 10000 Test7 /home/stephane/Documents/2.Velo/Velo2010.ods
Failed.

Any idea ?

On 04.10.2014 17:21, stephane wrote:
> Question #255082 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/255082
>
> stephane gave more information on the question:
> well, the failure seems to come from the putrequest.
> Especially, when the upload is split in several segments and / or when the put request is done in chunk mode.
>
> Indeed,
>
> swift --auth-version $SWIFT_AUTHVERSION --os-auth-url $SWIFT_AUTHURL --os-username $SWIFT_USERNAME --os-password $SWIFT_PASSWORD upload Test7 /home/stephane/Documents/2.Velo/Velo2010.ods
> Succeed.
>
> swift --auth-version $SWIFT_AUTHVERSION --os-auth-url $SWIFT_AUTHURL --os-username $SWIFT_USERNAME --os-password $SWIFT_PASSWORD upload -S 10000 Test7 /home/stephane/Documents/2.Velo/Velo2010.ods
> Failed.
>

you can try to ask the swift authors on their mailinglist or such. they should know.

..ede/duply.net