Duplicity

duplicity URL & variable parsing (webDAV) in shell-script

Asked by nomennominatur

# I am not a seasoned Ubuntu admin/ Duplicity user, so please bear with me if I am missing something obvious...

1. Many WebDAV-providers supply usernames as "<email address hidden>". Although this does not seem to be RFC-compliant, users have to live with that anyway...

2. Many WebDAV backends ignore duplicity's FTP_PASSWORD environment variable.

3. Duplicity works OK when I provide cleartext password from CLI, e.g.

    <duplicity /localdir webdavs://'<email address hidden>':<email address hidden>/remotedir>

4. However, I do not want to use my cleartext password in the command line process so that other users might not spoof it from the process list

5. Trying to use duplicity from a shell script (bash), without using the cleartext password in the command, e.g.

    <PWVARIABLE='cleartextpassword'
    export $PWVARIABLE
    duplicity /localdir webdavs://'<email address hidden>':$<email address hidden>/remotedir>

I always run into parsing problems of the juxtaposed "@" symbol which, together with the WebDAV URL, will be parsed into the variable name. This does not seem to be relieved by exchanging "%40" for "@" as duplicity (or bash?) do not seem to parse this in a useful manner. Putting "$PWVARIABLE" into quotes does not work either, since this seems to be parsed as plaintext and not as variable.

My aim is to run the script with cron for off-site backups.

Is there a workable syntax for this? Any help is greatly appreciated!

# Thanks & kudos for this great tool!

<<edit>>
This site's laudable privacy filter automatically changes any string 'aaa<at>bbbbb.ccc' into a generic 'e-mail address'. This will, however, mess up other URLs in posts as well.

Just for clarification: In my syntax example above, it shoud read something like

<webdavs://'fullemailaddress':$PWVARIABLE<at>fullwebdavURL/remotedir>

Thanks!

Question information

Language:
English Edit question
Status:
Solved
For:
Duplicity Edit question
Assignee:
No assignee Edit question
Solved by:
edso
Solved:
Last query:
Last reply:
Revision history for this message
Best edso (ed.so) said : 		#1

On 31.03.2016 20:13, nomennominatur wrote:
> Question #289684 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/289684
>
> Description changed to:
> # I am not a seasoned Ubuntu admin/ Duplicity user, so please bear with
> me if I am missing something obvious...
>
> 1. Many WebDAV-providers supply usernames as
> "<email address hidden>". Although this does not seem to be RFC-
> compliant, users have to live with that anyway...

which RFC would that be

> 2. Many WebDAV backends ignore duplicity's FTP_PASSWORD environment
> variable.

the current two webdav backends use FTP_PASSWORD, if it ss set.

> 3. Duplicity works OK when I provide cleartext password from CLI, e.g.
>
> <duplicity /localdir
> webdavs://'<email address hidden>':<email address hidden>/remotedir>

make sure to url encode the @, the above is clearly breaking the standard eg. like
 http://stackoverflow.com/questions/10050877/url-username-with

> 4. However, I do not want to use my cleartext password in the command
> line process so that other users might not spoof it from the process
> list

as said above, use FTP_PASSWORD. it works

> 5. Trying to use duplicity from a shell script (bash), without using the
> cleartext password in the command, e.g.
>
> <PWVARIABLE='cleartextpassword'
> export $PWVARIABLE
> duplicity /localdir webdavs://'<email address hidden>':$<email address hidden>/remotedir>
>
> I always run into parsing problems of the juxtaposed "@" symbol which,
> together with the WebDAV URL, will be parsed into the variable name.
> This does not seem to be relieved by exchanging "%40" for "@" as
> duplicity (or bash?) do not seem to parse this in a useful manner.

that also works fine for me, stick to the url encoded %40

> Putting "$PWVARIABLE" into quotes does not work either, since this seems
> to be parsed as plaintext and not as variable.
>
> My aim is to run the script with cron for off-site backups.
>
> Is there a workable syntax for this? Any help is greatly appreciated!
>
> # Thanks & kudos for this great tool!
>
> <<edit>>
> This site's laudable privacy filter automatically changes any string 'aaa<at>bbbbb.ccc' into a generic 'e-mail address'. This will, however, mess up other URLs in posts as well.
>
> Just for clarification: In my syntax example above, it shoud read
> something like
>
> <webdavs://'fullemailaddress':$PWVARIABLE<at>fullwebdavURL/remotedir>
>
> Thanks!
>

try

FTP_PASSWORD='supersecretpass'
duplicity /source webdavs://user%<email address hidden>/targetfolder

quotation marks or @ have no place in the url's auth info.

make sure you use a recent duplicity. either 0.6.26 or the latest 0.7.x ..ede/duply.net

Revision history for this message
nomennominatur (datenschutz-123) said : 		#2

# Red face department: With the syntax you supplied, everything works fine! Sorry for having bothered you.
# Thank you for taking the time to help a non-pro.

(a)
> which RFC would that be
 OK, I do seem to have misread the discussion on http://bugs.python.org/issue1698

(b)
> make sure to url encode the @, the above is clearly breaking the standard
 I obviously encoded the wrong '@' (the second one), as I was confused by the syntax...

(c)
> try
> FTP_PASSWORD='supersecretpass'
> duplicity /source webdavs://user%<email address hidden>/targetfolder
 Backend accepts FTP_PASSWORD, indeed.

(d)
> make sure you use a recent duplicity. either 0.6.26 or the latest 0.7.x ..ede/duply.net
 Since I am on Ubuntu Server LTS 14.04, I am using duplicity 0.6.23, but as above, it works fine.

Revision history for this message
nomennominatur (datenschutz-123) said : 		#3

Thanks edso, that solved my question.

Revision history for this message
edso (ed.so) said : 		#4

On 01.04.2016 12:22, nomennominatur wrote:
> (d)
>> > make sure you use a recent duplicity. either 0.6.26 or the latest 0.7.x ..ede/duply.net
> Since I am on Ubuntu Server LTS 14.04, I am using duplicity 0.6.23, but as above, it works fine.

as backups are about data safety you might wanna check out the changelog for duplicity. after 0.6.23 there were some major issues fixed, so if you love your data use the latest greatest.

..ede/duply.net

Revision history for this message
nomennominatur (datenschutz-123) said : 		#5

I am hesitant to ask this rather newbie question in public ;-)

Which repository do I have to add to include the latest stable duplicity?

sudo add-apt-repository ppa:’???’
sudo apt-get update
sudo apt-get install duplicity’???’

Thank you again!

(I use the webdav only as tertiary off-site backup, but you are right, of course: if one backs up, one should do it properly…)

Am 01.04.2016 um 12:37 schrieb edso <email address hidden>:

> Your question #289684 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/289684
>
> edso posted a new comment:
> On 01.04.2016 12:22, nomennominatur wrote:
>> (d)
>>>> make sure you use a recent duplicity. either 0.6.26 or the latest 0.7.x ..ede/duply.net
>> Since I am on Ubuntu Server LTS 14.04, I am using duplicity 0.6.23, but as above, it works fine.
>
> as backups are about data safety you might wanna check out the changelog
> for duplicity. after 0.6.23 there were some major issues fixed, so if
> you love your data use the latest greatest.
>
> ..ede/duply.net
>
> --
> You received this question notification because you asked the question.

Revision history for this message
edso (ed.so) said : 		#6

how about the one mentioned on
 http://duplicity.nongnu.org/

..ede

On April 1, 2016 12:51:53 PM GMT+02:00, nomennominatur <email address hidden> wrote:
>Question #289684 on Duplicity changed:
>https://answers.launchpad.net/duplicity/+question/289684
>
>nomennominatur posted a new comment:
>I am hesitant to ask this rather newbie question in public ;-)
>
>Which repository do I have to add to include the latest stable
>duplicity?
>
>sudo add-apt-repository ppa:’???’
>sudo apt-get update
>sudo apt-get install duplicity’???’
>
>Thank you again!
>
>(I use the webdav only as tertiary off-site backup, but you are right,
>of course: if one backs up, one should do it properly…)
>
>
>Am 01.04.2016 um 12:37 schrieb edso
><email address hidden>:
>
>> Your question #289684 on Duplicity changed:
>> https://answers.launchpad.net/duplicity/+question/289684
>>
>> edso posted a new comment:
>> On 01.04.2016 12:22, nomennominatur wrote:
>>> (d)
>>>>> make sure you use a recent duplicity. either 0.6.26 or the latest
>0.7.x ..ede/duply.net
>>> Since I am on Ubuntu Server LTS 14.04, I am using duplicity 0.6.23,
>but as above, it works fine.
>>
>> as backups are about data safety you might wanna check out the
>changelog
>> for duplicity. after 0.6.23 there were some major issues fixed, so if
>> you love your data use the latest greatest.
>>
>> ..ede/duply.net
>>
>> --
>> You received this question notification because you asked the
>question.
>
>--
>You received this question notification because your team
>duplicity-team
>is an answer contact for Duplicity.
>
>_______________________________________________
>Mailing list: https://launchpad.net/~duplicity-team
>Post to : <email address hidden>
>Unsubscribe : https://launchpad.net/~duplicity-team
>More help : https://help.launchpad.net/ListHelp

Revision history for this message
nomennominatur (datenschutz-123) said : 		#7

thx.
Am 01.04.2016 um 13:07 schrieb edso <email address hidden>:

> Your question #289684 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/289684
>
> edso posted a new comment:
> how about the one mentioned on
> http://duplicity.nongnu.org/
>
> ..ede
>
>
> On April 1, 2016 12:51:53 PM GMT+02:00, nomennominatur <email address hidden> wrote:
>> Question #289684 on Duplicity changed:
>> https://answers.launchpad.net/duplicity/+question/289684
>>
>> nomennominatur posted a new comment:
>> I am hesitant to ask this rather newbie question in public ;-)
>>
>> Which repository do I have to add to include the latest stable
>> duplicity?
>>
>> sudo add-apt-repository ppa:’???’
>> sudo apt-get update
>> sudo apt-get install duplicity’???’
>>
>> Thank you again!
>>
>> (I use the webdav only as tertiary off-site backup, but you are right,
>> of course: if one backs up, one should do it properly…)
>>
>>
>> Am 01.04.2016 um 12:37 schrieb edso
>> <email address hidden>:
>>
>>> Your question #289684 on Duplicity changed:
>>> https://answers.launchpad.net/duplicity/+question/289684
>>>
>>> edso posted a new comment:
>>> On 01.04.2016 12:22, nomennominatur wrote:
>>>> (d)
>>>>>> make sure you use a recent duplicity. either 0.6.26 or the latest
>> 0.7.x ..ede/duply.net
>>>> Since I am on Ubuntu Server LTS 14.04, I am using duplicity 0.6.23,
>> but as above, it works fine.
>>>
>>> as backups are about data safety you might wanna check out the
>> changelog
>>> for duplicity. after 0.6.23 there were some major issues fixed, so if
>>> you love your data use the latest greatest.
>>>
>>> ..ede/duply.net
>>>
>>> --
>>> You received this question notification because you asked the
>> question.
>>
>> --
>> You received this question notification because your team
>> duplicity-team
>> is an answer contact for Duplicity.
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~duplicity-team
>> Post to : <email address hidden>
>> Unsubscribe : https://launchpad.net/~duplicity-team
>> More help : https://help.launchpad.net/ListHelp
>
> --
> You received this question notification because you asked the question.