some problems with OSX

On 29.02.2012 14:41, Anton wrote:
> New question #189204 on Duplicity:
> https://answers.launchpad.net/duplicity/+question/189204
>
> I've been using 0.6.15 release through duply in a debian machine without problems.
> Now i'm using osx and installed 0.6.17 with macports, I use it as root
>
> Three problems have arised since:
> 1.- ssh backend asks for password even when authorized_keys exists in server (command line ssh works without it)

could you pastebin a complete console output with debug verbosity (-v9)? obfuscate private strings in it before uploading.

> 2.- Huge amounts of memory being used (machine swaps even when having 4Gb/RAM). (Sometimes duplicity even stops)

should be solved with next release, you could manually apply the patch attached to the affected versions 0.6.16/17
https://bugs.launchpad.net/duplicity/+bug/908228

> 3.- After first full backup, the next and first incremental marks all files under /Applications and many other system files as Deleted

there's another complaint about files erronously being marked deleted.
A) do you use a exclusion file? please send the contents.
B) can you try duplicity 0.6.15 and see if the error persists?

..ede/duply.net

Hello,
First of all, thank you so much for this packages and for giving a very
useful and simple backup solution with duply.

As for the problem, attached I enclose:
The dump (-9), with private info and many unuseful lines removed
(replaced by a line of dots).
The exclude file that duply uses. The exclude file is my try to merge
Apple Time Machine own excludes (extracted from a config file) with my
own excludes in my previous debian system

In the dump you can see:
1.- That the program asks for the ssh password even when I can do ssh
username@server in same terminal and do login without password
2.- The files marked for deletion (all from /) in the last part after
comparing a file in /Users/...

Could be useful to note that:
About 615187 Files were compared (some correctly flagged with D or M)
before wrong part
There was one more file (mysqldump_projectname_20110221.sql.bz2) in the
last directory compared before the wrong part, and many other files were
not compared. It seems for me that the program ended the comparisons
before checking all files, and started again from root dir but without
filenames to compare (Comparing None) and so, marked all as deleted.

In the dump, in line 66 the signatures and manifest files were treated
as not part of known set, even when the difftar are detected as part of
known set. Signatures and manifest were uploaded fine at the end of the
full backup, and they were correctly stored in .cache directory.
signatures file have 8.2Gb

I was not able to do the initial full backup in only a session. Due to
its size (about 130Gb), I had to kill the process because of the memory
problems cited, lack of time to upload the full set during work time, or
connection issues. Three or four restarts were done for having the
initial full set in the server.

Also, I've applied the patch for avoiding memory problems. I was not
able to reproduce the problem in this last test, but tomorow will try a
more intensive task.

Thanks for your express support,
Anton

On 29/2/12 17:10, edso wrote:
> Your question #189204 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/189204
>
> Status: Open => Answered
>
> edso proposed the following answer:
> On 29.02.2012 14:41, Anton wrote:
>> New question #189204 on Duplicity:
>> https://answers.launchpad.net/duplicity/+question/189204
>>
>> I've been using 0.6.15 release through duply in a debian machine without problems.
>> Now i'm using osx and installed 0.6.17 with macports, I use it as root
>>
>> Three problems have arised since:
>> 1.- ssh backend asks for password even when authorized_keys exists in server (command line ssh works without it)
> could you pastebin a complete console output with debug verbosity (-v9)?
> obfuscate private strings in it before uploading.
>
>> 2.- Huge amounts of memory being used (machine swaps even when having
> 4Gb/RAM). (Sometimes duplicity even stops)
>
> should be solved with next release, you could manually apply the patch attached to the affected versions 0.6.16/17
> https://bugs.launchpad.net/duplicity/+bug/908228
>
>> 3.- After first full backup, the next and first incremental marks all
> files under /Applications and many other system files as Deleted
>
> there's another complaint about files erronously being marked deleted.
> A) do you use a exclusion file? please send the contents.
> B) can you try duplicity 0.6.15 and see if the error persists?
>
> ..ede/duply.net
>

Sorry, missed the pastebins...

Duply dump (-9)
http://pastebin.com/QEyrzELn

exclude file
http://pastebin.com/EBfKuDsJ

On 29.02.2012 22:20, Anton wrote:
> Question #189204 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/189204
>
> Anton gave more information on the question:
> Sorry, missed the pastebins...
>
> Duply dump (-9)
> http://pastebin.com/QEyrzELn
>
> exclude file
> http://pastebin.com/EBfKuDsJ
>

do you have enough local space to try to reproduce this with the file:// backend. i somewhat guess that your restarting the initial full backup might have caused this.
the full backup should run through completely (much faster of course) and after that check if the incremental shows the same behavior.

..ede

Sorry , I had not enough local space for trying the local backend.
However, I have been able to do a new full backup in a faster server, interrupting it only two times with Ctrl+z, in order to not terminate the process.
This backup has been finished ok, without memory problems after aplying your patch.
Also, I've tried to do a incremental backup after the full, and modified files were detected ok, with no strange deletions.

The only problem that remains is the one with the ssh with no password. Strange here is that I'm able to ssh without pass from the same terminal, and same keys and configuration works ok from a linux vm. Perhaps a macports issue with no further relevance for me

On 06.03.2012 14:20, Anton wrote:
> Question #189204 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/189204
>
> Status: Answered => Solved
>
> Anton confirmed that the question is solved:
> Sorry , I had not enough local space for trying the local backend.
> However, I have been able to do a new full backup in a faster server, interrupting it only two times with Ctrl+z, in order to not terminate the process.
> This backup has been finished ok, without memory problems after aplying your patch.
> Also, I've tried to do a incremental backup after the full, and modified files were detected ok, with no strange deletions.
>
> The only problem that remains is the one with the ssh with no password.
> Strange here is that I'm able to ssh without pass from the same
> terminal, and same keys and configuration works ok from a linux vm.
> Perhaps a macports issue with no further relevance for me
>

Could you please post your complete command line and pastebin the console output of a run with -v9 ?
Maybe we can see why duplicitry is asking for a password.
What is your setup keyauth? PASSPHRASE var?

You could make a simple testbackup of some testfolder (might save you some obfuscating).

..ede/duply.net

On 06.03.2012 14:31, edso wrote:
> Question #189204 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/189204
>
> edso posted a new comment:
> On 06.03.2012 14:20, Anton wrote:
>> Question #189204 on Duplicity changed:
>> https://answers.launchpad.net/duplicity/+question/189204
>>
>> Status: Answered => Solved
>>
>> Anton confirmed that the question is solved:
>> Sorry , I had not enough local space for trying the local backend.
>> However, I have been able to do a new full backup in a faster server, interrupting it only two times with Ctrl+z, in order to not terminate the process.
>> This backup has been finished ok, without memory problems after aplying your patch.
>> Also, I've tried to do a incremental backup after the full, and modified files were detected ok, with no strange deletions.
>>
>> The only problem that remains is the one with the ssh with no password.
>> Strange here is that I'm able to ssh without pass from the same
>> terminal, and same keys and configuration works ok from a linux vm.
>> Perhaps a macports issue with no further relevance for me
>>
>
> Could you please post your complete command line and pastebin the console output of a run with -v9 ?
> Maybe we can see why duplicitry is asking for a password.
> What is your setup keyauth? PASSPHRASE var?
>
> You could make a simple testbackup of some testfolder (might save you
> some obfuscating).
>

Also, please try 0.6.18, which has a complete new ssh backend, which is not relying on system ssh binaries anymore.

..ede/duply,net

Hello again,
Command is: duply test backup
previously y tested that ssh username@servername logged in without pass
full log pastebin is at: http://pastebin.com/tTp1Rrfe
It asks for password with:
Password for 'username@servername':

I tested with the next configurations in .duply/test/conf:
1:
GPG_PW='111'
2:
GPG_KEY='disabled'
GPG_PW='111'
3:
GPG_KEYS_ENC='1B1C395C'
GPG_PW=''

With all of them the pass was asked for

On 06.03.2012 15:11, Anton wrote:
> Question #189204 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/189204
>
> Status: Solved => Open
>
> Anton is still having a problem:
> Hello again,
> Command is: duply test backup
> previously y tested that ssh username@servername logged in without pass
> full log pastebin is at: http://pastebin.com/tTp1Rrfe
> It asks for password with:
> Password for 'username@servername':
>
> I tested with the next configurations in .duply/test/conf:
> 1:
> GPG_PW='111'
> 2:
> GPG_KEY='disabled'
> GPG_PW='111'
> 3:
> GPG_KEYS_ENC='1B1C395C'
> GPG_PW=''
>
> With all of them the pass was asked for
>

ok, you're suffering from a duply bug. see changelog
http://duply.net/?title=Duply-Changelog
search for 'askpass'

update to the latest version and everything should be swell.

..ede/duply.net

Ok,

I've updated duply to 1.5.5.5 and problem was fully solved.
I've also filed a ticket in MacPorts with diff in portfile to upgrade to that release, so that maintainer upgrades the port.

Thanks a lot
Anton