Failed backups: /etc/.pwd.lock

Hello,

Can you please confirm what version you are running?
duplicity --version

Why can you not exclude single files?

@Aaron -

Version: 0.7.08

I am using duplicity through the backup utility that is accessible through the Settings in Ubuntu 16.04 (Gnome). That interface does not allow for the exclusion of single files, only directories.

On 06.07.2016 00:02, DiagonalArg wrote:
> Question #295948 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/295948
>
> Status: Needs information => Open
>
> DiagonalArg gave more information on the question:
> @Aaron -
>
> Version: 0.7.08
>
> I am using duplicity through the backup utility that is accessible
> through the Settings in Ubuntu 16.04 (Gnome). That interface does not
> allow for the exclusion of single files, only directories.
>

Diagonal,

there is a workaround for Deja Dup. you might edit the exclusion list via gconf-edit and add the file path there manually
  https://bugs.launchpad.net/deja-dup/+bug/374274/comments/7

generally - if Deja Dup misses functionality you need, consider using duplicity directly or via duply shell frontend.

..ede/duply.net

Interesting, thanks.

The backup program in Ubuntu is called Deja Dup and you are correct that it uses duplicity. I would like to narrow down whether the problem is in Deja Dup or duplicity. Are you able to find the duplicity command that Deja Dup is running? You may be able to find that by running Deja Dup from the terminal. If you can please paste that here, that would be helpful.

Even more helpful would be if you can please copy and paste that command into the terminal and run it directly, to see if that also fails. Please add "--verbosity 9" to the command to increase the messages that duplicity spits out. Please also copy and paste the output here so that we can try to see what's going on.

I made some changes that are included in the 0.7.08 release relating to the errors on locked files, so it would also be interesting to downgrade your duplicity to an earlier version (anything released before June should be fine). I suspect doing that would solve your immediate problem, but it would be great if you could help us solve the problem properly. Once I have enough information to replicate the issue at my end, I should be able to fix it for you.

Ok, I just triggered another backup and copied the command by using `ps ux | grep duplicity`:

/usr/bin/python /usr/bin/duplicity --exclude=/home/user/Data/Backup.T530 --include=/home/user/.cache/deja-dup/metadata --exclude=/etc/apparmor.d --exclude=/etc/sudoers.d --exclude=/etc/ssl/private --exclude=/etc/postgresql/9.5/main --exclude=/etc/security --exclude=/etc/ppp --exclude=/etc/ufw --exclude=/etc/cups --exclude=/etc/apparmor --include=/etc --exclude=/home/user/mnt --exclude=/home/user/.cache --exclude=/home/user/Data --exclude=/home/user/T500-etc --exclude=/home/user/T530-etc --exclude=/home/user/.local/share/Trash --exclude=/home/user/.cache/deja-dup/tmp --exclude=/home/user/.thumbnails --exclude=/home/user/.adobe/Flash_Player/AssetCache --exclude=/home/user/.cache/deja-dup --exclude=/home/user/.cache --include=/home/dev --exclude=/sys --exclude=/run --exclude=/proc --exclude=/var/tmp --exclude=/tmp --exclude=** --gio --dry-run --volsize=50 / file:///home/user/Data/Backup.T530 --no-encryption --verbosity=9 --gpg-options=--no-use-agent --archive-dir=/home/user/.cache/deja-dup --tempdir=/home/user/.cache/deja-dup/tmp --log-fd=16

Give me a little time and I'll use that command with `--verbosity 9` and share the output.

Ok, I'm not clear what happened, but that command with `--verbosity 9` and `| tee duplicity.out` crashed right away. That's not the behavior of deja-dup when it runs daily. The output on standard error is:

Warning: Option --gio is pending deprecation and will be removed in a future release.
Use of default filenames is strongly suggested.
Using temporary directory /tmp/duplicity-iAdZew-tempdir
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1546, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1540, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1375, in main
    action = commandline.ProcessCommandLine(sys.argv[1:])
  File "/usr/lib/python2.7/dist-packages/duplicity/commandline.py", line 1085, in ProcessCommandLine
    args = parse_cmdline_options(cmdline_list)
  File "/usr/lib/python2.7/dist-packages/duplicity/commandline.py", line 628, in parse_cmdline_options
    (options, args) = parser.parse_args()
  File "/usr/lib/python2.7/optparse.py", line 1400, in parse_args
    stop = self._process_args(largs, rargs, values)
  File "/usr/lib/python2.7/optparse.py", line 1440, in _process_args
    self._process_long_opt(rargs, values)
  File "/usr/lib/python2.7/optparse.py", line 1515, in _process_long_opt
    option.process(opt, value, values, self)
  File "/usr/lib/python2.7/optparse.py", line 789, in process
    self.action, self.dest, opt, value, values, parser)
  File "/usr/lib/python2.7/dist-packages/duplicity/commandline.py", line 197, in take_action
    self, action, dest, opt, value, values, parser)
  File "/usr/lib/python2.7/optparse.py", line 809, in take_action
    self.callback(self, opt, value, parser, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/duplicity/commandline.py", line 448, in <lambda>
    callback=lambda o, s, v, p: set_log_fd(v))
  File "/usr/lib/python2.7/dist-packages/duplicity/commandline.py", line 242, in set_log_fd
    log.add_fd(fd)
  File "/usr/lib/python2.7/dist-packages/duplicity/log.py", line 403, in add_fd
    handler = logging.StreamHandler(os.fdopen(fd, 'w'))
OSError: [Errno 9] Bad file descriptor

Actually, I see that `--verbosity=9` was already one of the options (closer to the end). So I removed `--verbosity 9` and ran it again. The result was the same - a crash with the above output.

Quick question - When you installed the 0.7 version, did you uninstall the
previous 0.6 version. On some distros the files from a repo install and
the files from a tarball install are in different, and conflicting,
locations.

On Thu, Jul 7, 2016 at 1:37 AM, DiagonalArg <
<email address hidden>> wrote:

> Question #295948 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/295948
>
> DiagonalArg gave more information on the question:
> Actually, I see that `--verbosity=9` was already one of the options
> (closer to the end). So I removed `--verbosity 9` and ran it again.
> The result was the same - a crash with the above output.
>
> --
> You received this question notification because your team duplicity-team
> is an answer contact for Duplicity.
>
> _______________________________________________
> Mailing list: https://launchpad.net/~duplicity-team
> Post to : <email address hidden>
> Unsubscribe : https://launchpad.net/~duplicity-team
> More help : https://help.launchpad.net/ListHelp
>

I didn't install any versions. This is what came with Ubuntu 16.04 Gnome:

Version: 0.7.08-0ubuntu1ppa1227~ubuntu16.04.1

So you understand that while deja-dup is crashing on that locked file after running for quite some time, when I run the duplicity command as shown in `ps ux`, it crashes right away. So something different is happening. But I haven't changed versions of anything.

I used the work-around that you mentioned in #3. (Using dconf editor to exclude the /etc/.pwd.lock file.) Now it's crashing on another file: /etc/NetworkManager/system-connections/<my vpn>.

In Comment #8 you said:
> Actually, I see that `--verbosity=9` was already one of the options (closer to the end). So I removed `--verbosity 9`
> and ran it again. The result was the same - a crash with the above output.

Can you please run the duplicity command (not through Deja Dup) and copy and paste the output here? I'm hopeful it will give a little more contextual information than the error in your post.

Hi DiagonalArg,

> I didn't install any versions. This is what came with Ubuntu 16.04 Gnome:
> Version: 0.7.08-0ubuntu1ppa1227~ubuntu16.04.1

I'm pretty sure that you must have added the duplicity PPA for that version:
https://launchpad.net/~duplicity-team/+archive/ubuntu/ppa

as the version that comes with 16.04 is (http://packages.ubuntu.com/xenial/duplicity ):
0.7.06-2ubuntu2

That doesn't mean that you've done anything wrong, as the behaviour in this area shouldn't have changed between those two versions, but could give you an easy solution if you revert back to the system version.

I've run the latest (trunk) version of duplicity against my /etc/ (which has some locked files) and it gives an error count, but duplicity exits with a zero exit code. I'm trying to replicate this locally, haven't yet.

I have the same problem with deja-dup starting today. I'm on Arch.

I think what changed is that now duplicity fails/stops on permission denied.
Before, at end of backup I was getting a list of files not backed-up, all from /etc. Now it fails on trying to access those files at the beginning of the backup process.

Vlatko/DiagonalArg,

Can one of you please run the command directly in duplicity (DiagonalArg found his command using `ps ux | grep duplicity`) and copy and paste the results here?

I am trying to figure out if duplicity is failing or if duplicity has just changed its output slightly and that is causing (only) deja dup to fail.

This is a bit of a tough one. Technically you have asked duplicity to back up all of /etc/ with no exceptions and duplicity couldn't do that because some of the files could not be read. From duplicity's perspective, it did really fail and the proper way to deal with this at the command line is to just an exception not to back up the files that you don't have access to read.

Historically, however, I believe that this warned on each locked file and gave an error count that said how many files could not be read, but actually exited without error. I didn't intend to change that behaviour with the relevant commit here, to avoid these kinds of problems, so my plan is to make it work the same way that it did before (if I have in fact changed it somehow).

Hi Aron,

I took cmd line with ps, as you suggested, and this is the output when run from shell, as normal user. I'm backing up $HOME and /etc, with some ignored dirs in both.

Warning: Option --gio is pending deprecation and will be removed in a future release.
Use of default filenames is strongly suggested.
Using temporary directory /tmp/duplicity-mKAK6u-tempdir
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1546, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1540, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1375, in main
    action = commandline.ProcessCommandLine(sys.argv[1:])
  File "/usr/lib/python2.7/site-packages/duplicity/commandline.py", line 1085, in ProcessCommandLine
    args = parse_cmdline_options(cmdline_list)
  File "/usr/lib/python2.7/site-packages/duplicity/commandline.py", line 628, in parse_cmdline_options
    (options, args) = parser.parse_args()
  File "/usr/lib/python2.7/optparse.py", line 1400, in parse_args
    stop = self._process_args(largs, rargs, values)
  File "/usr/lib/python2.7/optparse.py", line 1440, in _process_args
    self._process_long_opt(rargs, values)
  File "/usr/lib/python2.7/optparse.py", line 1515, in _process_long_opt
    option.process(opt, value, values, self)
  File "/usr/lib/python2.7/optparse.py", line 789, in process
    self.action, self.dest, opt, value, values, parser)
  File "/usr/lib/python2.7/site-packages/duplicity/commandline.py", line 197, in take_action
    self, action, dest, opt, value, values, parser)
  File "/usr/lib/python2.7/optparse.py", line 809, in take_action
    self.callback(self, opt, value, parser, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/duplicity/commandline.py", line 448, in <lambda>
    callback=lambda o, s, v, p: set_log_fd(v))
  File "/usr/lib/python2.7/site-packages/duplicity/commandline.py", line 242, in set_log_fd
    log.add_fd(fd)
  File "/usr/lib/python2.7/site-packages/duplicity/log.py", line 403, in add_fd
    handler = logging.StreamHandler(os.fdopen(fd, 'w'))
OSError: [Errno 9] Bad file descriptor

> In Comment #8 you said:
>> Actually, I see that `--verbosity=9` was already one of the options (closer to the end). So I removed `--verbosity 9`
>> and ran it again. The result was the same - a crash with the above output.

> Can you please run the duplicity command (not through Deja Dup) and copy and paste the output here? I'm hopeful it will give a little more contextual information than the error in your post.

That's exactly what I did, with the result being the crash.

> Hi DiagonalArg,

>> I didn't install any versions. This is what came with Ubuntu 16.04 Gnome:
>> Version: 0.7.08-0ubuntu1ppa1227~ubuntu16.04.1

> I'm pretty sure that you must have added the duplicity PPA for that version:
> https://launchpad.net/~duplicity-team/+archive/ubuntu/ppa

Oops, yes. You are right.

> Quick question - When you installed the 0.7 version, did you uninstall the
> previous 0.6 version. On some distros the files from a repo install and
> the files from a tarball install are in different, and conflicting,
> locations.

Ok, I don't see how to do this. I used `aptitude search duplicity` and see it installed, but I don't see how to identify if more than one version is installed nor how to remove an old one. (Ubuntu 16.04 Gnome.)

(Travelling, so will be slow responding, but still having the problem on this laptop. Finally gave up on backing up /etc, but don't really want to do that.)

Thanks Aaron. :-)

Too old