Duplicity never ending job, lots of instances

What OS and version? Where did you get the 0.7.11 version from and how did you install it? The reason I ask is that the problems you are having, multiple instances of duplicity, and multiple instances of GPG, were all fixed a long time ago. They should not be happening.

If you installed via tarball after using a repo install, you will need to uninstall the repo version and reinstall the tarball version. That should fix the obvious problems.

OS is Ubuntu 14.04.5 LTS.

0.7.11 was indeed installed through apt.

I will try purging that one and install the tarball version.

Thanks for the quick reply.

On 20.03.2017 20:28, Jasper wrote:
> New question #577611 on Duplicity:
> https://answers.launchpad.net/duplicity/+question/577611
>
> I'm running Duplicity with Duply on a couple servers with S3 as target, and experience the same issue on all servers.
>
> It seems like Duplicity never actually ends its job, resulting in constant high memory usage and 38 Duplicity instances and 673 GPG instances.
>
> duply -v
> duply version 2.0.1
> (http://duply.net)
>
> Using installed duplicity version 0.7.11, python 2.7.6, gpg 1.4.16 (Home: ~/.gnupg), awk 'GNU Awk 4.0.1', grep 'grep (GNU grep) 2.16', bash '4.3.11(1)-release (x86_64-pc-linux-gnu)'.
>
> The Duply profile is nearly identical for each server, here's an example:
>
> GPG_KEY='FOO'
> GPG_PW='FOO'
>
> TARGET='s3://s3-eu-central-1.amazonaws.com/foo-bucket/bar-location'
>
> export AWS_ACCESS_KEY_ID='FOO'
> export AWS_SECRET_ACCESS_KEY='FOO'
>
> # base directory to backup
> SOURCE='/'
>
> # exclude folders containing exclusion file (since duplicity 0.5.14)
> # Uncomment the following two lines to enable this setting.
> FILENAME='.duplicity-ignore'
> DUPL_PARAMS="$DUPL_PARAMS --exclude-if-present '$FILENAME'"
>
> # Time frame for old backups to keep, Used for the "purge" command.
> # see duplicity man page, chapter TIME_FORMATS)
> MAX_AGE=2M
>
> # Number of full backups to keep. Used for the "purge-full" command.
> # See duplicity man page, action "remove-all-but-n-full".
> MAX_FULL_BACKUPS=2
>
> # Number of full backups for which incrementals will be kept for.
> # Used for the "purge-incr" command.
> # See duplicity man page, action "remove-all-inc-of-but-n-full".
> MAX_FULLS_WITH_INCRS=1
>
> # activates duplicity --full-if-older-than option (since duplicity v0.4.4.RC3)
> # forces a full backup if last full backup reaches a specified age, for the
> # format of MAX_FULLBKP_AGE see duplicity man page, chapter TIME_FORMATS
> # Uncomment the following two lines to enable this setting.
> MAX_FULLBKP_AGE=1M
> DUPL_PARAMS="$DUPL_PARAMS --full-if-older-than $MAX_FULLBKP_AGE "
>
> # sets duplicity --volsize option (available since v0.4.3.RC7)
> # set the size of backup chunks to VOLSIZE MB instead of the default 25MB.
> # VOLSIZE must be number of MB's to set the volume size to.
> # Uncomment the following two lines to enable this setting.
> VOLSIZE=100
> DUPL_PARAMS="$DUPL_PARAMS --volsize $VOLSIZE "
>
> # more duplicity command line options can be added in the following way
> # don't forget to leave a separating space char at the end
> #DUPL_PARAMS="$DUPL_PARAMS --put_your_options_here "
>
> The backups are run via cron:
>
> 12 3 * * * nice -n19 ionice -c2 -n7 duply database backup_verify_purge --force --name foo_database >> /var/log/duplicity.log 2>&1
>
> 26 3 * * * nice -n19 ionice -c2 -n7 duply websites backup_verify_purge --force --name foo_websites >> /var/log/duplicity.log 2>&1
>
> 53 4 * * * nice -n19 ionice -c2 -n7 duply system backup_verify_purge --force --name foo_system >> /var/log/duplicity.log 2>&1
>
>

are you saying that multiple instances for one target are running on the same box?

this should be prevented by the lockfile. anyhow, if for some reason not, that definitely leads to an undefined behaviour of several instances writing to the same target.

so, is it possible that new instances are started while the older one is still running?

..ede/duply.net

Multiple instances are running to S3 as target, but to different buckets. Does that count as the same target?

I removed the apt installed version and installed the tarball, but duply doesn't seem to find it when ran in a cronjob.

Sorry. A fatal ERROR occured:

duplicity missing. installed und available in path?
PATH='/usr/bin:/bin'

which duplicity
/usr/local/bin/duplicity

which duply
/usr/bin/duply

The tarball version did indeed solve the problem, and Duplicity not being available was due to cron not having the correct path.

Many thanks.