taking multiple full backups

Hi Satish,

I understand from your earlier question that you are using the command:
duplicity \
    --num-retries 20 \
    --archive-dir "~/.cache/duplicity/" \
    --tempdir "/tmp" \
    --s3-use-new-style \
    --exclude "**cyrus.squat" \
    --file-prefix-archive "_" \
    --include-globbing-filelist "duplicity.list" \
    --exclude "**" \
    --encrypt-key {removed} \
    --volsize 256 \
    --verbosity 5 \
    --s3-use-multiprocessing \
    ${SOURCE} ${DEST} > ${DAILYLOGFILE} 2>&1

If you take a look at the manual:
http://duplicity.nongnu.org/duplicity.1.html

"If the above is run repeatedly, the first will be a full backup, and subsequent ones will be incremental. To force a full backup, use the full action:

    duplicity full [options] /home/me sftp://<email address hidden>/some_dir

or enforcing a full every other time via --full-if-older-than <time> , e.g. a full every month:

    duplicity --full-if-older-than 1M /home/me sftp://<email address hidden>/some_dir
"

Are you doing any purging of old backups? If you are generating full backups on a regular basis (say every month), you may like to automatically remove old full backups. See the descriptions for the following commands in the man page for more detail:
remove-older-than <time> [--force] <url>
remove-all-but-n-full <count> [--force] <url>
remove-all-inc-of-but-n-full <count> [--force] <url>

You may also like to look at duply (http://duply.net/) which makes maintaining profiles of complex commands (as you are starting to have above) a bit easier, as you just set them up once in a configuration file.

exactly what Aaron said. nothing more, nothing less.. thx!! Aaron.. ede/duply.net

Hi Aaron / Edso,

Thanks for the suggestions, I tried adding your suggestion with Time format but I'm unable to increase the full backup frequency.

Just to brief you about my setup, I have followed the howto from : http://old.blog.phusion.nl/2013/11/11/duplicity-s3-easy-cheap-encrypted-automated-full-disk-backups-for-your-servers/ for setting up duplicity and S3 with duply

According to the howto, the configuration file I would be editing is under /root/.duply --> test --> conf file.

Could you guys guide me in adding / editing the correct location for increasing the full backup frequency.

Thanks
Satish

Hi Satish,

Yes, that sounds like the file you would be editing.

So the first point is that you can manually make it do a full backup by typing "duply test full", but to make it a bit more automatic going forward, perhaps try adding the below to your conf file (you will likely find the below parts in your conf file, but commented out, so you can just remove the "#" to make them active) - obviously you should adjust the 13, 2 and 6 as necessary:
"
# Time frame for old backups to keep, Used for the "purge" command.
# see duplicity man page, chapter TIME_FORMATS)
# defaults to 1M, if not set
MAX_AGE=13M

# Number of full backups to keep. Used for the "purge-full" command.
# See duplicity man page, action "remove-all-but-n-full".
# defaults to 1, if not set
MAX_FULL_BACKUPS=2

# activates duplicity --full-if-older-than option (since duplicity v0.4.4.RC3)
# forces a full backup if last full backup reaches a specified age, for the
# format of MAX_FULLBKP_AGE see duplicity man page, chapter TIME_FORMATS
# Uncomment the following two lines to enable this setting.
MAX_FULLBKP_AGE=6M
DUPL_PARAMS="$DUPL_PARAMS --full-if-older-than $MAX_FULLBKP_AGE "
"

This setup will automatically create a new full backup if it has been more than 6 months since your last backup, so if you run:
duply test backup
it will normally do an incremental backup, unless it is more than 6 months since your last backup, in which case it will do a full backup. Note that the command above will not delete old backups. To do that, add "purge" and "purge-full" to your commandline. So, with the above setup:
duply test backup_purge_purge-full --force
would do an incremental if there had been a recent full backup, a full if it had been more than six months since the last full backup, would delete any old backups that were older than 13 months and would delete the oldest full backups if you had more than two. This is the command that I run in my cron jobs.

If you choose a relatively long period in between backups, you will also want to run:
duply test verify
manually every so often, to check that the chain of backups is able to restore as expected - note that this will require downloading backup files off the remote server.

Hope that helps.

Hi Aaron,

Many thanks for your quick reply, will try that these options and update you.

--Satish

Hello Aaron,

I hope you had a good weekend :),

Regarding the testing, to get the quick results for my setup I have modified the duply conf file as below : 30m , 5, 15m

# Time frame for old backups to keep, Used for the "purge" command.
# see duplicity man page, chapter TIME_FORMATS)
# defaults to 1M, if not set
MAX_AGE=30m

# Number of full backups to keep. Used for the "purge-full" command.
# See duplicity man page, action "remove-all-but-n-full".
# defaults to 1, if not set
MAX_FULL_BACKUPS=5

# activates duplicity --full-if-older-than option (since duplicity v0.4.4.RC3)
# forces a full backup if last full backup reaches a specified age, for the
# format of MAX_FULLBKP_AGE see duplicity man page, chapter TIME_FORMATS
# Uncomment the following two lines to enable this setting.
MAX_FULLBKP_AGE=15m
DUPL_PARAMS="$DUPL_PARAMS --full-if-older-than $MAX_FULLBKP_AGE "
"
Our acutal requirement is we would like to store at all times 2 weeks of full backups and incremental at anyone time – so there will always be just less than 3 weeks of backups at any one time – the backups will then rotate with the oldest full backups and associated incremental been deleted

So to test setup and get the results quick, I have made 30m , 5, 15m . Now the full backup rotates every 15 minutes, it is giving total of 1 full backup and two incremental backup. I was expecting 4 full backups and 1 incremental backup. Please advise / correct my mistake.

Thanks in advance.
Satish

On 16.03.2015 09:36, satish kumar wrote:
> Question #263472 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/263472
>
> Status: Answered => Open
>
> satish kumar is still having a problem:
> Hello Aaron,
>
> I hope you had a good weekend :),
>
> Regarding the testing, to get the quick results for my setup I have
> modified the duply conf file as below : 30m , 5, 15m
>
> # Time frame for old backups to keep, Used for the "purge" command.
> # see duplicity man page, chapter TIME_FORMATS)
> # defaults to 1M, if not set
> MAX_AGE=30m
>
> # Number of full backups to keep. Used for the "purge-full" command.
> # See duplicity man page, action "remove-all-but-n-full".
> # defaults to 1, if not set
> MAX_FULL_BACKUPS=5
>
> # activates duplicity --full-if-older-than option (since duplicity v0.4.4.RC3)
> # forces a full backup if last full backup reaches a specified age, for the
> # format of MAX_FULLBKP_AGE see duplicity man page, chapter TIME_FORMATS
> # Uncomment the following two lines to enable this setting.
> MAX_FULLBKP_AGE=15m
> DUPL_PARAMS="$DUPL_PARAMS --full-if-older-than $MAX_FULLBKP_AGE "
> "
> Our acutal requirement is we would like to store at all times 2 weeks of full backups and incremental at anyone time – so there will always be just less than 3 weeks of backups at any one time – the backups will then rotate with the oldest full backups and associated incremental been deleted
>
> So to test setup and get the results quick, I have made 30m , 5, 15m .
> Now the full backup rotates every 15 minutes, it is giving total of 1
> full backup and two incremental backup. I was expecting 4 full backups
> and 1 incremental backup. Please advise / correct my mistake.
>

hey Satish,

what is your duply crontab entry, incl. time cols?

you read the manpage description for '--full-if-older-than' to understand what it does, right?
 http://duplicity.nongnu.org/duplicity.1.html

..ede/duply.net

Hi Edso,

My crontabe entry is : */5 * * * * env HOME=/root duply test backup_purge_purge-full --force

I have read the manpage description : enforcing a full every other time via --full-if-older-than <time> , e.g. a full every month:

    duplicity --full-if-older-than 1M

I'm little confused with the command --full-if-older-than , Should I use duply test backup --full-if-older-than ?

Thanks
Satish

On 16.03.2015 16:26, satish kumar wrote:
> Question #263472 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/263472
>
> Status: Answered => Open
>
> satish kumar is still having a problem:
> Hi Edso,
>
> My crontabe entry is : */5 * * * * env HOME=/root duply test
> backup_purge_purge-full --force

ok, your doing backups every 5 min

your doing 2 kinds of purges.. choose one of them and stick with it, either

purge (this purges everything older than time setting, except it is still needed eg. based on an older full)

 or

purge-full (here you can simply say i always want to keep a number of fulls and their respective incrementals)

let me suggest to you to verify after backups eg.

duply test backup_verify_purge --force

> I have read the manpage description : enforcing a full every other time
> via --full-if-older-than <time> , e.g. a full every month:
>
> duplicity --full-if-older-than 1M
>
> I'm little confused with the command --full-if-older-than , Should I use
> duply test backup --full-if-older-than ?

you already do via your duply conf file ;)

wrt. your test setup and your result:

it sound's perfectly reasonable if you make a backup every 5 min but advise duplicity to make a full if the last full is older that 15min that you will end up with 1 full and 2 incr in 15min.. if you disable the purging this pattern would continue..

understood? ..ede/duply.net

Thanks Edso,

So If I understood correctly, I should be using the duply test backup_verify_purge-full --force ( This would be my requirement, as I want to keep number of fulls and their respective incrementals) i.e 2 fulls and 1 incr in 15 min.

--Satish

not sure what you mean.

i understood you wanted to have a new full every 2 weeks and incrementals in between, is that right?

the backup frequency should be 15min? that is quite short.

..ede/duply.net

Hi Edso,

Your correct, we would like to store at all times 2 weeks of full backups and incremental at anyone time ( This would be my acutal setup ) But this would require us to wait for 2 weeks to get the results.

Instead of waiting for weeks, I thought of making the test setup with 4 full backup and 1 incremental with : 30 , 5, 15

# Time frame for old backups to keep, Used for the "purge" command.
# see duplicity man page, chapter TIME_FORMATS)
# defaults to 1M, if not set
MAX_AGE=30m

# Number of full backups to keep. Used for the "purge-full" command.
# See duplicity man page, action "remove-all-but-n-full".
# defaults to 1, if not set
MAX_FULL_BACKUPS=5

# activates duplicity --full-if-older-than option (since duplicity v0.4.4.RC3)
# forces a full backup if last full backup reaches a specified age, for the
# format of MAX_FULLBKP_AGE see duplicity man page, chapter TIME_FORMATS
# Uncomment the following two lines to enable this setting.
MAX_FULLBKP_AGE=15m
DUPL_PARAMS="$DUPL_PARAMS --full-if-older-than $MAX_FULLBKP_AGE "

Crontab entry :

*/5 * * * * env HOME=/root duply test backup_verify_purge-full --force

If this is successfull I would change the value in my actaul setup. Basically we would need 4 full backup and one incremental.

Thanks
Satish

Satish,

Try commenting out the MAX_AGE and see what happens. I suspect that the issue is that this is set to 30 mins. Because you are creating a new full every 15 mins, four full backups would mean that the oldest is at least 60 mins old - and your MAX_AGE is telling duply to delete them after 30 mins.

As Edso says, you want to be using either the max full backups (combined with the max full backup age) OR the max age parameter to govern when old backups are deleted. In your example setup, your max full backups * max fullbkp age = 60 mins, which is longer than your max age, so the shorter max age is probably forcing a delete.

Alternatively, you should probably be able to achieve your four fulls by decreasing your max fullbkp age to 6 mins or something similar. As a side note - you mention wanting only one incremental. I don't think that there is a way (using the parameters above, at least) of deleting the incrementals from between the full backups, but leaving the old full backups. What I'm describing would leave the four full backups and the incrementals that followed each of them.

Aaron

Hi Aaron,

Thank you for the explanation, above configuration was for the test setup.

My production servers requirenment would be to store at all times 2 weeks of full backups and incremental at anyone time – so there will always be just less than 3 weeks of backups at any one time – the backups will then rotate with the oldest full backups and associated incremental been deleted.

Please suggest a configuration for my server requirenment.

I have configured the below settings on production server.

# Time frame for old backups to keep, Used for the "purge" command.
# see duplicity man page, chapter TIME_FORMATS)
# defaults to 1M, if not set
MAX_AGE=2M

# Number of full backups to keep. Used for the "purge-full" command.
# See duplicity man page, action "remove-all-but-n-full".
# defaults to 1, if not set
MAX_FULL_BACKUPS=8

# activates duplicity --full-if-older-than option (since duplicity v0.4.4.RC3)
# forces a full backup if last full backup reaches a specified age, for the
# format of MAX_FULLBKP_AGE see duplicity man page, chapter TIME_FORMATS
# Uncomment the following two lines to enable this setting.
MAX_FULLBKP_AGE=1W
DUPL_PARAMS="$DUPL_PARAMS --full-if-older-than $MAX_FULLBKP_AGE "

My crontab entry :
0 */6 * * * env HOME=/root duply backups backup_purge-full --force

Thanks in advance
Satish

This question was expired because it remained in the 'Open' state without activity for the last 15 days.