OSError: [Errno 24] Too many open files

On 27.11.2011 10:20, Rodrigo Alvarez wrote:
> New question #180135 on Duplicity:
> https://answers.launchpad.net/duplicity/+question/180135
>
> A perfectly good backup set suddenly goes wrong and verification reports "OSError: [Errno 24] Too many open files" during the attempt.
>
> running ulimit -n returns: 8192
>
> Just last month this backup set passed my monthly verification test with flying colors but now it is somehow corrupted and throwing this error. What concerns me the most is that this is not the first time it happens nor the only set. Something in the backup process is corrupting my sets.
>
> To rule out hard disk corruption I compare (on a monthly basis) my backup volumes with an externally mirrored set of backup volumes (diff -rqs source target) and this does not flag any disk changes.
>
> In the past I've fixed this by simply biting the bullet and deleting some dates from my backup set until I could verify the set again but this involves a lot of guesswork to find the date at which the set became corrupted. Can you please include extra feedback in duplicity so that when "OSError: [Errno 24] Too many open files" is triggered I can tell which volume and what date was being accessed? This as a first step to try to debug why backup sets are being corrupted.
>

which duplicity version do you use?

what is your operating system?

this comes up from time to time. read e.g. in the mailing list
http://lists.nongnu.org/archive/cgi-bin/namazu.cgi?query=too+many+open+files&idxname=duplicity-talk

good news is: your backup is _not_ corrupt. in short, you probably have a very long backup chain and your python/os combination does either
- not perfectly clean up open files used in duplicity
or
- not fast enough

try raising the open files limit on your platform and/or shorten your backup chains. i for example use monthly fulls and daily backups.

good luck ede/duply.net

Thanks Edso, I increased the file limit and it successfully verified.

I do have a long backup chain. To me this is the essence of incremental backups.

This is another discussion but I've been spoiled by the TimeMachine that automatically prunes old backups. It would be great if duplicity did something similar. I would love the following functionality:
1. Start with a full backup.
2. Follow with daily backups.
3. At the end of the month, do a monthly backup. A backup that stores the diff from day one to day 30, so that I could then delete all the daily backups.
4. Three months down the line delete the old daily backups.

This way, we keep long-lasting chains with relatively short depth without sacrificing daily backups.

I know I could do this in a script, but it'd be really good if duplicity could handle this internally....

Thanks again,

+R