Cannot allocate memory with large manifest file

Siva,

also these tickets might be of help

https://answers.launchpad.net/duplicity/+question/270906
https://bugs.launchpad.net/bugs/1684312
https://bugs.launchpad.net/bugs/896728

long story short. start a new backup chain. limit the amount of files/data so that, as memory usage grows with backup size/chain length.

..ede/duply.net

On 26.09.2017 09:15, Sivanath wrote:
> Question #658653 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/658653
>
> Description changed to:
> Hi team
>
> I have used the duplicity for backup my server files from few months
> before. And It was working fine. Currently I have upgraded the
> duplicity new version(0.7.14), after upgraded the backup manifest cache
> file written in large size(near to GBs) due to bulk delete or changes in
> files.
>
> It was stopped the next backup process with this error - "OSError:
> [Errno 12] Cannot allocate memory".
>
> Please help me on this. Is there any way to solve this problem?
>
> Herewith I have placed the logged error information for your reference.
>
> Traceback (most recent call last):
> File "/usr/bin/duplicity", line 1548, in <module>
> with_tempdir(main)
> File "/usr/bin/duplicity", line 1534, in with_tempdir
> fn()
> File "/usr/bin/duplicity", line 1383, in main
> do_backup(action)
> File "/usr/bin/duplicity", line 1515, in do_backup
> check_last_manifest(col_stats) # not needed for full backup
> File "/usr/bin/duplicity", line 1217, in check_last_manifest
> last_backup_set.check_manifests()
> File "/usr/local/lib/python2.7/dist-packages/duplicity/collections.py", line 208, in check_manifests
> remote_manifest = self.get_remote_manifest()
> File "/usr/local/lib/python2.7/dist-packages/duplicity/collections.py", line 241, in get_remote_manifest
> manifest_buffer = self.backend.get_data(self.remote_manifest_name)
> File "/usr/local/lib/python2.7/dist-packages/duplicity/backend.py", line 677, in get_data
> fin = self.get_fileobj_read(filename, parseresults)
> File "/usr/local/lib/python2.7/dist-packages/duplicity/backend.py", line 671, in get_fileobj_read
> return tdp.filtered_open_with_delete("rb")
> File "/usr/local/lib/python2.7/dist-packages/duplicity/dup_temp.py", line 120, in filtered_open_with_delete
> fh = FileobjHooked(path.DupPath.filtered_open(self, mode))
> File "/usr/local/lib/python2.7/dist-packages/duplicity/path.py", line 779, in filtered_open
> return gpg.GPGFile(False, self, gpg_profile)
> File "/usr/local/lib/python2.7/dist-packages/duplicity/gpg.py", line 217, in __init__
> 'logger': self.logger_fp})
> File "/usr/local/lib/python2.7/dist-packages/duplicity/gpginterface.py", line 374, in run
> create_fhs, attach_fhs)
> File "/usr/local/lib/python2.7/dist-packages/duplicity/gpginterface.py", line 414, in _attach_fork_exec
> process.pid = os.fork()
> OSError: [Errno 12] Cannot allocate memory
>

Siva,

are you sure that this is caused by the version upgrade? maybe it is coincidental? i would try two things.

1. make sure that nothing lingers in your ram and there is plenty memory free before you run duplicity.

2. if that doesn't help, downgrade to the version of duplicity you used before and see if the error vanishes.

what's your distro? what version of duplicity did you use before?

..ede/duply.net

On 26.09.2017 09:15, Sivanath wrote:
> Question #658653 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/658653
>
> Description changed to:
> Hi team
>
> I have used the duplicity for backup my server files from few months
> before. And It was working fine. Currently I have upgraded the
> duplicity new version(0.7.14), after upgraded the backup manifest cache
> file written in large size(near to GBs) due to bulk delete or changes in
> files.
>
> It was stopped the next backup process with this error - "OSError:
> [Errno 12] Cannot allocate memory".
>
> Please help me on this. Is there any way to solve this problem?
>
> Herewith I have placed the logged error information for your reference.
>
> Traceback (most recent call last):
> File "/usr/bin/duplicity", line 1548, in <module>
> with_tempdir(main)
> File "/usr/bin/duplicity", line 1534, in with_tempdir
> fn()
> File "/usr/bin/duplicity", line 1383, in main
> do_backup(action)
> File "/usr/bin/duplicity", line 1515, in do_backup
> check_last_manifest(col_stats) # not needed for full backup
> File "/usr/bin/duplicity", line 1217, in check_last_manifest
> last_backup_set.check_manifests()
> File "/usr/local/lib/python2.7/dist-packages/duplicity/collections.py", line 208, in check_manifests
> remote_manifest = self.get_remote_manifest()
> File "/usr/local/lib/python2.7/dist-packages/duplicity/collections.py", line 241, in get_remote_manifest
> manifest_buffer = self.backend.get_data(self.remote_manifest_name)
> File "/usr/local/lib/python2.7/dist-packages/duplicity/backend.py", line 677, in get_data
> fin = self.get_fileobj_read(filename, parseresults)
> File "/usr/local/lib/python2.7/dist-packages/duplicity/backend.py", line 671, in get_fileobj_read
> return tdp.filtered_open_with_delete("rb")
> File "/usr/local/lib/python2.7/dist-packages/duplicity/dup_temp.py", line 120, in filtered_open_with_delete
> fh = FileobjHooked(path.DupPath.filtered_open(self, mode))
> File "/usr/local/lib/python2.7/dist-packages/duplicity/path.py", line 779, in filtered_open
> return gpg.GPGFile(False, self, gpg_profile)
> File "/usr/local/lib/python2.7/dist-packages/duplicity/gpg.py", line 217, in __init__
> 'logger': self.logger_fp})
> File "/usr/local/lib/python2.7/dist-packages/duplicity/gpginterface.py", line 374, in run
> create_fhs, attach_fhs)
> File "/usr/local/lib/python2.7/dist-packages/duplicity/gpginterface.py", line 414, in _attach_fork_exec
> process.pid = os.fork()
> OSError: [Errno 12] Cannot allocate memory
>

Hi edso,

Thanks for your timely response.

When I use the version duplicity-0.7.08 - the manifest files are written in small size, after upgraded the new version (duplicity-0.7.13.1 to duplicity-0.7.14) the manifest file written with large size (near to GB) because every changed/new/deleted files history written in single line.

I this case, once large manifest file created means the next backup process is getting long time to start (read last manifest file) and throw memory allocation issue or simply exit with killed.

Anyway - I follow above your comments. Once find the solution will let you know.

Hi edso

I have tried with new backup chain, once after completed the full backup, the immediate incremental backup getting stopped with "Killed" text. Can you please provide me the any solution for this. Thanks in advance.

Siva,

1.
can you pastebin or put it somewhere online, examples (created w/ 0.7.08,13) on how manifests are suddenly bigger?

2.
can you show me that your ram is really empty between the full/incr runs?
  https://www.cyberciti.biz/faq/linux-check-memory-usage/

3.
are there any lingering gpg processes still in memory between duplicity runs?
  'ps -C gpg,gpg2'

@Ken: any idea?

..ede/duply.net

Yes, there was a change a long while back to add the list of changed files
to the manifest. This allowed us to do collection status with individual
file tracking to see where the file had changed. I think this should have
been an optional list or in a separate database. It seems to be a burden
on small machines.

...Ken

On Thu, Sep 28, 2017 at 4:33 AM, edso <email address hidden>
wrote:

> Question #658653 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/658653
>
> edso requested more information:
> Siva,
>
> 1.
> can you pastebin or put it somewhere online, examples (created w/
> 0.7.08,13) on how manifests are suddenly bigger?
>
> 2.
> can you show me that your ram is really empty between the full/incr runs?
> https://www.cyberciti.biz/faq/linux-check-memory-usage/
>
> 3.
> are there any lingering gpg processes still in memory between duplicity
> runs?
> 'ps -C gpg,gpg2'
>
> @Ken: any idea?
>
> ..ede/duply.net
>
> --
> You received this question notification because your team duplicity-team
> is an answer contact for Duplicity.
>
> _______________________________________________
> Mailing list: https://launchpad.net/~duplicity-team
> Post to : <email address hidden>
> Unsubscribe : https://launchpad.net/~duplicity-team
> More help : https://help.launchpad.net/ListHelp
>

Ken,

1.
can you pinpoint a version, when that happend? so we can suggest users to stick w/ it until this is solved in a future release?

2.
manifests are still one file right? not splitted like volumes on the backend.

..ede/duply.net

On 28.09.2017 12:19, Kenneth Loafman wrote:
> Question #658653 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/658653
>
> Kenneth Loafman proposed the following answer:
> Yes, there was a change a long while back to add the list of changed files
> to the manifest. This allowed us to do collection status with individual
> file tracking to see where the file had changed. I think this should have
> been an optional list or in a separate database. It seems to be a burden
> on small machines.
>
> ...Ken
>
>
> On Thu, Sep 28, 2017 at 4:33 AM, edso <email address hidden>
> wrote:
>
>> Question #658653 on Duplicity changed:
>> https://answers.launchpad.net/duplicity/+question/658653
>>
>> edso requested more information:
>> Siva,
>>
>> 1.
>> can you pastebin or put it somewhere online, examples (created w/
>> 0.7.08,13) on how manifests are suddenly bigger?
>>
>> 2.
>> can you show me that your ram is really empty between the full/incr runs?
>> https://www.cyberciti.biz/faq/linux-check-memory-usage/
>>
>> 3.
>> are there any lingering gpg processes still in memory between duplicity
>> runs?
>> 'ps -C gpg,gpg2'
>>
>> @Ken: any idea?
>>
>> ..ede/duply.net
>>
>> --
>> You received this question notification because your team duplicity-team
>> is an answer contact for Duplicity.
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~duplicity-team
>> Post to : <email address hidden>
>> Unsubscribe : https://launchpad.net/~duplicity-team
>> More help : https://help.launchpad.net/ListHelp
>>
>

Edso and Ken,

Yes. This is my issues - once after creating large manifest file (lot of new, changed and deleted entries in it), the immediate next backup process was stopped due to memory allocation problem or simply skip with "killed" text. Please help me to solve this issue. And Is there any option skip write these logs? Great Thanks for your help.

A couple of answers:

No, the sigtar and manifest files are not split yet.

Version 0.7.03, 05/11/2015, was the first release with the --file-changed
option, so backing off to 0.7.02 should solve your problem.

On Thu, Sep 28, 2017 at 6:38 AM, Sivanath <
<email address hidden>> wrote:

> Question #658653 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/658653
>
> Sivanath posted a new comment:
> Edso and Ken,
>
> Yes. This is my issues - once after creating large manifest file (lot of
> new, changed and deleted entries in it), the immediate next backup
> process was stopped due to memory allocation problem or simply skip with
> "killed" text. Please help me to solve this issue. And Is there any
> option skip write these logs? Great Thanks for your help.
>
> --
> You received this question notification because your team duplicity-team
> is an answer contact for Duplicity.
>
> _______________________________________________
> Mailing list: https://launchpad.net/~duplicity-team
> Post to : <email address hidden>
> Unsubscribe : https://launchpad.net/~duplicity-team
> More help : https://help.launchpad.net/ListHelp
>

Hi Ken

Thanks for your solution.

I'll note that this is not just a "small machines" problem. I'm getting the same OutofMemory error with 8 GBs of physical RAM and an additional 14 GB of swap.

The latest duplicity file I can find is in my `~/.cache/deja-dup` directory has a file size of about 600 MB. It's some sort of encrypted temporary file.

I'll have to switch to a new backup solution if this isn't resolved soon.

FYI - chasing one other bug, but the fix for this has been committed. It will be released with 0.7.15, probably next week. You can DL the fix now from the trunk or the PPA.