Bazaar

Problem with large PDF

Asked by TomekPaczkowski

I have large PDF (50MB) that I want store in my bzr repo, but when adding it, it's marked as normal text file, which in result couses MemoryError on commit:
Committing to: /srv/bzr/epp/
bzr: ERROR: exceptions.MemoryError:

Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 846, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 797, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 499, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 2364, in run
    author=author)
  File "/usr/lib/python2.5/site-packages/bzrlib/decorators.py", line 165, in write_locked
    return unbound(self, *args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/workingtree_4.py", line 240, in commit
    result = WorkingTree3.commit(self, message, revprops, *args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/decorators.py", line 165, in write_locked
    return unbound(self, *args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/mutabletree.py", line 197, in commit
    revprops=revprops, *args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/commit.py", line 355, in commit
    self._update_builder_with_changes()
  File "/usr/lib/python2.5/site-packages/bzrlib/commit.py", line 655, in _update_builder_with_changes
    self._populate_from_inventory(specific_files)
  File "/usr/lib/python2.5/site-packages/bzrlib/commit.py", line 783, in _populate_from_inventory
    content_summary)
  File "/usr/lib/python2.5/site-packages/bzrlib/commit.py", line 825, in _record_entry
    self.parent_invs, path, self.work_tree, content_summary)
  File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 361, in record_entry_contents
    ie.file_id, lines, heads, nostore_sha)
  File "/usr/lib/python2.5/site-packages/bzrlib/repofmt/pack_repo.py", line 97, in _add_text_to_weave
    self.random_revid)
  File "/usr/lib/python2.5/site-packages/bzrlib/repofmt/pack_repo.py", line 1180, in _add_text_to_weave
    random_id=random_revid, check_content=False)[0:2]
  File "/usr/lib/python2.5/site-packages/bzrlib/versionedfile.py", line 137, in add_lines_with_ghosts
    parent_texts, nostore_sha, random_id, check_content, left_matching_blocks)
  File "/usr/lib/python2.5/site-packages/bzrlib/knit.py", line 999, in _add_lines_with_ghosts
    parent_texts, left_matching_blocks, nostore_sha, random_id)
  File "/usr/lib/python2.5/site-packages/bzrlib/knit.py", line 1088, in _add
    lines, [line_bytes])
  File "/usr/lib/python2.5/site-packages/bzrlib/knit.py", line 2414, in _record_to_data
    ["end %s\n" % version_id]))
MemoryError

bzr 1.5 on python 2.5.2 (linux2)
arguments: ['/usr/bin/bzr', 'ci', 'static', '-m', 'Added hi-res epp pdf']
encoding: 'UTF-8', fsenc: 'UTF-8', lang: 'en_DK.UTF-8'
plugins:
  bzrtools /usr/lib/python2.5/site-packages/bzrlib/plugins/bzrtools [1.5.0]
  launchpad /usr/lib/python2.5/site-packages/bzrlib/plugins/launchpad [unknown]

I think bzr is trying to put whole PDF in memory (for what reason?).

Is there any solution for that, so i can commit it?

I don't really care about diffs from this file, so perhaps I could mark it as binary?

Question information

Language:
English Edit question
Status:
Solved
For:
Bazaar Edit question
Assignee:
No assignee Edit question
Solved by:
TomekPaczkowski
Solved:
Last query:
Last reply:
Revision history for this message
John A Meinel (jameinel) said : 		#1

I haven't seen memory errors with 50MB files before, are you sure it isn't bigger? Marking it as "binary" wouldn't effect how we commit the data into the repository, so it won't change anything here. (All contents are left unmodified, we just split on '\n' as a convenience.)

I would be curious to know how much memory bzr is consuming at the time you try to commit this file. 100MB wouldn't be too surprising, but I *would* be surprised if it was a lot more than that.

Revision history for this message
TomekPaczkowski (oinopion) said : 		#2

So if I tried to add 1 GB file, I had to have 2 GB of memory (just for python/bzr)?

My problem was that i was committing from my VPS, where only ~100 MB of RAM is available for non-apache processes. I've shut apache down and committed with over 200 MB of RAM under my belt.