Sorry but Bazaar isn't ready yet?

Asked by yohan

I have a 30 gb repo which was on perforce earlier but I thought I'd try something new. First I tried subversion:
Didn't work. There was some kind of fault when running add/commit using windows 7 and apparently there is a hot fix to work it out. Anyways I didn't want to spend time on ironing out details so I tried GIT:
Doesn't work. Sorry but git can't handle large files. It doesn't scale. I tried changing the config settings, but no it can't handle the repo. Git malloc fails whatever I do. 6 cores and 6gb of RAM should be enough for anything.

So..

I tried Bazaar. Excellent GUIs, great crossplatform support, written in python...great first impression, however:

Sorry but, didn't work. Creating a repo worked fine, adding the files worked fine. But. Commiting didn't work. Just as with GIT I got out of memory errors. I understand these errors, and I understand that python32 might have bugs pertaining to this fault, but in the end I don't care. I need something that works. My verdict on Bazaar is that it's not ready yet, and that I think that too much effort has been put on versioning smaller files, i.e source code. And yes I understand that this is of great importance, but for some projects such as games, you need to version everything. And sadly the only revision control system that does the job is perforce.

Will check back at bazaar in a few months or years, hopefully everything will be just as super awesome as is advertised.

What would be so awesome:
1. Install software
2. Add files
3. commit files
4. Push repo to a server

with no errors! Why not add tests for different kinds of repos? Very simple test.

Question information

Language:
English Edit question
Status:
Solved
For:
Bazaar Edit question
Assignee:
No assignee Edit question
Solved by:
yohan
Solved:
Last query:
Last reply:
Revision history for this message
Martin Pool (mbp) said :
#1

Thanks for the feedback, and I'm glad at least you liked some parts of Bazaar. If the Windows error is what I think it is, then installing the msvc runtime should fix it, see <https://bugs.launchpad.net/bugs/632465>.

Our test framework does repeat tests for every repo format so I'm not sure you're refering to there.

Revision history for this message
yohan (yohan-phobicgames) said :
#2

I had that installed, and I've tried the same thing with fresh installs on two seperate computers.

My repo is 30gb, 60000 files, 20000 folders with some files that are 1-2 gb and thousands of tiny files. Does your test framework have this repo format?

Johan

Revision history for this message
Martin Pool (mbp) said :
#3

On 9 October 2010 02:55, yohan <email address hidden> wrote:
> Question #128390 on Bazaar changed:
> https://answers.launchpad.net/bzr/+question/128390
>
>    Status: Answered => Open
>
> yohan is still having a problem:
> I had that installed, and I've tried the same thing with fresh installs
> on two seperate computers.
>
> My repo is 30gb, 60000 files, 20000 folders with some files that are 1-2
> gb and thousands of tiny files. Does your test framework have this repo
> format?

Ah ok, when we say "repository format" we normally mean "repository
data format". We do test on large tree; not normally that large. Is
that all one source tree, or are you trying to use bzr to backup a
whole home directory?

That tree would probably be on the edge of working ok at the moment on
Ubuntu. When we get a distributed 64-bit windows build it may be ok
there too.

--
Martin

Revision history for this message
yohan (yohan-phobicgames) said :
#4

Hi,

I'm using this version of bazaar:
http://launchpad.net/bzr/2.2/2.2.1/+download/bzr-2.2.1-setup.exe

I'm on Windows 7 64.

That is all one source tree. I am not using bazaar to backup my home directory. We are developing a game, so we need to have all assets versioned as well as a large source code base. Why wouldn't such a large source tree work with bazaar? Is it because bazaar is storing delta information like git, so that the whole repo needs to be read when performing actions?

It might seem counterintuitive to use bazaar to version assets but not when you are developing games. We need to be able to rollback the whole source tree to any commit, including game assets. We often have a case where assets have been changed or updated and you need to rollback changes in order to see what went wrong.

If handling large binary files does not work well with the packing algo, it might be a good idea to be able to mark files as being binary and have these packed in a different way, maybe in a similar way to how subversion does it. I actually managed to get subversion to work, but the only tool that seems to scale well with this kind of source base is perforce. If bazaar manages to solve this problem, I think that it would have a major advantage over git. It would at least scale as well as Subversion.

Johan

Revision history for this message
Martin Pool (mbp) said :
#5

No, I understand there's a lot of value in storing assets along with the source for game development. There's been a lot of discussion with this recently, and things are getting better. For instance from irc this morning:

<peitschie> i've just branched our entire repo with the process taking a peak of 2Gb under linux
<peitschie> avg around 500mb-1gb
<peitschie> 6500 revs.... with some largish individual files in it
<peitschie> i couldn't do this in one sitting 3 months ago :O
<peitschie> (this is bzr+svn btw as well)

You can join the list <https://lists.ubuntu.com/mailman/listinfo/bazaar> if you want to talk about this more or test things.

Revision history for this message
yohan (yohan-phobicgames) said :
#6

Might do that, thanks for the info.

Johan

Revision history for this message
J Bridges (jbridges-yahoo) said :
#7

This gives me cause for concern about our Perforce database, which is about 120 GBytes and many, many branches with large and small files, binary as well as text.

Revision history for this message
yohan (yohan-phobicgames) said :
#8

Try creating a repo with it and see if it works, should be interesting to see if you run out of memory. If you don't, please write your tech specs on your computer :).

Revision history for this message
Martin Pool (mbp) said :
#9

J the total size of the repository is not likely to be a problem. The
main limit at the moment is the size of any single file. Either way,
we would like to know if it works or not, and if it fails, to see the
traceback.