Bazaar

Branch size dramatically increased

Asked by David Roth

I'm the maintainer of chive, a open source mysql management solution, available here: http://launchpad.net/chive

Something strange happend with our branch size: It dramatically increased from 7,6MB (lp:chive/1.0) to 67MB (lp:chive).
There are only a handful commits and merges between these two branches, so not something notably.

You can verify this by yourself:

1) bzr checkout lp:chive/1.0 => du -sh .bzr => 7,6MB
2) bzr checkout lp:chive => d -sh .bzr => 67MB

Any ideas what may caused this drastic increase of storage size?
Is it possible to somehow analyize the .pack files in the .bzr/repository directory?

Question information

Language:
English Edit question
Status:
Solved
For:
Bazaar Edit question
Assignee:
No assignee Edit question
Solved by:
Jelmer Vernooij
Solved:
Last query:
Last reply:
Revision history for this message
John A Meinel (jameinel) said : 		#1

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 4/25/2012 9:25 PM, David Roth wrote:
> New question #194724 on Bazaar:
> https://answers.launchpad.net/bzr/+question/194724
>
> I'm the maintainer of chive, a open source mysql management
> solution, available here: http://launchpad.net/chive
>
> Something strange happend with our branch size: It dramatically
> increased from 7,6MB (lp:chive/1.0) to 67MB (lp:chive). There are
> only a handful commits and merges between these two branches, so
> not something notably.
>
> You can verify this by yourself:
>
> 1) bzr checkout lp:chive/1.0 => du -sh .bzr => 7,6MB 2) bzr
> checkout lp:chive => d -sh .bzr => 67MB
>
> Any ideas what may caused this drastic increase of storage size? Is
> it possible to somehow analyize the .pack files in the
> .bzr/repository directory?
>

I can confirm the size difference. The specific revision which bloated
was rev 428.

Looking closely at the below log, I do see this comment:
       421.1.16 mburtscher 2012-03-20
                Updated export > 3.8M!!!
                M pharExport.php

I wonder if there was just a bunch of churn on pharExport.php? Or
maybe on an associated file, that then got reverted back and forth a
bit...

'bzr log -r 427..428 -v' shows this:
  428 David Roth 2012-04-14 [merge]
      Merged from lp:~mburtscher/chive/phar
      A pharExport.php
      A protected/components/phar/
      A protected/components/phar/PharAssetManager.php
      A protected/components/phar/PharHttpRequest.php
      A protected/config/phar.php
      A protected/controllers/AssetController.php
      M archiveExport.sh
      M images/icons/fugue/16/home.png*
      M images/icons/fugue/16/logout.png*
      M images/icons/fugue/16/refresh.png*
      M index.php
      M protected/components/messages/CXmlMessageSource.php
      M protected/controllers/SiteController.php

        421.1.1 mburtscher 2012-03-11
                All modifications needed to run chive in a single-file
phar installation... roXx!!!
                A chive.phar
                A protected/config/phar.php
                A protected/controllers/AssetController.php
                A stub.php
                M index.php

        421.1.2 mburtscher 2012-03-11
                Refactor ...
                A pharExport.sh
                M chive.phar

        421.1.3 mburtscher 2012-03-12
                Fixed base url of asset manager
                Automatically detect whether we are in a phar or not ...
                D chive.phar
                D stub.php
                A chive.phar.php
                M index.php
                M pharExport.sh
                M protected/config/phar.php

        421.1.4 mburtscher 2012-03-12
                fixed
                M chive.phar.php
                M pharExport.sh

        421.1.5 mburtscher 2012-03-19
                Necessary changes ...
                A protected/components/phar/
                A protected/components/phar/PharAssetManager.php
                A protected/components/phar/PharHttpRequest.php
                M chive.phar.php
                M index.php
                M pharExport.sh
                M protected/config/phar.php
                M protected/controllers/AssetController.php

        421.1.6 mburtscher 2012-03-19
                Abstract realpath() ... *ugly*
                M index.php
                M protected/components/ChiveHttpSession.php
                M yii/base/CApplication.php
                M yii/base/CModule.php
                M yii/cli/commands/MessageCommand.php
                M yii/cli/commands/ShellCommand.php
                M yii/cli/commands/WebAppCommand.php
                M yii/console/CConsoleApplication.php
                M yii/logging/CFileLogRoute.php
                M yii/web/CAssetManager.php
                M yii/web/CHttpRequest.php
                M yii/web/CThemeManager.php
                M yii/web/CWebApplication.php
                M yii/web/CWebModule.php
                M yii/web/renderers/CViewRenderer.php

        421.1.7 mburtscher 2012-03-19
                Rename phar export ...
                D pharExport.sh
                A pharExport.php
                M archiveExport.sh

        421.1.8 mburtscher 2012-03-19
                Better export ...
                D chive.phar.php
                M archiveExport.sh
                M pharExport.php

        421.1.9 mburtscher 2012-03-19
                Export ...
                M archiveExport.sh
                M pharExport.php

       421.1.10 mburtscher 2012-03-19
                Fix + fast export

       421.1.11 mburtscher 2012-03-19
                Fixes
                M archiveExport.sh
                M index.php
                M pharExport.php

       421.1.12 mburtscher 2012-03-20
                Fixed login screen not showing any languages
                Remove i18n data automatically at phar creation
                M pharExport.php
                M protected/controllers/SiteController.php

       421.1.13 mburtscher 2012-03-20
                Test
                M index.php
                M pharExport.php

       421.1.14 mburtscher 2012-03-20
                ...
                M index.php

       421.1.15 mburtscher 2012-03-20
                Fixed language selection working in phar too
                Updated export script
                M archiveExport.sh
                M pharExport.php
                M protected/components/messages/CXmlMessageSource.php
                M protected/controllers/SiteController.php

       421.1.16 mburtscher 2012-03-20
                Updated export > 3.8M!!!
                M pharExport.php

       421.1.17 mburtscher 2012-03-20 [merge]
                Merged 1.0.3
                M protected/components/export/CsvExporter.php
                M protected/config/main.php
                M protected/messages/en/core.xml
                M protected/models/Row.php
                M protected/views/site/index.php

       421.1.18 mburtscher 2012-03-22
                Get rid of realpath_chive() by replacing realpath() on
phar creation.
                M index.php
                M pharExport.php
                M yii/base/CApplication.php
                M yii/base/CModule.php
                M yii/cli/commands/MessageCommand.php
                M yii/cli/commands/ShellCommand.php
                M yii/cli/commands/WebAppCommand.php
                M yii/console/CConsoleApplication.php
                M yii/logging/CFileLogRoute.php
                M yii/web/CAssetManager.php
                M yii/web/CHttpRequest.php
                M yii/web/CThemeManager.php
                M yii/web/CWebApplication.php
                M yii/web/CWebModule.php
                M yii/web/renderers/CViewRenderer.php

       421.1.19 mburtscher 2012-03-22
                Fixed ChiveHttpSession not to use realpath_chive.
                Improved CXmlMessageSource with use of CFileHelper.
                M protected/components/ChiveHttpSession.php
                M protected/components/messages/CXmlMessageSource.php

       421.1.20 mburtscher 2012-03-22
                Fixed phar export ...
                M pharExport.php

       421.1.21 mburtscher 2012-03-22 [merge]
                Merged JSMin
                M pharExport.php

            421.2.1 mburtscher 2012-03-21
                    Disable debug
                    Do minify for all js-files
                    M index.php
                    M pharExport.php

       421.1.22 mburtscher 2012-03-22
                Add license text
                M protected/components/phar/PharAssetManager.php
                M protected/components/phar/PharHttpRequest.php

       421.1.23 mburtscher 2012-03-22
                another one ...
                M protected/controllers/AssetController.php

       421.1.24 mburtscher 2012-04-05 [merge]
                Merge from trunk
                Use Google Closure for phar creation
                D js/components/
                D js/components/EditArea/
                D js/components/EditArea/edit_area_full.js
                D js/components/EditArea/fusonic_extensions/
                D js/components/EditArea/fusonic_extensions/editarea.js
                D js/components/EditArea/images/
                D js/components/EditArea/images/autocompletion.gif
                D js/components/EditArea/images/close.gif
                D js/components/EditArea/images/fullscreen.gif
                D js/components/EditArea/images/go_to_line.gif
                D js/components/EditArea/images/help.gif
                D js/components/EditArea/images/highlight.gif
                D js/components/EditArea/images/load.gif
                D js/components/EditArea/images/move.gif
                D js/components/EditArea/images/newdocument.gif
                D js/components/EditArea/images/opacity.png
                D js/components/EditArea/images/processing.gif
                D js/components/EditArea/images/redo.gif
                D js/components/EditArea/images/redo.png
                D js/components/EditArea/images/reset_highlight.gif
                D js/components/EditArea/images/save.gif
                D js/components/EditArea/images/search.gif
                D js/components/EditArea/images/smooth_selection.gif
                D js/components/EditArea/images/spacer.gif
                D js/components/EditArea/images/statusbar_resize.gif
                D js/components/EditArea/images/undo.gif
                D js/components/EditArea/images/undo.png
                D js/components/EditArea/images/word_wrap.gif
                D js/components/EditArea/langs/
                D js/components/EditArea/langs/bg.js
                D js/components/EditArea/langs/cs.js
                D js/components/EditArea/langs/de.js
                D js/components/EditArea/langs/dk.js
                D js/components/EditArea/langs/en.js
                D js/components/EditArea/langs/eo.js
                D js/components/EditArea/langs/es.js
                D js/components/EditArea/langs/fi.js
                D js/components/EditArea/langs/fr.js
                D js/components/EditArea/langs/hr.js
                D js/components/EditArea/langs/it.js
                D js/components/EditArea/langs/ja.js
                D js/components/EditArea/langs/mk.js
                D js/components/EditArea/langs/nl.js
                D js/components/EditArea/langs/pl.js
                D js/components/EditArea/langs/pt.js
                D js/components/EditArea/langs/ru.js
                D js/components/EditArea/langs/sk.js
                D js/components/EditArea/langs/zh.js
                D js/components/EditArea/license_apache.txt
                D js/components/EditArea/license_bsd.txt
                D js/components/EditArea/license_lgpl.txt
                D js/components/EditArea/reg_syntax/
                D js/components/EditArea/reg_syntax/basic.js
                D js/components/EditArea/reg_syntax/brainfuck.js
                D js/components/EditArea/reg_syntax/c.js
                D js/components/EditArea/reg_syntax/coldfusion.js
                D js/components/EditArea/reg_syntax/cpp.js
                D js/components/EditArea/reg_syntax/css.js
                D js/components/EditArea/reg_syntax/html.js
                D js/components/EditArea/reg_syntax/java.js
                D js/components/EditArea/reg_syntax/js.js
                D js/components/EditArea/reg_syntax/pas.js
                D js/components/EditArea/reg_syntax/perl.js
                D js/components/EditArea/reg_syntax/php.js
                D js/components/EditArea/reg_syntax/python.js
                D js/components/EditArea/reg_syntax/robotstxt.js
                D js/components/EditArea/reg_syntax/ruby.js
                D js/components/EditArea/reg_syntax/sql.js
                D js/components/EditArea/reg_syntax/tsql.js
                D js/components/EditArea/reg_syntax/vb.js
                D js/components/EditArea/reg_syntax/xml.js
                D protected/components/ChiveWebApplication.php
                D protected/components/EditArea.php
                D protected/components/SqlEditor.php
                D themes/standard/css/edit_area.css
                A js/ace/
                A js/ace/ace.js
                A js/ace/mode-sql.js
                A js/ace/theme-chive-uncompressed.js
                A protected/components/AceEditor.php
                M archiveExport.sh
                M index.php
                M js/chive.js
                M js/views/global/browse.js
                M pharExport.php
                M protected/components/helpers/AjaxResponse.php
                M protected/controllers/BookmarkController.php
                M protected/controllers/ColumnController.php
                M protected/controllers/Controller.php
                M protected/controllers/IndexController.php
                M protected/controllers/InformationController.php
                M protected/controllers/PrivilegesController.php
                M protected/controllers/RoutineController.php
                M protected/controllers/RowController.php
                M protected/controllers/SchemaController.php
                M protected/controllers/TableController.php
                M protected/controllers/TriggerController.php
                M protected/controllers/ViewController.php
                M protected/models/ImportPage.php
                M protected/views/global/browse.php
                M protected/views/information/about.php
                M protected/views/layouts/schema.php
                M themes/standard/css/component.css
                M yii/yii.php

       421.1.25 mburtscher 2012-04-05
                changes ...
                M index.php
                M pharExport.php

       421.1.26 mburtscher 2012-04-05
                optimizations ...
                M archiveExport.sh
                M pharExport.php
                M protected/components/messages/CXmlMessageSource.php

       421.1.27 mburtscher 2012-04-05
                removed xml optimization, does not work well
                M pharExport.php
                M protected/components/messages/CXmlMessageSource.php

       421.1.28 mburtscher 2012-04-05
                Another XML message try ...
                M pharExport.php
                M protected/components/messages/CXmlMessageSource.php

       421.1.29 mburtscher 2012-04-05
                asdf
                M protected/components/messages/CXmlMessageSource.php

       421.1.30 mburtscher 2012-04-05
                Remove executable flag from icons ...
                M images/icons/fugue/16/home.png*
                M images/icons/fugue/16/logout.png*
                M images/icons/fugue/16/refresh.png*

       421.1.31 mburtscher 2012-04-05
                Currently at 2.6M :-)
                M pharExport.php

       421.1.32 mburtscher 2012-04-06
                Removed copying pharExport script explicitely
                M archiveExport.sh

I don't know specifically why. I'll note that if I manually run 'bzr
pack --clean-obsolete-packs' it drops from about 60 down to about 33.

My best guess is that you have some big files that changed a few
times, and get stored a bit loose without repack.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk+YhZAACgkQJdeBCYSNAAMcIQCgs32P+xCuN/ck0vRfCjjh62WB
cTYAoLLcCcqO6/iYE3lj0l0GlLBD+fB/
=ysyj
-----END PGP SIGNATURE-----

Revision history for this message
Jelmer Vernooij (jelmer) said : 		#2

Hi David,

On Wed, Apr 25, 2012 at 07:25:44PM -0000, David Roth wrote:
> New question #194724 on Bazaar:
> https://answers.launchpad.net/bzr/+question/194724

> I'm the maintainer of chive, a open source mysql management solution, available here: http://launchpad.net/chive

> Something strange happend with our branch size: It dramatically increased from 7,6MB (lp:chive/1.0) to 67MB (lp:chive).
> There are only a handful commits and merges between these two branches, so not something notably.

> You can verify this by yourself:

> 1) bzr checkout lp:chive/1.0 => du -sh .bzr => 7,6MB
> 2) bzr checkout lp:chive => d -sh .bzr => 67MB

> Any ideas what may caused this drastic increase of storage size?
> Is it possible to somehow analyize the .pack files in the .bzr/repository directory?
I just had a quick look at lp:chive.

This is happening because of revision 421.1.1 by mburtscher, which
adds a file "chive.phar" that is 19Mb, later updated to another file
that is 28Mb, etc etc.

Cheers,

Jelmer

Revision history for this message
David Roth (davrot) said : 		#3

Hi Jelmer,

thanks! Aahh I should have seen this! ;)
It was also my first guess but I did not see it the first time :D (and I did not expect it either) ...

I guess the only way to get rid of this huge file-history is to manually revert all changes to the revision before the merge from chive-phar happened, and then checkin all files again, but without a real merge and without the blobs?!

Or is there a way to only remove certain files in the history, but without loosing all other changes/history-entries?

Cheers,
David

Revision history for this message
Best Jelmer Vernooij (jelmer) said : 		#4

Hi David,

On Thu, Apr 26, 2012 at 06:10:52AM -0000, David Roth wrote:
> thanks! Aahh I should have seen this! ;)
> It was also my first guess but I did not see it the first time :D (and I did not expect it either) ...
>
> I guess the only way to get rid of this huge file-history is to manually
> revert all changes to the revision before the merge from chive-phar
> happened, and then checkin all files again, but without a real merge and
> without the blobs?!
 Yes.

> Or is there a way to only remove certain files in the history, but
> without loosing all other changes/history-entries?
You might be able to use fastexport to filter out the problematic
file. I don't have any experience with this myself though. Either way,
this will involve changing your branch in a way that will break "bzr
pull".

Cheers,

Jelmer

Revision history for this message
David Roth (davrot) said : 		#5

Ok, thanks for your feedback.

cheers,
david