Zim

What are the limitations on size of wiki & how to maintain performance??

Asked by Jon

Hi, I have been using Zim successfully (and gratefully!) for the last few months, and have a handful of wikis. They are only a few hundred pages in total, but growing.

So my question is to the devs/ experts and/or anyone using Zim who has experienced large datasets. i.e. thousands of pages, large pages, attachments, deep trees, extensive use of links etc - i.e. calling all power users...

E.g. I observe the duration of time to perform a search is getting longer - almost approaching 'shall I make a coffee' long. (Would be nice to have a progress bar #wishlist)
E.g. 2 (some might consider a bug) The tree often takes a while to refresh after moving or deleting a page. Sometimes so slow I worry I might cause corruption by trying to perform other actions until the engine has caught up.
E.g. 2.5 (really is a bug) Sometimes parts of the tree disappear after a big reorganisation (a restart of the app fixes this).

I am interested in any tricks to improve performance (aka errors to avoid) in the structure of an 'ideal' wiki. And especially how to maintain stability to avoid corruption etc. Can the devs point to any obvious no-nos?

e.g. what number of pages would be advisable to break out into separate wikis? 1k, 10k, 100k??
e.g. 2 Does attachment size/type (total wiki size on disk) matter?

My experience with my newish win10 PC is that the bottlenecks I experience generally (not just Zim) are never resolved by more processor (Intel i7 hardly gets above 20% on an average day). Why doesn't processor utilisation ramp up e.g. when searching? (A typical layman question, but why did i pay for that 'power'??)

My vanilla memory is nothing special in terms of speed I guess (and 8GB is half used due to the heavy payload of bloat in win10). But is at least upgradable: Offtopic slightly but does anyone think 'of course, essential', or 'don't waste your money'?

For reference I imagine a fully loaded wiki might be 30*30*30*2k = 27k pages = 54MB of pure text.
If it had media attachments there might be 10GB of images say.

Thanks in advance!

Question information

Language:
English Edit question
Status:
Solved
For:
Zim Edit question
Assignee:
No assignee Edit question
Solved by:
Jon
Solved:
Last query:
Last reply:
Revision history for this message
Jaap Karssenberg (jaap.karssenberg) said : 		#1

See Performance large notebook · zim-desktop-wiki/zim-desktop-wiki Wiki ·
GitHub
<https://github.com/zim-desktop-wiki/zim-desktop-wiki/wiki/Performance-large-notebook>
for
benchmark on data load zim will accept

Search is indeed slow, and slows done for more files. This is due to it
being a full content search. Searching index keywords (e.g. tags, links,
pagenames) will be fast.

The issues you refer with the index view I do not recognize - are you using
the latest version ?

On Thu, Aug 12, 2021 at 2:50 PM Jon <email address hidden>
wrote:

> New question #698343 on Zim:
> https://answers.launchpad.net/zim/+question/698343
>
> Hi, I have been using Zim successfully (and gratefully!) for the last few
> months, and have a handful of wikis. They are only a few hundred pages in
> total, but growing.
>
> So my question is to the devs/ experts and/or anyone using Zim who has
> experienced large datasets. i.e. thousands of pages, large pages,
> attachments, deep trees, extensive use of links etc - i.e. calling all
> power users...
>
> E.g. I observe the duration of time to perform a search is getting longer
> - almost approaching 'shall I make a coffee' long. (Would be nice to have a
> progress bar #wishlist)
> E.g. 2 (some might consider a bug) The tree often takes a while to refresh
> after moving or deleting a page. Sometimes so slow I worry I might cause
> corruption by trying to perform other actions until the engine has caught
> up.
> E.g. 2.5 (really is a bug) Sometimes parts of the tree disappear after a
> big reorganisation (a restart of the app fixes this).
>
> I am interested in any tricks to improve performance (aka errors to avoid)
> in the structure of an 'ideal' wiki. And especially how to maintain
> stability to avoid corruption etc. Can the devs point to any obvious
> no-nos?
>
> e.g. what number of pages would be advisable to break out into separate
> wikis? 1k, 10k, 100k??
> e.g. 2 Does attachment size/type (total wiki size on disk) matter?
>
> My experience with my newish win10 PC is that the bottlenecks I experience
> generally (not just Zim) are never resolved by more processor (Intel i7
> hardly gets above 20% on an average day). Why doesn't processor utilisation
> ramp up e.g. when searching? (A typical layman question, but why did i pay
> for that 'power'??)
>
> My vanilla memory is nothing special in terms of speed I guess (and 8GB is
> half used due to the heavy payload of bloat in win10). But is at least
> upgradable: Offtopic slightly but does anyone think 'of course, essential',
> or 'don't waste your money'?
>
> For reference I imagine a fully loaded wiki might be 30*30*30*2k = 27k
> pages = 54MB of pure text.
> If it had media attachments there might be 10GB of images say.
>
> Thanks in advance!
>
> --
> You received this question notification because you are an answer
> contact for Zim.
>

Revision history for this message
Jon (jondap) said : 		#2

Jaap - Thanks for your fast reply! (And thanks for Zim, it really is great!)

I didn't see your performance page earlier sorry. It gives me confidence.

That test was on Linux - I use win10 with Bitlocker running, would that be significantly different?

The next time I get a glitch I will try to screencap it and send you an example. It's not happening often.