Variety

How to use big local folders

Asked by pedrogaza

I would like to know if it is possible to raise the limit of 10,000 files on local folder. Because I put my photos folder in variety picture folder, but it failed to fetch it because there is to much files :

INFO:variety: list_files() 'More than 10000 files in the folders, stop listing'

Is there a way to do that (even an ugly way with config file edit)

Question information

Language:
English Edit question
Status:
Solved
For:
Variety Edit question
Assignee:
No assignee Edit question
Solved by:
pedrogaza
Solved:
Last query:
Last reply:
Revision history for this message
Peter Levi (peterlevi) said : 		#1

Don't worry, Variety works with big folders fine. This limit is just to the number of files it will traverse on one search for images, but it randomizes the order every time so next time it searches, it will select from another 10000 files.

On the other hand, reaching this limit means Variety has searched 10000 files and not yet found 100 images that match your criteria, so you either have instructed it to use a big folder that contains all types of files, thus making it search harder for images (i.e. more CPU and HDD usage, so not a good idea), or you are using relatively strict Color and size filters, again resulting in more CPU and HDD usage.

Revision history for this message
pedrogaza (pubpedro) said : 		#2

Yes my filter is based on the ratings of photos. On the 13000 photos, only 50 matchs this criteria.
But the problem is that it often choose photo which are not rated...
Have I done something wrong ?

Revision history for this message
pedrogaza (pubpedro) said : 		#3

Ok in fact, I think I have found what is going wrong. If I well understood Variety try to find 100 photos to build a buffer. If it does not find the 100 photos, it add photos which does not match criteria, am I true ?
In this case I just have to rate more photos. Now it seems ok.

Thanks.

Revision history for this message
Peter Levi (peterlevi) said : 		#4

Yes, you are generally correct. Variety does not keep an index or a DB of the images, it uses a simple approach of searching for matching images on demand. It is not exhaustive - it will pick up 100 random images out of all those 13000, then try to find matching images within those 100 to build its queue, when it does not find enough it starts to relax the criteria till it finds something. With 50 rated out of 13000 you are reaching this step very often.

Actually the previous description I gave was not very correct - it will only reach the 10000 limit and log the message you saw the first time, when it tries to evaluate how many images roughly are in the folders. (It needs to know, since things are very different when you have 20 and when you have 10000, but not that different when you have 10000 or 50000, hence the limit 10000).

Variety is not intended to be a full-fledged image management program, so if you rely heavily on this type of functionality, you might be better off rating the images in some external program, copying the rated images to a separate folder and then feeding this folder into Vairety - it'll consume less resources this way.