file limits on archive creation?

Asked by Baradé

hi,
I use "find . -type f | format | xargs smpq -cA -M 1 War3Patch.mpq" to create my custom MPQ archive but when it grows up to ~ 450 MiByte I get "War3Patch.mpq': File exists
xargs: smpq: exited with status 255; aborting". Afterwards, I can add all remaining files manually via Konqueror. Do you have any idea if there are some limits when adding files at the beginning on archive creation?

Question information

Language:
English Edit question
Status:
Answered
For:
SMPQ Edit question
Assignee:
Pali Edit question
Last query:
Last reply:
Revision history for this message
Pali (pali) said :
#1

Of course, MPQ archives are using hash tables for storing file names. And hash table has limited size, so has limited file count. When you want to add more files then maximum file count, you need to rebuild and increase size of hash table.

smpq has option -m (--max-file-count) which can be used to specifing maximum file count. It is good to specifing maximum file count when creating archive. if -m <count> is not specified (or is 0) smpq try to rebuild bash tables and increase size automatically. This operation can spend a lot of time (if you have a lot of files)... and maybe is not working correctly always.

if you want to see informations about archive (maximum file count too), use option -i (--info).

See manpage or smpq --help for all smpq options.

And option -c is for creating archive, option -a is for appending file(s) to existing archive. If you know how many file do you want to add into archive I suggest you to first create archive (with correct maximum file count) and then run shell program/scripts (find) to addding files into archive.

Your problem is maybe also with xargs. It can run program smpq more times if find outputs a lot of files (more then number of arguments allowed by sheel for external program). And next time when smpq is started it try to create that archive again (option -c) and it fail at: "War3Patch.mpq': File exists
Solution is again with creating archive first and and then appending files.

Revision history for this message
Baradé (barade-barade) said :
#2

Wouldn't it be a good idea to detect hash table size automatically by the number of passed arguments/file paths?

Revision history for this message
Baradé (barade-barade) said :
#3

Or is this already done by passing -m autodetect which I am trying at the moment. Thanks a lot for your fast help! I just love your program because it helps me to make everything automatically now!

Revision history for this message
Pali (pali) said :
#4

-m 0 is autodetect and it is default value if -m is not specified (it is calculated by number of files in archive, number of which which will be added and actual file limit).
but do not forget that rebuilding hash table is slow and is really better to specify it once

I think that your problem was with xargs (my first comment).

Revision history for this message
Baradé (barade-barade) said :
#5

Thanks a lot. Just one final thing. It seems that I always have to pass at least one file path when creating an archive. It would be useful to be able to create empty archives.
Maybe you could add the ability to specify directories or file lists of paths so I do not have to use xargs anymore.

Revision history for this message
Pali (pali) said :
#6

Now it is possible to create empty mpq archive without specifing files. Support added in this commit: http://bazaar.launchpad.net/~pali/smpq/trunk/revision/97

Revision history for this message
Pali (pali) said :
#7

Now it is in new release 1.5: https://launchpad.net/smpq/trunk/1.5

Can you help with this problem?

Provide an answer of your own, or ask Baradé for more information if necessary.

To post a message you must log in.