Excessive swapping during hash

Unable to reproduce. Gentoo x86, 2.6.27-gentoo-r10, 4G RAM.

Hashed 4G file without swapping.

--RZ

Could you try any from 2.6.28 or 2.6.29?
2.6.28-gentoo-r5 is stabilized

2.6.27 is not affected, only 2.6.28+. I've added a picture of memory usage with some milestones.
Before 2.6.28 linuxdcpp did not affect swapping at all. Well, after 2.6.28 swapping began, so I had to get a "workaeound" - a small 10M swap-file, which filled instantly, but did not fit enough pages to affect performance.

Ok, tested with 2.6.28-gentoo-r5 and I do see some swapping while hashing.

So is this a problem/feature of the kernel or should we doing something different?

--RZ

I don't know (if that question was for me). I hoped some of the devs could tell whether this is a linuxdcpp problem or not.

Question was mainly to Steven/individ... or anyone else who might have a clue.

Though I suspect that this isn't a Linuxdcpp problem as it seems that 2.6.28 kernel is being more unresponsive and swapping more than the previous ones. (no I don't really have any hard data of this, just a gut feeling)

--RZ

Looking through the code, I've found out there are two implementations of hashing function. Swapping happens only when the fast one is in use.

I've added some additional error checking to fast hash a while ago in upstream dc++, but it has not made its way to linuxdcpp yet. It may show that madvise is indicating an error and fallback to regular hash in your case. Can you try grabbing the latest copy of HashManager.cpp and .h from DC++ and retrying? Also, what is your max hash speed size preference set to?

http://bazaar.launchpad.net/~dcplusplus-team/dcplusplus/trunk/files/head%3A/dcpp/

Rebuilt with hashmanager from dcpp fith debug=true (not sure whether this matters here). Started from terminal and turned on system log (also not sure). There are no error messages from mmap/madvise either in the terminal nor in system.log

Did I do everything right?

Sounds like it. They probably just didn't have any errors to report. Did you see any difference in the swapping?

Nope. Swaps like hell
BTW, I've found some discussion on linux-mm, which could be related to this case: http://marc.info/?l=linux-kernel&m=121616312201185&w=2
Also mysql guys had similar problems with mmaps: http://bugs.mysql.com/bug.php?id=37408

Having exactly the same problem. 3gigs of RAM on a karmic 64bit installation...
2GB free and when linuxdcpp hashes large files, it is using more than 1GB of swap that makes my computer very slow at the time of this operation...

Might be related to bug #517948.

There's a patchset in kernel dev. list, which looks to be related to this trouble:
http://lkml.org/lkml/2010/2/22/274

This is the same in more details:
http://marc.info/?l=linux-mm&m=125297194724716&w=2

Indeed, that patch looks very promising. If I understand it correctly, it could help with reducing swapping, reducing memory usage, and decreasing CPU utilization. Let's hope someone has the ability to test this patch to verify it solves this bug.

Oh, I also added an option to disable fast hashing until this issue is resolved. You can find it in the advanced tab of the preferences.

Well, 2.6.34-r1 is out and it contains patch under question. I've managed to try in on a testbox and compare with 2.6.32-gentoo-r3 on a x86_64 machine with 4G of ram

The file list is clean. After restart, adding a single 8G file causes ZERO swapping, and the system stays responsive. With 2.6.32 swapping occurs instantly when free memory reaches it's minimum and kswapd begins reclaiming pages. The system becomes sluggish, mouse cursor jumps.
Overall hashing speed with the new kernel is very stable - 57Mb/s on my old SATA drive. With 2.6.32, hashing speed is also very stable until kswapd steps in - after that, hashing speed drops to ~50Mb/sec and oscillates around it.
When hashing thread is finished, new kernel still shows ZERO swap usage, whereas 2.6.32 shows this:
$ free
             total used free shared buffers cached
Mem: 4055264 4022896 32368 0 2952 3824016
-/+ buffers/cache: 195928 3859336
Swap: 2008084 122796 1885288

So, this post is not a confirmation of a problem resolution, more testing is needed, and probably not until the patchset is backported on 2.6.33 series. Or, maybe, not until 2.6.34 is stable.