RTU Slave 2.0.3 modbus_listen issue

I am working in a device with master and slave capabilities. Basically I use the master to query some sensors (like 100), and then using shared memory I pass this info to the slave part, which waits to be polled by another device.
The problem is I am having big delays. The slave part is being polled every 10 seconds, because if I poll it more quickly, I won't answer in time. For example, if I poll it at 5 seconds, External Master polls, slave answer in 2 secs, external master polls, slave answer in 3 secs, external master polls, slaves answer in 4 secs, external master polls, external master polls, slave answer, external master polls, external master polls, external master polls.. slave answer... well you get the point.
With 10 seconds I don't have this problem however it's quite quite slow, I am just sending one query with a 50 bytes answer.
First I thought it was because I wasn't correctly synchronizing the shared memory part, however, when I start debugging the problem, I notice that the function "modbus_listen" will absorb most of the time. I mean, sometimes it answer in 10 seconds, sometimes in 15, sometimes in a minute.
I am using RTU.. I didn't find some function specifically for RTU, so the line is

"ret = modbus_listen(&mb_param,query,&query_size);"

I am still using 2.0.3 because I started the code with that library and all the other parts are well working, but if the issue is in actually using the 2.1.0 I will change. Or if you have any other idea of what ca be wrong? Or maybe it is not the correct use of the function.
I am using TS-7400 board, with a ARM EP9302 @200Mhz, with 32 RAM, Linux Debian 2.6.29 kernel.
Regards,
Enrique