eCAP

getting partial virgin buffer

Asked by nir

Hi,
Running Ubuntu 12.04 x86_64 + squid 3.4.2 + libecap 0.2.0 + modifying adapter.
I'm doing GET requests of >40KBytes.
When debugging noteVbContentAvailable & noteVbContentDone, I see that I get only 16383Bytes of the virgin buffer.

In the logs I see:
2014/01/17 22:46:25.730 kid1| http.cc(1065) persistentConnStatus: local=10.96.69.46:57339 remote=192.168.5.156:5554 FD 22 flags=1 eof=0
2014/01/17 22:46:25.730 kid1| http.cc(1085) persistentConnStatus: persistentConnStatus: content_length=48312
2014/01/17 22:46:25.730 kid1| http.cc(1089) persistentConnStatus: persistentConnStatus: clen=48312
2014/01/17 22:46:25.730 kid1| http.cc(1102) persistentConnStatus: persistentConnStatus: body_bytes_read=16456 content_length=48312
2014/01/17 22:46:25.730 kid1| http.cc(1436) processReplyBody: processReplyBody: INCOMPLETE_MSG from local=10.96.69.46:57339 remote=192.168.5.156:5554 FD 22 flags=1
2014/01/17 22:46:25.730 kid1| comm.cc(768) commSetConnTimeout: local=10.96.69.46:57339 remote=192.168.5.156:5554 FD 22 flags=1 timeout 900
2014/01/17 22:46:25.730 kid1| Server.cc(1008) replyBodySpace: ServerStateData may read up to min(49152, 16383) bytes
2014/01/17 22:46:25.731 kid1| http.cc(1500) maybeReadVirginBody: may read up to 16383 bytes from local=10.96.69.46:57339 remote=192.168.5.156:5554 FD 22 flags=1

I suspected linux TCP memory but changing it didn't help.
Is it related to SQUID_TCP_SO_SNDBUF? It is the only squid value I see configured to 16384.
What does INCOMPLETE_MSG mean?

Any help will be appreciated.

Thanks,
Nir

Question information

Language:
English Edit question
Status:
Solved
For:
eCAP Edit question
Assignee:
No assignee Edit question
Solved by:
nir
Solved:
Last query:
Last reply:
Revision history for this message
Alex Rousskov (rousskov) said : 		#1

eCAP adapters work inside a message processing _pipe_. When the message is large and/or it is delivered to the host application in smaller pieces, your adapter may never see the whole or "complete" message at once, unless the adapter accumulates what it sees. This is similar to ICAP services. Tuning variables such as TCP-related buffer sizes may change the amount of data the adapter sees, but does not guarantee that the adapter is going to see the whole message at once.

Accumulating message bodies makes coding a lot easier, but it consumes more resources and is bad for user-perceived response times because it breaks the message delivery pipeline. Pick your poison.

Revision history for this message
nir (nirb999) said : 		#2

Hi,
I found the problem in my code.
For now it is OK for me to accumulate the message body, I need it for processing.
Thank you.