eCAP

adapted body size limit = 65535

Asked by Tom

What would cause an adapter to limit the size of the adapted body? Pages returned through my ecap adapter are being limited to 65535 characters.

My noteVbContentAvailable() is nicely being called with each chunk and I am building up my own internal buffer which grows larger than 65535 just fine.

During execution, I can see the following sequence of three function calls to my adapter for each piece of content:

abMake()
abContent(offset=0, size=4294967295)
abContentShift(size=65535)

The call to abContent with size=4294967295 seems to indicate that I should be able to have a buffer as large as I want, but the call to abContentShift with size=65535 confuses me and seems to indicate a problem. Why is abContentShift being called with size=65535?

I can see that the "configure" script sets the variable lt_cv_sys_max_cmd_len=65536. What is the significance of that variable and does that need to be changed? Seems unlikely, but it's the one magic number that I can find with that value.

I also greg'ed the squid source and found many references to 65535 and 65536 (e.g. SQUID_TCP_SO_RCVBUF, MEM_MAX_FREE, SMBLIB_MAX_XMIT). Is the limitation due to squid (not to eCap)? Even if squid has such a limitation, I would have expected abContent and abContentShift to work together to retrieve all the necessary content in multiple calls, just as noteVbContentAvailable provides each chunk of data.

The test case I found was simply using squid as a reverse proxy for www.google.com. Google includes more than 65KB of javascript in the page itself.

Question information

Language:
English Edit question
Status:
Solved
For:
eCAP Edit question
Assignee:
No assignee Edit question
Solved by:
Alex Rousskov
Solved:
Last query:
Last reply:
Revision history for this message
Alex Rousskov (rousskov) said : 		#1

> Why is abContentShift being called with size=65535?

Your host application is probably using a 64K buffer to send adapted body to the next HTTP hop.

> What is the significance of lt_cv_sys_max_cmd_len and does that need to be changed?

Please consult autotools documentation about that variable. I do not think libecap is using it directly.

> I would have expected abContent and abContentShift to work together to retrieve all the necessary content in multiple calls, just as noteVbContentAvailable provides each chunk of data.

Your expectations match eCAP design. I bet abContent() caller is just lazy and does not compute the exact area size limit, using libecap::nsize instead of the exact number. If you have one contiguous buffer, it should not matter. Nevertheless, consider filing a bug report with your host application provider because knowing a precise buffer space size may be useful for other adapters.

Revision history for this message
Tom (web-l) said : 		#2

Thanks. I posted a note to the squid-users list. I believe this link should show it:

http://squid-web-proxy-cache.1019090.n4.nabble.com/ecap-adapted-body-limited-to-64k-td4657413.html

Your are correct about the use of libecap::nsize, as I found that being used in squid's src/adaptation/ecap/XactionRep.cc in the function Adaptation::Ecap::XactionRep::moveAbContent(). It also does not appear to repeatedly or recursively call abContent() when the size of the adapted body is larger than squid's BodyPipe can handle.

Can you think of any workaround to get more than 64k to squid? You said that it should not matter if I have one contiguous buffer, which I do, but I'm still getting stopped at 64k. Is there some way that I can force squid to get all of my adapter body? Would it help to call noteAbContentAvailable() from within abContent() before it returns?

I feel like I can't be the first person to try returning pages over 64k, so I'm worried that something else might be wrong. I'm re-reading both the modifying adapter sample and the clamav adapter, and I've found a few other replies on here about the use of notifications (https://answers.launchpad.net/ecap/+question/63068 , https://answers.launchpad.net/ecap/+question/63147), but I'm at a loss for what else I can do as a workaround.

Revision history for this message
Best Alex Rousskov (rousskov) said : 		#3

> Is there some way that I can force squid to get all of my adapter body?

Under normal conditions, the host application will get all of the adapted body, but not necessarily in one call. You should not expect the host application to have an unlimited buffer or even a buffer as big as your adapter buffer. The host application may need time to drain its [smaller] buffer before it can get back to your adapter from more adapted content. If the host application does not come back ever, and there is no external error/problem, then there is probably a bug in the host application.

> Would it help to call noteAbContentAvailable() from within abContent() before it returns?

No. You should call noteAbContentAvailable() every time new adapted body content becomes available. There is not need to call that method more often, and I do not recommend doing it as it may trigger otherwise benign bugs in the host application.

Revision history for this message
Tom (web-l) said : 		#4

Thank you again!

One last question on this topic. In the adapter_modifying.cc sample, in noteVbContentAvailable and noteVbContentDone, both check for "if (sendingAb == opOn) before making a call to either hostx->noteAbContentAvailable() or hostx->noteAbContentDone(). What is the purpose of this test, or perhaps a better question is what is the consequence if this test did not exist?

I can see that sendingAb is set to opOn in abMake(), but I believe you've previously stated that the adapter should not depend upon the sequence of calls. The test for "if (sendingAb == opOn) seems to imply that abMake() should/must always be called prior to noteVbContentAvailable/Done and that it is the responsibility of the adapter to test for that.

I do think I have things working now, but I just want to make sure I understand the responsibilities of the adapter vs. the host.

Thanks again!

Revision history for this message
Tom (web-l) said : 		#5

Thanks Alex Rousskov, that solved my question.

Revision history for this message
Alex Rousskov (rousskov) said : 		#6

The adapter should not tell the host application that new adapted content is available if the host application did not ask the adapter for adapted content.

Whether the adapter delays adapted body production until abMake() is called is up to the adapter.