eCAP

Can I wait for noteVbContentDone before consuming vb?

Asked by Jacob

I am in the process of building an eCAP adapter against Squid 3.5.6. Squid is intercepting traffic and sending it through the adapter.

For most websites which I access through my adapter, everything seems to be fine. For example, I can go to ksl.com and everything finishes loading.

Odd behavior I am seeing in testing:
In some cases, websites freeze and never finish loading. By this I mean that the spinning 'loading' or 'connecting' indicator in FireFox does not disappear and the site is missing content. One example of this is Google: https://www.google.com/search?safe=strict&client=ubuntu&channel=fs&q=test&ie=utf-8&oe=utf-8

^^ With my adapter, nothing loads at all for that URL and FireFox remains 'spinning' forever.

Underlying behavior:
I added log messages to my eCAP adapter, and I am seeing the Xaction::noteVbContentAvailable method get called repeatedly. This is expected behavior, as data is coming down from the end server.

But Adapter::Xaction::noteVbContentDone is not being called at all. I can sit and watch the log for more than a minute with no additional 'noteVbContentAvailable '. The only way I have found to get Adapter::Xaction::noteVbContentDone to be called in these cases is for the end user to click 'stop' in FireFox. Then suddenly 'noteVbContentDone ' is called with 'atEnd = false'.

Comments:
This does not appear to be an issue with my adapter and HTTPS, as it will load other HTTPS sites with no issues.

My adapter's behavior:
The adapter currently does not pull virgin response data when 'noteVbContentAvailable' is called. I am waiting for 'noteVbContentDone' to be called before I dump the data to a process that needs all of the data present.

I was hoping to avoid the requirement of buffering the data in eCAP, to make using the virgin response body easier (just use 'lastHostCall()->useVirgin();' in the adapter); Why spend a bunch of time shuffling data back and forth when I need all of it anyway and could dump it all at once?

I am not certain whether the issue is in my adapter or Squid. What are your thoughts?

One last thing: What's the difference between 'noteVbContentDone' with 'atEnd' = false vs 'atEnd' = true? Does 'atEnd' = false mean there might be more data?

Thanks for your time.

Question information

Language:
English Edit question
Status:
Solved
For:
eCAP Edit question
Assignee:
No assignee Edit question
Solved by:
Alex Rousskov
Solved:
Last query:
Last reply:
Revision history for this message
Best Alex Rousskov (rousskov) said : 		#1

> The adapter currently does not pull virgin response data when 'noteVbContentAvailable' is called.

... which results in the host application suspending pulling virgin data from the origin server. Most host applications have finite buffers and will not load the entire [large] object.

> Why spend a bunch of time shuffling data back and forth when I need all of it anyway and could dump it all at once?

... because the host application is not required to buffer the response for you unless, of course, you can configure your host application to do so. With Squid, you might be able to do so by specifying a huge read_ahead_gap value but I am not sure Squid will handle such [ab]use well. Also, be careful not to run out of RAM -- imagine a malicious client fetching a few infinitely large objects concurrently.

> One last thing

One Question per question, please.

> What's the difference between 'noteVbContentDone' with 'atEnd' = false vs 'atEnd' = true?

A true atEnd means the body was terminated normally. A false atEnd means the message was aborted (e.g., the sender terminated the HTTP connection unexpectedly).

Revision history for this message
Alex Rousskov (rousskov) said : 		#2

BTW, the host application should not stop sending vb to the adapter after vbMake()+useVirgin() combination IMO. If your host application stops vb delivery to the adapter after useVirgin() and before the vbStopMaking() call, it is probably their bug.

Revision history for this message
Jacob (jacob-3) said : 		#3

This solved my issue. Thanks!

Revision history for this message
Jacob (jacob-3) said : 		#4

Thanks Alex Rousskov, that solved my question.