eCAP

How to supply metainformation to host after noteVbContentDone (a.k.a. eCAP trailers)

Asked by Christof Gerber

Hi

I am working on an eCAP adapter for Squid 3.5 which computes content hashes and writes them to access.log.
So far the eCAP adapter does what it suppose to the only thing missing is to communicate the hash back to the host for writing to the access.log. The problem is that the host is calling visitEachOption(), which I use to handover the hash to the host with an eCAP header, directly after hostx->useAdapted() or hostx->useVirgin() is called. By this time the hash is not ready yet, because the file has not passed the adapter entirely.

At the moment I see the following solution for my problem:

Can I somehow force squid to call visitEachOption() at an other point? After the message has passed the adapter.

Is there another way to handover a string to squid for writing to access.log (linked to the http message)?

Is it possible for an adapter to get the entire message before calling hostx->useAdapted()?

Would you recommend to change the host-adapter (squid src code) in order to call visitEachOption at the right point?

Thank you in advance for your help.

Question information

Language:
English Edit question
Status:
Solved
For:
eCAP Edit question
Assignee:
No assignee Edit question
Solved by:
Alex Rousskov
Solved:
Last query:
Last reply:
Revision history for this message
Best Alex Rousskov (rousskov) said : 		#1

What you want is similar to ICAP trailers -- metainformation available at the end of the adaptation transaction rather than at the beginning. Squid eCAP implementation does not support that yet. Squid ICAP implementation does (since v5 r14941).

To support eCAP trailers, the host application (e.g., Squid) must call adapter::Xaction::visitEachOption() at the end of the transaction. Implementing this requires careful planning because (a) some transactions may not expect such late calls and crash the host application and (b) such calls may be rather expensive from performance point of view (and usually unnecessary). In eCAP v1.0, some custom signaling during the _first_ visitEachOption() call (at the beginning of the transaction) could trigger a second call. In the future eCAP API version, we can add a dedicated host::Xaction::noteOptionsChange() or similar call to trigger additional call(s) at the right time.

IMO, the Squid project should welcome quality patches implementing both short-term and long-term signaling discussed above.

Needless to say, you can always hack your host application and your adapter to do something custom, outside the eCAP API. This is technically trivial but may have serious code maintenance and licensing implications.

> Is it possible for an adapter to get the entire message before calling hostx->useAdapted()?

Sure! You adapter controls the timing of useAdapted()/useVirgin() calls. The adapter may get the whole virgin message before telling the host transaction how to proceed. However, delaying those calls requires buffering the message body and delaying forwarding of the message, which is often expensive and leads to bad UX.

BTW, if an adapter does nothing but checksum calculation, it should you call useVirgin() instead of useAdapted().

Revision history for this message
Christof Gerber (sitschi) said : 		#2

Hi Alex

First of all. Thanks for your qualified answer.

When I first started the project and I was new to eCAP my first approach was exactly what you suggested to call useVirgin() because I have no intention to adapt anything. But since the host seems to call Xaction::stop as soon as I call useVirgin() I thaught this is the sign that the API does not allow me to eavesdrop the virgin body after calling useVirgin().

Specifically I tried those workflows to reach the goal of visitEachOption after I have the hash computed:

Alpha:

2017/03/08 16:46:44.820 kid1| eCAP fcap: Xaction::Xaction| filecount: 2| received: 0
2017/03/08 16:46:44.820 kid1| eCAP fcap: Xaction::start| filecount: 2| received: 0
2017/03/08 16:46:44.820 kid1| eCAP fcap: hostx->vbMake| filecount: 2| received: 0
2017/03/08 16:46:44.820 kid1| eCAP fcap: Xaction:noteVbContentAvailable| filecount: 2| received: 0
2017/03/08 16:46:44.828 kid1| eCAP fcap: hostx->vbContentShift(size)| filecount: 2| received: 267
2017/03/08 16:46:44.828 kid1| eCAP fcap: Xaction::noteVbContentDone| filecount: 2| received: 267
2017/03/08 16:46:44.828 kid1| eCAP fcap: hostx->useVirgin()| filecount: 2| received: 267
2017/03/08 16:46:44.829 kid1| eCAP fcap: Xaction::stop| filecount: 2| received: 267
2017/03/08 16:46:44.829 kid1| eCAP fcap: Xaction::~Xaction| filecount: 2| received: 267

leads to an ICAP protocol error message in the browser and besides this the visitEachOption() is not even called after useVirgin() -> reason?

Beta:

2017/03/08 16:53:50.524 kid1| eCAP fcap: Xaction::Xaction| filecount: 3| received: 0
2017/03/08 16:53:50.524 kid1| eCAP fcap: Xaction::start| filecount: 3| received: 0
2017/03/08 16:53:50.524 kid1| eCAP fcap: hostx->vbMake| filecount: 3| received: 0
2017/03/08 16:53:50.524 kid1| eCAP fcap: Xaction:noteVbContentAvailable| filecount: 3| received: 0
2017/03/08 16:53:50.528 kid1| eCAP fcap: hostx->useVirgin()| filecount: 3| received: 0
2017/03/08 16:53:50.528 kid1| eCAP fcap: Xaction::visitEachOption| filecount: 3| received: 0
2017/03/08 16:53:50.528 kid1| eCAP fcap: Xaction::stop| filecount: 3| received: 0
2017/03/08 16:53:50.528 kid1| eCAP fcap: Xaction::~Xaction| filecount: 3| received: 0

In this workflow useVirgin() triggers the visitEachOption and the stop. This seems not promessing at all because I do not have the hash when visitEachOption is called and useVirgin() seems to stop the Xaction. -> not useful

What is your idea of a workflow that would let the message pass the adapter (compute sha256) without modification before visitEachOption() / useVirgin() is called?

Revision history for this message
Alex Rousskov (rousskov) said : 		#3

For the record, there is another question about trailers-like API changes: https://answers.launchpad.net/ecap/+question/253254

Revision history for this message
Alex Rousskov (rousskov) said : 		#4

I know it is difficult, but please do your best to avoid asking different questions inside a single Question thread!

Another question (https://answers.launchpad.net/ecap/+question/292595) appears to contain the following answer to some of your useVirgin()-related questions: The host application should not stop sending vb to the adapter after vbMake()+useVirgin() combination IMO. If your host application stops vb delivery to the adapter after useVirgin() and before the vbStopMaking() call, it is probably their bug.

AFAICT, your "Beta" debugging indicates that there is such a bug in your host application (i.e., Squid). I encourage your to report it and get it fixed. Unless specifically noted otherwise, my answers apply to bug-free eCAP implementations only (naturally!).

> the visitEachOption() is not even called after useVirgin() -> reason?

A host application misconfiguration or host application bug. If the host application needs transaction metadata, then visitEachOption() must be called for all eCAP transactions, including those that call useVirgin(). Please note that the host application does not have to call visitEachOption() if it has no use for the answer.

> What is your idea of a workflow that would let the message pass the adapter (compute sha256) without modification

An immediate pair of vbMake(), useVirgin() calls followed by an asynchronous sequence of appropriate vbContentShift() calls. Please note that this approach will not allow you to supply the computed checksum to the host application unless the host application calls visitEachOption() at the end of the transaction. However, that "missing" second visitEachOption() call is a separate question/problem (that we have already discussed above).

> before visitEachOption() is called?

I consider this an invalid requirement - the adapter transaction should be prepared to handle zero or more visitEachOption() calls, regardless of their exact timing. (Again, whether your host application actually calls visitEachOption() when you need it is a different matter that we have already discussed above).

> before useVirgin() is called?

Your adapter can certainly delay the useVirgin() call but delaying that call leads to bad UX due to message forwarding delays.

Revision history for this message
Christof Gerber (sitschi) said : 		#5

Thank you for your further explanation. This is very helpful in the process of coping with the advantages and limitations of eCAP.

Revision history for this message
Christof Gerber (sitschi) said : 		#6

Thanks Alex Rousskov, that solved my question.