Saving the specific content type

Asked by Sandeep Kuttal on 2010-08-09

Thanks Alex for helping me for dealing with POST messages. I am looking- further down to just save the contents of the packets which have content-type as application/json (in other words specific content type only.) I can check the content-type in the header in start() function. But I want to just save the contents of json. Kindly suggest which function will be best to start with.

Thanks
Sandeep

Question information

Language:
English Edit question
Status:
Solved
For:
eCAP Edit question
Assignee:
No assignee Edit question
Solved by:
Sandeep Kuttal
Solved:
2010-08-19
Last query:
2010-08-19
Last reply:
2010-08-19
Alex Rousskov (rousskov) said : #1

You can save virgin message headers in start().
You can save virgin message body bytes in libecap::adapter::noteVbContentAvailable().

Sandeep Kuttal (skuttal) said : #2

Thanks Alex. Actually I was already doing this. Let me elaborate what I want to do..
For my research I need to see
1) the URI of the server interacting with client.
2) Need to just save application/json formats in files but leeting all other contents to pass by

Sandeep Kuttal (skuttal) said : #3

What I have done till now is

I am able to see the json format in start function. I was already capturing all the messages in the libecap::adapter::noteVbContentAvailable().

so i need to just save the json format contents (that too for specific uri) and let all other messages pass. Kindly suggest..

Sandeep Kuttal (skuttal) said : #4

What I have done till now is

I am able to see the json format in start function. I was already capturing all the messages in the libecap::adapter::noteVbContentAvailable().

so i need to just save the json format contents (that too for specific uri) and let all other messages pass. Kindly suggest..

Sandeep Kuttal (skuttal) said : #5

What I have done till now is

I am able to see the json format in start function. I was already capturing all the messages in the libecap::adapter::noteVbContentAvailable().

so i need to just save the json format contents (that too for specific uri) and let all other messages pass. Kindly suggest..

Alex Rousskov (rousskov) said : #6

In Adapter::Xaction::start(), if you decide not to adapt the message, do what the sample minimal adapter does: call useVirgin() and reset hostx.

As for accessing the Request URI, you can get it from the hostx->virgin() or hostx->cause() message, depending on whether you are doing REQMOD or RESPMOD. Once you have a message pointer, use firstLine() method to get RequestLine. Once you have RequestLine, call its uri() method.

Sandeep Kuttal (skuttal) said : #7

Hi Alex,

Thanks again for your constant support. I have made few changes to the code which are as below:

 libecap::shared_ptr<libecap::Message> adapted = hostx->virgin().clone();
        // Lines added just testing for getting URI //sandeep
        libecap::FirstLine *firstLine1 = &(hostx->virgin().firstLine());
        libecap::RequestLine *requestLine1 = NULL;
        requestLine1 = dynamic_cast<libecap::RequestLine*>(firstLine1);
        std::string uri_req1 = requestLine1->uri().toString();

                std::ofstream myfile;
                myfile.open ("/tmp/vb.txt",std::ios::app);
                myfile << uri_req1;
                myfile.close();

The problem now I am facing is that on starting the browser I am not able to see the contents for the sites. Sometimes on refreshing I see the proxy connection refusal. I was creating a file to store the contents of the body in libecap::adapter::noteVbContentAvailable(). But That file is not even created. I feel that the statment
  std::string uri_req1 = requestLine1->uri().toString();
   is bringing all the trouble.
Can you kindly give a look to this code and let me know what is going wrong?

Thanks
Sandeep

(I want to get the url of all the servers which are refereed by the website. Will this be possible with the above code. )

Sandeep Kuttal (skuttal) said : #8

after this code I think all other code doesn't work at all. Since I was creating files for checking JSON format and another file for saving the contents which I can no more find now.

Alex Rousskov (rousskov) said : #9

Perhaps requestLine1 is nil after dynamic_cast because virgin() is a response because you are in a RESPMOD? In RESPMOD, you should use cause() instead of virgin() to get to the request message.

Even if requestLine1 is not nil in your test case, you should not dynamic_cast to a pointer and then dereference the result without checking the resulting pointer for being nil.

Sandeep Kuttal (skuttal) said : #10

Yes, you are right. It was getting a NULL pointer. However, using cause() is resulting in an error and the code is not compiling. This is the changes I made:

 libecap::shared_ptr<libecap::Message> adapted = hostx->virgin().clone();
        // Lines added just testing for getting URI //sandeep
        //libecap::FirstLine *firstLine1 = &(hostx->virgin().firstLine());
        libecap::FirstLine *firstLine1 = &(hostx->cause().firstLine());
        libecap::RequestLine *requestLine1 = NULL;
        requestLine1 = dynamic_cast<libecap::RequestLine*>(firstLine1);
        std::string uri_req1;
        if (! requestLine1)
                 uri_req1 = requestLine1->uri().toString();
        else
                uri_req1 = "null pointer";

Sandeep Kuttal (skuttal) said : #11

I forgot to paste the error message. I have pasted it below, if it helps.

adapter_modifying.cc: In member function ‘virtual void Adapter::Xaction::start()’:
adapter_modifying.cc:200: error: invalid conversion from ‘const libecap::FirstLine*’ to ‘libecap::FirstLine*
make[2]: *** [adapter_modifying.lo] Error 1

Can you please tell what I am missing here? Thanks a lot again for all your help.

Alex Rousskov (rousskov) said : #12

You must use constant firstLine1 because cause() (i.e., the request that caused the current virgin response) cannot be modified.

Technically, the virgin() message should not be modified either. I am not sure why we did not make that method constant. Perhaps that will change in future releases.

Sandeep Kuttal (skuttal) said : #13

Hi Alex,

I changed the firstline to constant for cause. The code I changed is as follows

 const libecap::FirstLine *firstLine1 = &(hostx->cause().firstLine());
        //libecap::RequestLine *requestLine1 = NULL;
        //requestLine1 = dynamic_cast<libecap::RequestLine*>(firstLine1);
        const libecap::RequestLine *requestLine1 = dynamic_cast<const libecap::RequestLine*>(firstLine1);
        std::string uri_req1;
        if (! requestLine1)
                 uri_req1 = requestLine1->uri().toString();
        else
                uri_req1 = "null pointer";
  std::ofstream myfile;
                myfile.open ("/tmp/vb.txt",std::ios::app);
                myfile << uri_req1;
// myfile << contentTypeString1;
                myfile.close();

The code compiled successfully but on opening browser got ICAP protocol error.

Kindly suggest whats going wrong?

Thanks

Alex Rousskov (rousskov) said : #14

You need to enable verbose Squid logging (debug_options ALL,9) and try to understand what caused the error response. I cannot tell by looking at your code excerpt.

Sandeep Kuttal (skuttal) said : #15

Hi Alex,

I checked my log file and couldn't find the exact cause. Is it because of some locks or some other reason. I have uploaded my log file on following link:
http://www.cse.unl.edu/~skuttal/cache.log.gz

Start from line 90800 for this problem.
 client_side_request.cc(1502) handleAdaptationFailure: ICAP REQMOD callout failed, responding with error (on line 90859)

Can you kindly give a look since I am not able to figure out what's going wrong.

Thanks
Sandeep

(P.S The code is same as pasted above and I didn't made any changes in other parts of the file)

When I was using the hostx->virgin().firstLine() at that time some memory segmentation fault was occurring.

Alex Rousskov (rousskov) said : #16

2010/08/20 00:45:49.884| AsyncJob.cc(218) dial: AsyncJob::noteStart threw exception: theCauseRep != NULL
2010/08/20 00:45:49.884| Adaptation::Ecap::XactionRep will stop, reason: exception
2010/08/20 00:45:49.884| AsyncJob::noteStart() ends job [ ecapx5]
2010/08/20 00:45:49.884| Initiate.cc(51) swanSong: fatal failure; sending abort notification

You are probably calling the cause() method in REQMOD, resulting in an exception:

const libecap::Message &
Adaptation::Ecap::XactionRep::cause()
{
    Must(theCauseRep != NULL);
    return *theCauseRep;
}

See my earlier comments on how to get to the Request-URI in REQMOD and RESPMOD. The method differs depending on the adaptation mode.

Sandeep Kuttal (skuttal) said : #17

Hi Alex,
Can you tell how to access request message of REQMOD ?
Thanks

Alex Rousskov (rousskov) said : #18

host::Xaction::virgin() points to request message during REQMOD.

Sandeep Kuttal (skuttal) said : #19

Actually I was using the same function before also. But i was getting exception when i tried to dynamic cast the RequestLine

libecap::RequestLine *requestLine1 = dynamic_cast<libecap::RequestLine*>(firstLine1);

and requestLine is returning null pointer.

Sandeep Kuttal (skuttal) said : #20

sorry for wrong information it is not giving any exception. But I am not getting any contents on the browser

Sandeep Kuttal (skuttal) said : #21

Thanks Alex actually now I can see the contents.. Will look for url ad let you know

Sandeep Kuttal (skuttal) said : #22

Hi Alex,
Can you tell functions in ecap to access
REQMOD
request message
response message
RESPMOD
request message
response message

Thanks
Sandeep

Alex Rousskov (rousskov) said : #23

REQMOD request: host::Xaction::virgin()
REQMOD response: there is no response

RESPMOD request: host::Xaction::cause()
RESPMOD response: host::Xaction::virgin()