Current CID in use

Asked by Mohammad Abyan Abdullah on 2015-11-30

Hi!!
  Is there any way to know how many CID's being used by the ROHC by the time ?

Regards
Abyan

Question information

Language:
English Edit question
Status:
Answered
For:
rohc Edit question
Assignee:
No assignee Edit question
Last query:
2016-08-07
Last reply:
2016-08-07

Hello,

You may get the information from the ROHC compressor with the rohc_comp_get_general_info() function and then reading the contexts_nr field of the rohc_comp_general_info_t structure.

* documentation for rohc_comp_get_general_info():
https://rohc-lib.org/support/documentation/API/rohc-doc-1.7.0/group__rohc__comp.html#ga558fa49910707ba69845d881ba054fcf

* documentation for rohc_comp_general_info_t:
https://rohc-lib.org/support/documentation/API/rohc-doc-1.7.0/structrohc__comp__general__info__t.html

* example in the non-regression test:
https://github.com/didier-barvaux/rohc/blob/master/test/non_regression/test_non_regression.c#L535

A similar function exists for the decompressor rohc_decomp_get_general_info():
https://rohc-lib.org/support/documentation/API/rohc-doc-1.7.0/group__rohc__decomp.html#ga37cf0aa64dbe8c89223de7f15a424a90

Regards,
Didier

Hi!
  Thanks for the reply. It almost solved my problem. I was searching at last_packet_info thats why can't see that. Anyways now I can see the context used at the time.
  But what I see is the value is increasing only even the streams are reducing but the value is still rising. Like at 10 streams it says current context used = 31 next at 15 streams its says 42 and later after few minutes at 12 streams its says there is 103 contexts are being used. Why its always increasing ?

Regards
Abyan

Hello,

The contexts_nr field represents the number of contexts that are used for a stream. The ROHC compressor doesn't know exactly when a stream ends. So, it avoids as much as possible to re-use previous contexts for new streams. So, contexts_nr will keep increasing until the compressor reaches the MAX_CID parameter (at that time, the compressor will re-use the context that was used the least recently because it is most probably a stream that ended sometimes ago).

Knowing the real number of _active_ streams is impossible from the ROHC compressor. The ROHC compressor doesn't have the information itself.

Maybe I could add a new API function for applications to tell the ROHC compressor when a stream ended. This would make contexts_nr more accurate (if the application helps with the new API) and maybe it would also helps the ROHC compressor for managing contexts. The ROHC compressor could re-use contexts with smaller CIDs much earlier, leading to better compression ratio.

Does your application knows when the stream ended?

Regards,
Didier

Hi!!
   Sorry was sick that's why couldn't answer earlier. Sorry my tunnel isn't aware of the stream ended. I don't think iprohc is able to detect the stream end. At each tunnel end I'm using SIP softswitch( And SIPp) to test the streams. I think only way to detect the stream end by SIP Bye message right ? But here streams have to be attached by same CID. I don't know if ROHC does that. It would be very much help full to have a such API which will detect the the unused CID and clean them for new use. And also more CID can be added on the fly and it send the state to the remote decompressor to enlarge the CID at remote side too. So that way we can avoid the conflict the CID mismatch with remote tunnel.

Regards
Abyan

Hello,

> Sorry my tunnel isn't aware of the stream ended. I don't think iprohc is able to
> detect the stream end.

You're right, the IP/ROHC application is not able to detect stream beginning/end.

> At each tunnel end I'm using SIP softswitch( And SIPp) to test the streams. I think
> only way to detect the stream end by SIP Bye message right ?

Yes, SIP INVITE and BYE messages may be used to detect stream beginning/end.

> But here streams have to be attached by same CID. I don't know if ROHC does that.

Sorry, I don't understand what do you mean. Do you say that the SIP and RTP streams
have to share the same CID?

> It would be very much help full to have a such API which will detect the the unused
> CID and clean them for new use.

I think that the detection should not be part of the ROHC library itself. However, the
library can provide a new function rohc_comp_free_context() or the like in its API and
the IP/ROHC application (or any other app using the ROHC library) can use it to indicate
that a stream ended. The application would be free to detect the end of a stream using
whatever mechanism. The IP/ROHC app might try to perform some DPI to detect SIP
INVITE and BYE messages.

> And also more CID can be added on the fly and it send the state to the remote
> decompressor to enlarge the CID at remote side too. So that way we can avoid the
> conflict the CID mismatch with remote tunnel.

It looks like a different problem than the "detect end of stream" problem, doesn't it? What
use case would you like to solve by adding new CIDs on the fly? Current versions of the
ROHC library let you define the maximum number of CIDs at initialization. Is it a limitation
for you?

Regards,
Didier

Hi!!
  Thanks for the quick answer.

>Sorry, I don't understand what do you mean. Do you say that the SIP and RTP streams
>have to share the same CID?

Yes.
 What I meant is if we can't detect a stream beginning and which CID attached to it we can't clean the CID. And I don't know ROHC does attach a CID with a SIP INVITE message.

>It looks like a different problem than the "detect end of stream" problem, doesn't it? What
>use case would you like to solve by adding new CIDs on the fly? Current versions of the
>ROHC library let you define the maximum number of CIDs at initialization. Is it a limitation
>for you?

Yes. Kind of that. At previous answer you already told that if there is less CID present and being used the compression would be good. And if more CID's being used in the application compression might be less right ? And also if at library initialization we create large CID value it'll use more memory overhead then it might need. So if application can create CID on the fly then it might reduce the memory overhead. Cause its only using the CID according to it need. Like and example.
 In a tunnel if we attach 300 CID for 100 concurrent streams. It doesn't means that 24 Hours we are having 100 streams at that tunnel. It might get reduce time to time even nill. So at that time application might use low memory overhead. And at current scenario you told me that ROHC already occupy all the CID gradually even its not that much streams on use.

Regards
Abyan

Hello,

> > Sorry, I don't understand what do you mean. Do you say that the SIP
> > and RTP streams have to share the same CID?
>
> Yes.
> What I meant is if we can't detect a stream beginning and which CID
> attached to it we can't clean the CID. And I don't know ROHC does
> attach a CID with a SIP INVITE message.

Yes. The DPI algorithm would look like:

For every incoming packet, inspect the packet to detect either a SIP
INVITE packet, a RTP packet, a SIP BYE packet or another type of packet:
 1/ if packet is a SIP INVITE packet:
         a/ extract RTP streams characteristics from the SDP payload,
         b/ store those information in a table of active RTP streams along
              with a unique identifier of the multimedia session,
         c/ perform ROHC compression of the SIP INVITE packet.
 2/ if packet is a RTP packet, search for a corresponding stream in the
      table of active RTP streams
         a/ if not found, stop DPI processing and perform ROHC compression
              of the RTP packet
         b/ if found and no RTP packet was seen for that RTP stream
              yet, record the CID used for ROHC compression in the table of
              active RTP streams, and perform ROHC compression of the RTP
              packet
         c/ if found and at least one RTP packet was seen for that RTP
              stream, perform ROHC compression
 3/ if packet is a SIP BYE packet:
         a/ remove any RTP streams related to the multimedia session from
              the table of active RTP streams (thanks to the unique identifier
              of the multimedia session),
         b/ tell the ROHC library that CIDs related to those streams may be
              discarded (using the new API),
         c/ perform ROHC compression of the SIP BYE packet
 4/ if packet is not a SIP INVITE/BYE or RTP packet, perform ROHC
      compression of the packet

The "difficult part" is the SIP INVITE/BYE parsing. Using an existing
DPI library such as nDPI might help:
http://www.ntop.org/products/deep-packet-inspection/ndpi/

> > It looks like a different problem than the "detect end of stream"
> > problem, doesn't it? What use case would you like to solve by adding
> > new CIDs on the fly? Current versions of the ROHC library let you
> > define the maximum number of CIDs at initialization. Is it a
> > limitation for you?
>
> Yes. Kind of that. At previous answer you already told that if there
> is less CID present and being used the compression would be good. And
> if more CID's being used in the application compression might be less
> right ?

Yes, using small numbers for CIDs is better.

When using small CIDs (16 contexts max):
 * CID 0 takes 0 byte (not transmitted at all),
 * CIDs 1-15 take 1 byte.

When using large CIDs (16384 contexts max):
 * CIDs 0-127 takes 1 byte,
 * CIDs 127-16383 take 2 bytes.

So, better use CID 0 as much as possible in case of small CIDs. And,
better use CIDs between 0 and 127 in case of large CIDs.

> And also if at library initialization we create large CID
> value it'll use more memory overhead then it might need. So if
> application can create CID on the fly then it might reduce the memory
> overhead.

The overhead doesn't seem that big. But, the definition of "big" highly
depends on what platform you target ;-)

A compression context weights for 160 bytes. A decompression context
for 376 bytes. The real numbers are probably larger because there are
some additional dynamic memory allocations involved, but let's use that
raw numbers as raw estimations.

So, for some MAX_CID values:
  MAX_CID = 15: (160+376)*15 = 8 KB
  MAX_CID = 300: (160+376)*300 = 157 KB
  MAX_CID = 1000: (160+376)*1000 = 523 KB
  MAX_CID = 5000: (160+376)*5000 = 2.6 MB
  MAX_CID = 16384: (160+376)*16384 = 8.4 MB

Does it seems "big" for your use case / platform ?

> Cause its only using the CID according to it need. Like and
> example. In a tunnel if we attach 300 CID for 100 concurrent streams.
> It doesn't means that 24 Hours we are having 100 streams at that
> tunnel. It might get reduce time to time even nill. So at that time
> application might use low memory overhead. And at current scenario
> you told me that ROHC already occupy all the CID gradually even its
> not that much streams on use.

I see your point, but I'm afraid that it will have some inconvenient.

The ROHC library stores contexts in an array. That array is allocated
at library init (and freed at exit) to avoid too much dynamic memory
allocation during packet processing. If the array size is dynamically
increased/decreased, then realloc() would happen during packet
processing and increase processing time (and network latency).

Another problem I see: if at a time, there are few contexts in use,
the array might not decreased in some scenarios. For example:
 * start with 100 contexts,
 * at peak time, many RTP streams are compressed at a time, so the
    library increases the array of contexts up to 16384,
 * at night, the number of RTP streams goes back under 100, but at
    least one of the long-lived remaining RTP streams uses the CID 16383,
 * the library cannot decrease the size of the context array (the
    index of array is the CID).

Regards,
Didier

Hi!

  Thanks for the memory related info. It helps me to understand that I can stop thinking about dynamic CID allocation and it seems also not feasible.
 And for the DPI I think it'll be helpful to have such function.
 But I think we will need a 3rd party library such as ndpi as you said.
 And at SIP BYE message sometimes if endpoint couldn't sent BYE message or a packet loss happens then that stream would be orphan and related CID will be still be occupied. At that point any idea to close the stream or CID ?
 Does /proc/sys/net/ipv4/netfilter/ip_conntrack_udp_timeout_stream has anything to do at that point ?

Regards
Abyan

Hello,

> And at SIP BYE message sometimes if endpoint couldn't sent
> BYE message or a packet loss happens then that stream would
> be orphan and related CID will be still be occupied. At that
> point any idea to close the stream or CID ?

You may either do nothing (context recycling will happen until all contexts are used, as now), or either implement some inactivity timeout. N seconds without RTP packets will make the application consider that the stream was interrupted without SIP BYE.

The timeout mechanism may work alone, without the DPI mechanism. It is simpler and it should reach almost the same behavior. DPI detects the end of RTP streams quicker, but with a small inactivity timeout there is almost no difference. Because of the simplicity, it is worth a try!

> Does /proc/sys/net/ipv4/netfilter/ip_conntrack_udp_timeout_stream has anything to do at that point ?

This file controls the inactivity timeout for UDP streams in the firewall/NAT rules handled by iptables/netfilter. There is no direct link with the ROHC library. You may however use netfilter connection tracking (conntrack) to detect inactivity among UDP streams. It would avoid to re-implement the stream identification/tracking in the IP/ROHC app.

The netfilter conntrack mechanism can warn the IP/ROHC app when a stream ends. The libnetfilter_conntrack library provides functions to do that:
* main page: https://www.netfilter.org/projects/libnetfilter_conntrack/index.html
* API documentation: http://www.netfilter.org/projects/libnetfilter_conntrack/doxygen/modules.html
* example printing creation/deletion of conntrack entries: https://git.netfilter.org/libnetfilter_conntrack/tree/examples/nfct-mnl-event.c

Regards,
Didier

Launchpad Janitor (janitor) said : #10

This question was expired because it remained in the 'Open' state without activity for the last 15 days.

Hi!
  Didier is it possible when we call c_create_context() if no usable context found can we recycle all the contexts which does have the
value more than like rtp time out (20 > now.sec - c->latest_used) ? In that way we might free few contexts which were not being used.

Regards
Abyan

Hello,

> Didier is it possible when we call c_create_context() if no usable context
> found can we recycle all the contexts which does have the value more than
> like rtp time out (20 > now.sec - c->latest_used) ? In that way we might free
> few contexts which were not being used.

I would like to better understand your situation, so I'll try to rephrase it and ask you some questions.

The current ROHC compressor does not use a timeout value for contexts. The ROHC compressor prefers using:
 a/ a free context (if any)
 b/ recycle the least used context (if no free context is found)

If I understand correctly, you would like in case b/ that the ROHC compressor frees more than one context (ie. all contexts that were not used since a defined timeout delay). Is my understanding correct?

If yes, what is the gain that you expect? It won't improve compression ratio. Maybe it will save some CPU cycles searching for the right context if you configured MAX_CID=16384 and many of your flows are short lived.

The purpose of those questions are for my understanding of your scenarios and goals. The ROHC compressor does not allow users to interact with the way it selects a compression context, adding some API to customize the packet classifier and the context recycling might be a great way to improve the library. Tell me more please.

Regards,
Didier

Hi!!
  Didier
   >>If I understand correctly, you would like in case b/ that the ROHC compressor frees more than one context (ie. all contexts that were not used since a defined timeout delay). Is my understanding correct?

Yes.
      I want to know exactly how many RTP or any streams are in progress. That's my only goal. So that way I can tell how many streams engaged in a tunnel. I couldn't find any other way.

Regards
Abyan

Hello,

Maybe I could add some API such as:

   /* free a context knowing its Context ID (CID) */
   bool rohc_comp_free_context(struct rohc_comp *const comp,
                                                                 const rohc_cid_t cid);

   /* free all contexts of one profile whose last packet is older or equal than now - timeout */
   bool rohc_comp_free_old_contexts(struct rohc_comp *const comp,
                                                                           const rohc_profile_t profile_id,
                                                                           const struct rohc_ts timeout);

The 2nd function should allow you to free all contexts corresponding to RTP streams in timeout (according to your own definition of timeout). What do you think of?

Regards,
Didier

Hi!
    Didier I think it'll help me to achieve my goal. That way I think we can search the next context faster too.

Regards
Abyan

Ok, will do it. I created a ticket: https://bugs.launchpad.net/rohc/+bug/1610702

Regards,
Didier

Can you help with this problem?

Provide an answer of your own, or ask Mohammad Abyan Abdullah for more information if necessary.

To post a message you must log in.