rohc

iprohc_server memory leak?

Asked by Atik

i got iprohc_server stopped two times with following message.

*** glibc detected *** iprohc_server: free(): invalid pointer: 0x083cc1d5 ***
======= Backtrace: =========
/lib/libc.so.6[0x5974a5]
/lib/libc.so.6(cfree+0x59)[0x5978e9]
iprohc_server[0x80e3662]
iprohc_server(asn1_delete_structure+0x54)[0x80e57c4]
iprohc_server(gnutls_global_deinit+0x47)[0x807ae07]
iprohc_server(main+0x1323)[0x8063445]
/lib/libc.so.6(__libc_start_main+0xdc)[0x543e9c]
iprohc_server[0x80619e1]
======= Memory map: ========
0050f000-0052a000 r-xp 00000000 08:03 3080226 /lib/ld-2.5.so
0052a000-0052b000 r--p 0001a000 08:03 3080226 /lib/ld-2.5.so
0052b000-0052c000 rw-p 0001b000 08:03 3080226 /lib/ld-2.5.so
0052e000-00682000 r-xp 00000000 08:03 3080232 /lib/libc-2.5.so
00682000-00683000 ---p 00154000 08:03 3080232 /lib/libc-2.5.so
00683000-00685000 r--p 00154000 08:03 3080232 /lib/libc-2.5.so
00685000-00686000 rw-p 00156000 08:03 3080232 /lib/libc-2.5.so
00686000-00689000 rw-p 00686000 00:00 0
00692000-006a7000 r-xp 00000000 08:03 3080493 /lib/libpthread-2.5.so
006a7000-006a8000 ---p 00015000 08:03 3080493 /lib/libpthread-2.5.so
006a8000-006a9000 r--p 00015000 08:03 3080493 /lib/libpthread-2.5.so
006a9000-006aa000 rw-p 00016000 08:03 3080493 /lib/libpthread-2.5.so
006aa000-006ac000 rw-p 006aa000 00:00 0
00820000-00821000 r-xp 00820000 00:00 0 [vdso]
009f6000-00a01000 r-xp 00000000 08:03 3081528 /lib/libgcc_s-4.1.2-20080825.so .1
00a01000-00a02000 rw-p 0000a000 08:03 3081528 /lib/libgcc_s-4.1.2-20080825.so .1
08048000-081cd000 r-xp 00000000 08:03 27152002 /usr/bin/iprohc_server
081cd000-081d0000 rw-p 00184000 08:03 27152002 /usr/bin/iprohc_server
081d0000-081d1000 rw-p 081d0000 00:00 0
083c4000-0842c000 rw-p 083c4000 00:00 0 [heap]
b6146000-b6147000 ---p b6146000 00:00 0
b6147000-b6b47000 rw-p b6147000 00:00 0
b6b47000-b6b48000 ---p b6b47000 00:00 0
b6b48000-b7548000 rw-p b6b48000 00:00 0
b7548000-b7549000 ---p b7548000 00:00 0
b7549000-b7f4a000 rw-p b7549000 00:00 0
b7f53000-b7f55000 rw-p b7f53000 00:00 0
bf826000-bf83b000 rw-p bffe9000 00:00 0 [stack]
Aborted

*** glibc detected *** iprohc_server: malloc(): smallbin double linked list corrupted: 0x08eaa238 ***
======= Backtrace: =========
/lib/libc.so.6[0x598b76]
/lib/libc.so.6(__libc_malloc+0x67)[0x599d97]
/lib/libc.so.6(__strdup+0x30)[0x59df80]
iprohc_server[0x80e3780]
iprohc_server[0x80e5dd5]
iprohc_server(asn1_der_coding+0x31)[0x80dcea1]
iprohc_server(_gnutls_x509_export_int_named+0x59)[0x8099139]
iprohc_server(_gnutls_x509_export_int+0x43)[0x8099383]
iprohc_server(gnutls_x509_crt_export+0x42)[0x809c052]
iprohc_server(_gnutls_x509_crt_to_gcert+0x189)[0x807a219]
iprohc_server(_gnutls_x509_raw_cert_to_gcert+0x50)[0x807a340]
iprohc_server(_gnutls_proc_cert_server_certificate+0x516)[0x807ee06]
iprohc_server(_gnutls_recv_client_certificate+0xc2)[0x8070c22]
iprohc_server(_gnutls_handshake_server+0x260)[0x806e370]
iprohc_server(gnutls_handshake+0x2a)[0x806e72a]
iprohc_server(new_client+0x11d)[0x8063731]
iprohc_server(main+0xd27)[0x8062e49]
/lib/libc.so.6(__libc_start_main+0xdc)[0x543e9c]
iprohc_server[0x80619e1]
======= Memory map: ========
0050f000-0052a000 r-xp 00000000 08:03 3080226 /lib/ld-2.5.so
0052a000-0052b000 r--p 0001a000 08:03 3080226 /lib/ld-2.5.so
0052b000-0052c000 rw-p 0001b000 08:03 3080226 /lib/ld-2.5.so
0052e000-00682000 r-xp 00000000 08:03 3080232 /lib/libc-2.5.so
00682000-00683000 ---p 00154000 08:03 3080232 /lib/libc-2.5.so
00683000-00685000 r--p 00154000 08:03 3080232 /lib/libc-2.5.so
00685000-00686000 rw-p 00156000 08:03 3080232 /lib/libc-2.5.so
00686000-00689000 rw-p 00686000 00:00 0
00692000-006a7000 r-xp 00000000 08:03 3080493 /lib/libpthread-2.5.so
006a7000-006a8000 ---p 00015000 08:03 3080493 /lib/libpthread-2.5.so
006a8000-006a9000 r--p 00015000 08:03 3080493 /lib/libpthread-2.5.so
006a9000-006aa000 rw-p 00016000 08:03 3080493 /lib/libpthread-2.5.so
006aa000-006ac000 rw-p 006aa000 00:00 0
009f6000-00a01000 r-xp 00000000 08:03 3081528 /lib/libgcc_s-4.1.2-20080825.so .1
00a01000-00a02000 rw-p 0000a000 08:03 3081528 /lib/libgcc_s-4.1.2-20080825.so .1
00a63000-00a64000 r-xp 00a63000 00:00 0 [vdso]
08047000-081cd000 r-xp 00000000 08:03 27152001 /usr/bin/iprohc_server
081cd000-081d0000 rw-p 00185000 08:03 27152001 /usr/bin/iprohc_server
081d0000-081d1000 rw-p 081d0000 00:00 0
08ea2000-08f0a000 rw-p 08ea2000 00:00 0 [heap]
b6000000-b6021000 rw-p b6000000 00:00 0
b6021000-b6100000 ---p b6021000 00:00 0
b6128000-b6129000 ---p b6128000 00:00 0
b6129000-b6b29000 rw-p b6129000 00:00 0
b6b29000-b6b2a000 ---p b6b29000 00:00 0
b6b2a000-b752a000 rw-p b6b2a000 00:00 0
b752a000-b752b000 ---p b752a000 00:00 0
b752b000-b7f2c000 rw-p b752b000 00:00 0
b7f35000-b7f37000 rw-p b7f35000 00:00 0
bfe63000-bfe78000 rw-p bffe9000 00:00 0 [stack]
Aborted

Question information

Language:
English Edit question
Status:
Answered
For:
rohc Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
matthew (xcalibre0) said : 		#1

Atik, could you please tell me how you progressed this far with iprohc? I understand how to build and run the program, but from that point I don't understand how to actually send the traffic. I have another question on the mailing list that highlights my question. It doesnt have to be didier that responds, can you tell me how you used the server and client to at least get as far as you have?

Revision history for this message
Atik (atik) said : 		#2

Hi,

This is what i did

on server i simply run ./iprohc_server ( make sure ipaddr: 172.31.4.1 or any private IP like 192.168.1.0/24 )

then on client PC run following

./iprohc_client --remote XXX.XXX.XXX.XXX ( server public IP ) --port 5000 --dev ROHC0 --debug --p12 /etc/server.p12 ( i used port: 5000 on my iprohc_server.conf )

once its connected check if its up by ifconfig command both server & client

First client will get IP of 172.31.4.11 ( or 192.168.1.11 if you configure this ip range )

you can ping server from client "ping 172.31.4.1" or ping client from server "ping 172.31.4.11"

Regards
Atik

Revision history for this message
matthew (xcalibre0) said : 		#3

Thank you Atik, that is what I have done. However, how do you know now that all traffic between the client and the server is now compressed? I see no indication in the logs that it is when I do that (i run both command with -debug), and when I use wireshark I do not see compressed traffic between the two machines.

Additionally when I run the "route" command I see multiple routes set up between the two boxes one that uses eth0 and the other that uses the new rohc0 interface.

What does your route command look like, and how do you know your traffic is in fact going through the tunnel at all?

Revision history for this message
matthew (xcalibre0) said : 		#4

ok i think i finally figured it out. The ip address that you put in the iprohc_server.conf file is NOT the local ip address of your eth0 card. In fact, in my case, i can not be so. Instead you set it to some made up ip address in an allowed NAT subnet like 10.0.0.X.

Then when you start up the iprohc_server the ip address assigned to your tun_ipip interface will be the ip address placed in that conf file.

Then on the client you put the ip address of the servers eth0 on the command line (not the ip you have in the iprohc_server.conf) and then the client program creates a rohc0 with an ip address on the same subnet as the ip in the iprohc_server.conf.

Revision history for this message
Didier Barvaux (didier-barvaux) said : 		#5

Matthew, Atik,

Matthew: you're right in your last comment. I suggest you to read some documentation about tunnels, it always works this way (configuration may change a little bit, but the general concept stays unchanged).

Atik: the problem occurs in the GnuTLS library, either because the IP/ROHC server mis-uses it, or because of a problem with the library itself. What version of the GnuTLS library do you use? If possible try to upgrade to the latest version available for your Linux distribution. If it still fails, could you please tell me more about the client certificate (how you created it, what is the SN...)? My hypothesis is that a field of the certificate is the root of the problem.

Regards,
Didier

Revision history for this message
Atik (atik) said : 		#6

I guess i messed up, i used same certificate file for both client & server.

it will be great if you include self-sign cert generation in documentation.

Regards
Atik

Revision history for this message
Didier Barvaux (didier-barvaux) said : 		#7

Atik,

> I guess i messed up, i used same certificate file for both client & server.

OK. That was wrong indeed.

> it will be great if you include self-sign cert generation in documentation.

I improved the installation [1] and start-up [2] procedures. Tell me if it clarifies things.

Regards,
Didier

[1] http://rohc-lib.org/wiki/doku.php?id=iprohc-install
[2] http://rohc-lib.org/wiki/doku.php?id=iprohc-run

Revision history for this message
Atik (atik) said : 		#8

its great to see everything in details.. however i am maybe missing something again.. getting new problem

May 8 17:40:55 Linux iprohc_client[3619]: Certificate can't be verified :
May 8 17:40:55 Linux iprohc_client[3619]: - Certificate issue is not a CA

Regards
Atik

Revision history for this message
Didier Barvaux (didier-barvaux) said : 		#9

Atik,

> its great to see everything in details.. however i am maybe missing something again.. getting new problem
>
> May 8 17:40:55 Linux iprohc_client[3619]: Certificate can't be verified :
> May 8 17:40:55 Linux iprohc_client[3619]: - Certificate issue is not a CA

There is a spelling mistake. You should read "Certificate issuer is not a CA". It means that you didn't sign the client certificate with a Certificate Authority (CA).

In the howto (http://rohc-lib.org/wiki/doku.php?id=iprohc-run), you may see that the CA contains:
    X509v3 Basic Constraints:
      CA:TRUE
while the other certificate contains:
    X509v3 Basic Constraints:
      CA:FALSE

Is it the case for you?

Regards,
Didier

Revision history for this message
Atik (atik) said : 		#10

i think so, is it possible if you share openssl.cnf to howto wiki page?

i made CA with

X509v3 Basic Constraints:
      CA:TRUE

but still having trouble, iprosh_client now hanging after sending following error, then stop.
Certificate can't be verified :

Regards
Atik

Revision history for this message
Didier Barvaux (didier-barvaux) said : 		#11

Atik,

> i think so, is it possible if you share openssl.cnf to howto wiki page?

I didn't used any specific openssl.cnf in my example. It defaulted to the system /etc/ssl/openssl.cnf I suppose. I didn't changed it. It is the one provided by Gentoo Linux.

> i made CA with
>
> X509v3 Basic Constraints:
> CA:TRUE
>
> but still having trouble, iprosh_client now hanging after sending following error, then stop.
> Certificate can't be verified :

No additional message after this one? What is the exit status of the iprohc_client process?

Regards,
Didier

Revision history for this message
Atik (atik) said : 		#12

On Client :
May 9 10:59:56 Linux iprohc_client[2887]: TLS handshake succeeded
May 9 10:59:56 Linux iprohc_client[2887]: Certificate can't be verified :

Then nothing, its doesnt exit or quit,

On Server :
May 9 18:50:04 Linux iprohc_server[17392]: TLS handshake succeeded

I will try on gentoo, will keep posted.

Thanks
Atik

Revision history for this message
Didier Barvaux (didier-barvaux) said : 		#13

> On Client :
> May 9 10:59:56 Linux iprohc_client[2887]: TLS handshake succeeded
>May 9 10:59:56 Linux iprohc_client[2887]: Certificate can't be verified :
>
> Then nothing, its doesnt exit or quit,

According to the source code, it may quit without message. The exit code of the process could help us determine if it quits because of the certificate problem or if an additional crash occurs.

Regards,
Didier

Revision history for this message
Atik (atik) said : 		#14

but "netstat -n" and "ps as" shows client process is running and connected to server,

after kill -USR1 $( cat /var/run/iprohc_server.pid ), gives the following. and server died with "Segmentation fault"

May 9 19:43:13 cb iprohc_server[17392]: Packing : 822941470
May 9 19:43:13 cb iprohc_server[17392]: Stats :
May 9 19:43:13 cb iprohc_server[17392]: . Failed decompression : 1426261525
May 9 19:43:13 cb iprohc_server[17392]: . Total decompression : 236126980
May 9 19:43:13 cb iprohc_server[17392]: . Failed compression : 1314082372
May 9 19:43:13 cb iprohc_server[17392]: . Total compression : 1377846089
May 9 19:43:13 cb iprohc_server[17392]: . Failed depacketization : 151064055
May 9 19:43:13 cb iprohc_server[17392]: . Total received packets on raw : 1628444161
May 9 19:43:13 cb iprohc_server[17392]: . Total compressed header size : 542396239 bytes
May 9 19:43:13 cb iprohc_server[17392]: . Total compressed packet size : 151395632 bytes
May 9 19:43:13 cb iprohc_server[17392]: . Total header size before comp : 523321667 bytes
May 9 19:43:13 cb iprohc_server[17392]: . Total packet size before comp : -2042067414 bytes
May 9 19:43:13 cb iprohc_server[17392]: Stats packing :

my certificate is wrong somehow, i will try to make certificate with gentoo and try again. ( just in-case if it might cause problem, client is big hind NAT )

Regards
Atik

Revision history for this message
Didier Barvaux (didier-barvaux) said : 		#15

Atik,

> but "netstat -n" and "ps as" shows client process is running and connected to server,

Hmm, maybe the client is frozen in the GnuTLS cleanup. Could you kill the current process, then following the instructions below:
 1/ enable coredums: ulimit -c unlimited
 2/ start the client as you did before
 3/ when the client "freezes", cause it to abort: kill -ABRT <pid of the iprohc_client process>

> after kill -USR1 $( cat /var/run/iprohc_server.pid ), gives the following. and server died with "Segmentation fault"
> ...

Printing statistics at server while client was not fully connected seems not to work. I fixed the problem in the main branch of the IP/ROHC application [1]. Take revision 105 or later if you want to test. Thank you for reporting that problem!

Regards,
Didier

[1] https://code.launchpad.net/~didier-barvaux/rohc/iprohc

Revision history for this message
Atik (atik) said : 		#16

This is what i got on client syslog

segfault at 19 ip b75aedb4 sp bf9072f8 error 6 in libtasn1.so.3.1.9[b75a5000+f000]

and from client coredump

Program terminated with signal 6, Aborted.
#0 0xb772c424 in __kernel_vsyscall ()
(gdb) bt
#0 0xb772c424 in __kernel_vsyscall ()
#1 0xb75ef361 in ?? ()
(gdb) thread apply all bt full

Thread 2 (Thread 18760):
#0 0xb75f60c3 in ?? ()
No symbol table info available.

Thread 1 (Thread 18757):
#0 0xb772c424 in __kernel_vsyscall ()
No symbol table info available.
#1 0xb75ef361 in ?? ()
No symbol table info available.
(gdb) q

also server died once again with following error

*** glibc detected *** iprohc_server: munmap_chunk(): invalid pointer: 0x09df9820 ***
======= Backtrace: =========
/lib/libc.so.6(cfree+0x188)[0x597a18]
iprohc_server[0x80e364c]
iprohc_server(asn1_delete_structure+0x54)[0x80e57c4]
iprohc_server(gnutls_global_deinit+0x47)[0x807ae07]
iprohc_server(main+0x1323)[0x8063445]
/lib/libc.so.6(__libc_start_main+0xdc)[0x543e9c]
iprohc_server[0x80619e1]

i should better start again with gentoo created cert. by the way latest iprohc gives me follwoing error

-- checking for module 'rohc'
-- found rohc, version trunk
-- Found ROHC: trunk
-- Looking for rohc_compress2 in rohc_comp
-- Looking for rohc_compress2 in rohc_comp - not found
CMake Error at common/FindROHC.cmake:41 (message):
  rohc_compress2() not available in the ROHC library, please upgrade to 1.6.0
  or greater
Call Stack (most recent call first):
  common/CMakeLists.txt:6 (find_package)

-- Configuring incomplete, errors occurred!

but i have latest rohc trunk installed

Regards
Atik

Revision history for this message
Didier Barvaux (didier-barvaux) said : 		#17

Atik,

> i should better start again with gentoo created cert. by the way latest iprohc gives me follwoing error
>
> -- checking for module 'rohc'
> -- found rohc, version trunk
> -- Found ROHC: trunk
> -- Looking for rohc_compress2 in rohc_comp
> -- Looking for rohc_compress2 in rohc_comp - not found
> CMake Error at common/FindROHC.cmake:41 (message):
> rohc_compress2() not available in the ROHC library, please upgrade to 1.6.0
> or greater
> Call Stack (most recent call first):
> common/CMakeLists.txt:6 (find_package)
>
> -- Configuring incomplete, errors occurred!
>
> but i have latest rohc trunk installed

What bazaar revision did you install?

What error message do you have in CMakeFiles/CMakeError.log ?

And what give you the following commands?
$ pkg-config --cflags rohc
$ pkg-config --libs rohc
$ grep rohc_compress2 /usr/include/rohc_comp.h

Regards,
Didier

Revision history for this message
Atik (atik) said : 		#18

Hi Didier,

# pkg-config --cflags rohc
( Blank )

 pkg-config --libs rohc
-lrohc_decomp -lrohc_comp -lrohc_common

grep rohc_compress2 /usr/include/rohc_comp.h
                        "use rohc_compress2() instead");
int ROHC_EXPORT rohc_compress2(struct rohc_comp *const comp,

this is how i build revision 718,

./autogen.sh --enable-static --disable-shared --disable-rohc-tests --disable-rohc-stats --disable-rohc-apps --disable-linux-kernel-module --disable-doc --prefix=/usr

Regards
Atik

Revision history for this message
Atik (atik) said : 		#19

if i remove "--enable-static --disable-shared" and make clean install rohc library, i can build iprohc.

:)

Revision history for this message
Didier Barvaux (didier-barvaux) said : 		#20

Atik,

> if i remove "--enable-static --disable-shared" and make clean install rohc library, i can build iprohc.

OK. The IP/ROHC application was never tested with a static ROHC library.

I updated the prerequisites section of the installation manual in consequence:
http://rohc-lib.org/wiki/doku.php?id=iprohc-install&#prerequisites

Regards,
Didier

Can you help with this problem?

Provide an answer of your own, or ask Atik for more information if necessary.

To post a message you must log in.