segmentation fault against Percona 5.5.13

Bug #813388 reported by Rene' Cannao'
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MySQL Data Dumper
Fix Released
Critical
Andrew Hutchings

Bug Description

Running mydumper against Percona 5.5.13 leads to segmentation fault.

Here some information from gdb:
$ gdb /root/mydumper/mydumper
GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-32.el5_6.2)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /root/mydumper/mydumper...done.
(gdb) run -B sbtest
Starting program: /root/mydumper/mydumper -B sbtest
warning: no loadable sections found in added symbol-file system-supplied DSO at 0x2aaaaaaab000
[Thread debugging using libthread_db enabled]
[New Thread 0x40a00940 (LWP 12660)]
[Thread 0x40a00940 (LWP 12660) exited]

Program received signal SIGSEGV, Segmentation fault.
0x0000003d5a07c0fc in strcasecmp () from /lib64/libc.so.6
(gdb) bt
#0 0x0000003d5a07c0fc in strcasecmp () from /lib64/libc.so.6
#1 0x00000000004067f2 in start_dump (conn=0x610f90) at /root/mydumper/mydumper.c:708
#2 0x0000000000407118 in main (argc=1, argv=0x7fffffffe508) at /root/mydumper/mydumper.c:556
(gdb) up
#1 0x00000000004067f2 in start_dump (conn=0x610f90) at /root/mydumper/mydumper.c:708
708 if (!strcasecmp(fields[i].name,"Command")) ccol=i;
(gdb) p *res
$1 = {row_count = 2, fields = 0x621a40, data = 0x611990, data_cursor = 0x61fa50, lengths = 0x611928, handle = 0x0, methods = 0x621a30, row = 0x0, current_row = 0x0,
  field_alloc = {free = 0x1fe000000020, used = 0x5, pre_alloc = 0x0, min_malloc = 11, block_size = 0, block_num = 0, first_block_usage = 0, error_handler = 0x1},
  field_count = 1581285888, current_field = 61, eof = 0 '\000', unbuffered_fetch_cancelled = 0 '\000', extension = 0x0}
(gdb) p i
$2 = 6
(gdb) p fields[0]
$10 = {name = 0x621f88 "Id", org_name = 0x621f90 "", table = 0x621f78 "", org_table = 0x621f80 "", db = 0x621f70 "", catalog = 0x621f68 "def", def = 0x0,
  length = 11, max_length = 5, name_length = 2, org_name_length = 0, table_length = 0, org_table_length = 0, db_length = 0, catalog_length = 3, def_length = 0,
  flags = 32897, decimals = 0, charsetnr = 63, type = MYSQL_TYPE_LONGLONG, extension = 0x621fb8}
(gdb) p fields[1]
$11 = {name = 0x621fc0 "", org_name = 0x621fa8 "", table = 0x621fb0 "", org_table = 0x621fa0 "", db = 0x621f98 "def", catalog = 0x0,
  def = 0x10 <Address 0x10 out of bounds>, length = 4, max_length = 4, name_length = 0, org_name_length = 0, table_length = 0, org_table_length = 3, db_length = 0,
  catalog_length = 1, def_length = 31, flags = 8, decimals = 253, charsetnr = 0, type = 6430696, extension = 0x621ff0}
(gdb) p fields[2]
$12 = {name = 0x621fd8 "", org_name = 0x621fe0 "", table = 0x621fd0 "", org_table = 0x621fc8 "def", db = 0x0, catalog = 0x40 <Address 0x40 out of bounds>,
  def = 0x9 <Address 0x9 out of bounds>, length = 4, max_length = 0, name_length = 0, org_name_length = 3, table_length = 0, org_table_length = 1, db_length = 31,
  catalog_length = 8, def_length = 253, flags = 0, decimals = 6430744, charsetnr = 0, type = 6430752, extension = 0x622008}
(gdb) p fields[3]
$13 = {name = 0x622010 "", org_name = 0x622000 "", table = 0x621ff8 "def", org_table = 0x0, db = 0x40 <Address 0x40 out of bounds>,
  catalog = 0x6 <Address 0x6 out of bounds>, def = 0x2 <Address 0x2 out of bounds>, length = 0, max_length = 12884901888, name_length = 0, org_name_length = 0,
  table_length = 31, org_table_length = 8, db_length = 253, catalog_length = 0, def_length = 6430792, flags = 0, decimals = 6430800, charsetnr = 0, type = 6430776,
  extension = 0x622040}
(gdb) p fields[4]
$14 = {name = 0x622030 "", org_name = 0x622028 "def", table = 0x0, org_table = 0x10 <Address 0x10 out of bounds>, db = 0x5 <Address 0x5 out of bounds>,
  catalog = 0x7 <Address 0x7 out of bounds>, def = 0x0, length = 12884901888, max_length = 4294967296, name_length = 31, org_name_length = 8, table_length = 253,
  org_table_length = 0, db_length = 6430840, catalog_length = 0, def_length = 6430848, flags = 0, decimals = 6430824, charsetnr = 0, type = 6430832,
  extension = 0x622060}
(gdb) p fields[5]
$15 = {name = 0x622058 "def", org_name = 0x0, table = 0x7 <Address 0x7 out of bounds>, org_table = 0x3 <Address 0x3 out of bounds>,
  db = 0x4 <Address 0x4 out of bounds>, catalog = 0x0, def = 0x300000000 <Address 0x300000000 out of bounds>, length = 141291539136512, max_length = 270582939648,
  name_length = 3, org_name_length = 0, table_length = 6430888, org_table_length = 0, db_length = 6430896, catalog_length = 0, def_length = 6430872, flags = 0,
  decimals = 6430880, charsetnr = 0, type = 6430864, extension = 0x622088}
(gdb) p fields[6]
$16 = {name = 0x0, org_name = 0x1e <Address 0x1e out of bounds>, table = 0x0, org_table = 0x5 <Address 0x5 out of bounds>, db = 0x0,
  catalog = 0x300000000 <Address 0x300000000 out of bounds>, def = 0x0, length = 34359738399, max_length = 253, name_length = 6430936, org_name_length = 0,
  table_length = 6430944, org_table_length = 0, db_length = 6430920, catalog_length = 0, def_length = 6430928, flags = 0, decimals = 6430912, charsetnr = 0,
  type = 6430904, extension = 0x0}

field_count = 1581285888 sounds pretty odd.

After some digging, it looks that the problem is in the use of libmysqlclient_r :
mydumper uses libmysqlclient_r.so.15 instead of libmysqlclient_r.so.18

Revision history for this message
Andrew Hutchings (linuxjedi) wrote :

Hi Rene,

Thanks for the bug report. Very interesting, I have not tried against Percona yet. It does look like that struct array hasn't been populated correctly/completely.

Can you please provide the output of mysql_config for that system and 'ldd mydumper'? It should help me determine where in the linker that is happening.

Also, what version of mydumper is used? (not that I think I have changed anything that could be related from 0.2 to 0.5)

Revision history for this message
Rene' Cannao' (rene-cannao) wrote :

Hi Andrew,

yes, interesting.
At first I thought it was Percona returning weird incorrect output, but forcing the use of libmysqlclient_r.so.18 instead of libmysqlclient_r.so.15 (in the dirty way, renaming files), mydumper worked fine.
So my guessing is that mydumper compiled against the wrong library.

Here are the outputs:

$ mysql_config
Usage: /usr/bin/mysql_config [OPTIONS]
Options:
        --cflags [-I/usr/include/mysql -fPIC -g -static-libgcc -fno-omit-frame-pointer -fPIC -g -static-libgcc -fno-omit-frame-pointer -fPIC -g -static-libgcc -fno-omit-frame-pointer -fno-strict-aliasing -DMY_PTHREAD_FASTMUTEX=1]
        --include [-I/usr/include/mysql]
        --libs [-L/usr/lib64 -lmysqlclient -lpthread -lm -lrt -ldl]
        --libs_r [-L/usr/lib64 -lmysqlclient_r -lpthread -lm -lrt -ldl]
        --plugindir [/usr/lib64/mysql/plugin]
        --socket [/var/lib/mysql/mysql.sock]
        --port [0]
        --version [5.5.13]
        --libmysqld-libs [-L/usr/lib64 -lmysqld]
        --variable=VAR VAR is one of:
                pkgincludedir [/usr/include/mysql]
                pkglibdir [/usr/lib64]
                plugindir [/usr/lib64/mysql/plugin]

$ ldd mydumper
        linux-vdso.so.1 => (0x00007fff93077000)
        libmysqlclient_r.so.15 => /usr/lib64/libmysqlclient_r.so.15 (0x0000003a3f600000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00000036ae800000)
        libm.so.6 => /lib64/libm.so.6 (0x00000036aec00000)
        librt.so.1 => /lib64/librt.so.1 (0x00000036afc00000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00000036ae400000)
        libglib-2.0.so.0 => /lib64/libglib-2.0.so.0 (0x00000036b0400000)
        libgthread-2.0.so.0 => /lib64/libgthread-2.0.so.0 (0x00000036b5000000)
        libpcre.so.0 => /lib64/libpcre.so.0 (0x000000370f400000)
        libz.so.1 => /usr/lib64/libz.so.1 (0x00000036af000000)
        libc.so.6 => /lib64/libc.so.6 (0x00000036ae000000)
        libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00000036b1000000)
        libnsl.so.1 => /lib64/libnsl.so.1 (0x00000036b0000000)
        /lib64/ld-linux-x86-64.so.2 (0x00000036adc00000)

Version used is the latest from trunk . Couldn't use 0.5 because I was hitting this other bug:
https://bugs.launchpad.net/mydumper/+bug/798536

Revision history for this message
Rene' Cannao' (rene-cannao) wrote :
Download full text (3.7 KiB)

Andrew, few more info:

$ ls -l /usr/lib64/libmysqlclient*
lrwxrwxrwx 1 root root 26 Jul 14 20:08 /usr/lib64/libmysqlclient_r.so -> libmysqlclient_r.so.15.0.0
lrwxrwxrwx 1 root root 26 Jul 20 09:05 /usr/lib64/libmysqlclient_r.so.15 -> libmysqlclient_r.so.15.0.0
-rwxr-xr-x 1 root root 2135504 Feb 17 22:14 /usr/lib64/libmysqlclient_r.so.15.0.0
lrwxrwxrwx 1 root root 17 Jul 14 20:08 /usr/lib64/libmysqlclient_r.so.18 -> libmysqlclient.so
lrwxrwxrwx 1 root root 17 Jul 14 20:08 /usr/lib64/libmysqlclient_r.so.18.0.0 -> libmysqlclient.so
lrwxrwxrwx 1 root root 24 Jul 14 20:08 /usr/lib64/libmysqlclient.so -> libmysqlclient.so.15.0.0
lrwxrwxrwx 1 root root 24 Jul 14 20:08 /usr/lib64/libmysqlclient.so.15 -> libmysqlclient.so.15.0.0
-rwxr-xr-x 1 root root 2128552 Feb 17 22:14 /usr/lib64/libmysqlclient.so.15.0.0
lrwxrwxrwx 1 root root 22 Jul 20 08:50 /usr/lib64/libmysqlclient.so.18 -> libmysqlclient_r.so.15
-rwxr-xr-x 1 root root 3509232 Jul 2 06:17 /usr/lib64/libmysqlclient.so.18.0.0

$ rpm -qa | egrep -i '(percona|mysql)'
percona-release-0.0-1
Percona-Server-devel-55-5.5.13-rel20.4.138.rhel5
Percona-SQL-shared-50-5.0.92-b23.85.rhel5
Percona-Server-client-55-5.5.13-rel20.4.138.rhel5
php53-mysql-5.3.3-1.el5_6.1
Percona-Server-shared-55-5.5.13-rel20.4.138.rhel5
Percona-Server-server-55-5.5.13-rel20.4.138.rhel5

$ rpm -ql Percona-SQL-shared-50-5.0.92-b23.85.rhel5
/usr/lib64/libmysqlclient.so
/usr/lib64/libmysqlclient.so.15
/usr/lib64/libmysqlclient.so.15.0.0
/usr/lib64/libmysqlclient_r.so
/usr/lib64/libmysqlclient_r.so.15
/usr/lib64/libmysqlclient_r.so.15.0.0

Tried also this:
yum remove Percona-SQL-shared-50-5.0.92-b23.85.rhel5
yum install Percona-Server-shared-compat

But mydumper compiles always against libmysqlclient_r.so.15 (that is still the predefined one):
$ ls -l /usr/lib64/libmysqlclient*
lrwxrwxrwx 1 root root 26 Jul 14 20:08 /usr/lib64/libmysqlclient_r.so -> libmysqlclient_r.so.15.0.0
lrwxrwxrwx 1 root root 26 Jul 20 10:21 /usr/lib64/libmysqlclient_r.so.12 -> libmysqlclient_r.so.12.0.0
-rwxr-xr-x 1 root root 262344 Jul 19 15:33 /usr/lib64/libmysqlclient_r.so.12.0.0
lrwxrwxrwx 1 root root 26 Jul 20 10:21 /usr/lib64/libmysqlclient_r.so.14 -> libmysqlclient_r.so.14.0.0
-rwxr-xr-x 1 root root 1239792 Jul 19 15:33 /usr/lib64/libmysqlclient_r.so.14.0.0
lrwxrwxrwx 1 root root 26 Jul 20 10:21 /usr/lib64/libmysqlclient_r.so.15 -> libmysqlclient_r.so.15.0.0
-rwxr-xr-x 1 root root 2136808 Jul 19 15:33 /usr/lib64/libmysqlclient_r.so.15.0.0
lrwxrwxrwx 1 root root 26 Jul 20 10:21 /usr/lib64/libmysqlclient_r.so.16 -> libmysqlclient_r.so.16.0.0
-rwxr-xr-x 1 root root 2157840 Jul 19 15:33 /usr/lib64/libmysqlclient_r.so.16.0.0
lrwxrwxrwx 1 root root 24 Jul 14 20:08 /usr/lib64/libmysqlclient.so -> libmysqlclient.so.15.0.0
lrwxrwxrwx 1 root root 24 Jul 20 10:21 /usr/lib64/libmysqlclient.so.12 -> libmysqlclient.so.12.0.0
-rwxr-xr-x 1 root root 253888 Jul 19 15:33 /usr/lib64/libmysqlclient.so.12.0.0
lrwxrwxrwx 1 root root 24 Jul 20 10:21 /usr/lib64/libmysqlclient.so.14 -> libmysqlclient.so.14.0.0
-rwxr-xr-x 1 root root 1231432 Jul 19 15:33 /usr/lib64/libmysqlclient.so.14.0.0
lrwxrwxrwx 1...

Read more...

Revision history for this message
Andrew Hutchings (linuxjedi) wrote :

So, what should happen is it should resolve /usr/lib64/libmysqlclient_r.so to whatever it points to, link to that and rpath it. I remember there being an ABI problem in 5.5, but I thought that was fixed now.

I'm going to have to try and reproduce this to see what is going on. Unfortunately due to internal commitments I cannot do this until probably mid-August. But I'll mark it as high priority to investigate.

Changed in mydumper:
importance: Undecided → High
assignee: nobody → Andrew Hutchings (linuxjedi)
Changed in mydumper:
importance: High → Critical
Revision history for this message
Andrew Hutchings (linuxjedi) wrote :

OK, I found a bug in the MySQL CMake script which is causing this...

Revision history for this message
Andrew Hutchings (linuxjedi) wrote :

Fixed in trunk rev. 102.

Changed in mydumper:
status: New → Fix Released
Revision history for this message
Andrew Hutchings (linuxjedi) wrote :

This was caused by a bug in the CMake script. The script was using mysql_config for the include and library directories but if there was a different libmysqlclient_r in a system library directory (such as /usr/lib) that was overriding the data supplied by mysql_config.

This means that the structs specified in the include file are different to those in the library causing this segfault.

Revision history for this message
Rene' Cannao' (rene-cannao) wrote :

Thanks Andrew!

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Related questions

Remote bug watches

Bug watches keep track of this bug in other bug trackers.