Galera after update to 2.0 + mysql 5.5 crash

Asked by Steven on 2012-03-08

Hi,

We have just updated from 0.8x + mysql 5.1 to 2.0 + 5.5. It seemed to be fine however today suddenly it was giving an error 'unknown command' with just selecting an mysql db.

I checked logs and here is the part where I think all goes wrong:

120306 9:49:45 [Note] WSREP: declaring 25a2f3ea-66c9-11e1-0800-aaf7ad456b15 stable
120306 9:49:45 [Note] WSREP: declaring b3157d8d-6771-11e1-0800-4c20bc75ca8d stable
120306 9:49:45 [Note] WSREP: New COMPONENT: primary = yes, my_idx = 1, memb_num = 3
120306 9:49:45 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
120306 9:49:46 [Note] WSREP: STATE EXCHANGE: sent state msg: b3696b6b-6771-11e1-0800-636cc2d87f37
120306 9:49:46 [Note] WSREP: STATE EXCHANGE: got state msg: b3696b6b-6771-11e1-0800-636cc2d87f37 from 0 (webserver.unlocksamsungonline.eu)
120306 9:49:46 [Note] WSREP: STATE EXCHANGE: got state msg: b3696b6b-6771-11e1-0800-636cc2d87f37 from 1 (webserver.unlocksamsungonline.com)
120306 9:49:46 [Note] WSREP: STATE EXCHANGE: got state msg: b3696b6b-6771-11e1-0800-636cc2d87f37 from 2 (mysql)
120306 9:49:46 [Note] WSREP: Quorum results:
 version = 2,
 component = PRIMARY,
 conf_id = 6,
 members = 3/3 (joined/total),
 act_id = 122788,
 last_appl. = 0,
 protocols = 0/3/1 (gcs/repl/appl),
 group UUID = 00d090b7-66c3-11e1-0800-23e68969619b
120306 9:49:46 [Note] WSREP: Flow-control interval: [14, 28]
120306 9:49:46 [Note] WSREP: New cluster view: global state: 00d090b7-66c3-11e1-0800-23e68969619b:122788, view# 7: Primary, number of nodes: 3, my index: 1, protocol version 1
120306 9:49:46 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
120306 9:49:46 [Note] WSREP: Assign initial position for certification: 122788, protocol version: 2
120306 9:49:46 [Note] WSREP: Member 2 (mysql) synced with group.
120306 14:38:48 [Note] WSREP: (52321e04-6770-11e1-0800-39f9cf850209, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://83.170.88.228:4567
120306 14:38:49 [Note] WSREP: (52321e04-6770-11e1-0800-39f9cf850209, 'tcp://0.0.0.0:4567') reconnecting to 25a2f3ea-66c9-11e1-0800-aaf7ad456b15 (tcp://83.170.88.228:4567), attempt 0
120306 14:38:50 [Note] WSREP: (52321e04-6770-11e1-0800-39f9cf850209, 'tcp://0.0.0.0:4567') turning message relay requesting off
120306 14:38:50 [Note] WSREP: (52321e04-6770-11e1-0800-39f9cf850209, 'tcp://0.0.0.0:4567') cleaning up duplicate 0x7f76c401bde0 after established 0x7f76b401f7d0
120307 5:01:55 [Note] WSREP: (52321e04-6770-11e1-0800-39f9cf850209, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://46.23.72.59:4567
120307 5:01:56 [Note] WSREP: (52321e04-6770-11e1-0800-39f9cf850209, 'tcp://0.0.0.0:4567') reconnecting to b3157d8d-6771-11e1-0800-4c20bc75ca8d (tcp://46.23.72.59:4567), attempt 0
120307 5:01:59 [Note] WSREP: (52321e04-6770-11e1-0800-39f9cf850209, 'tcp://0.0.0.0:4567') turning message relay requesting off
120307 5:02:00 [Note] WSREP: (52321e04-6770-11e1-0800-39f9cf850209, 'tcp://0.0.0.0:4567') cleaning up duplicate 0x7f76c401a850 after established 0x7f76c401c420
120307 23:53:54 [Note] WSREP: New COMPONENT: primary = yes, my_idx = 1, memb_num = 2
120307 23:53:54 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
120307 23:53:54 [Note] WSREP: GMCast::handle_stable_view: view(view_id(PRIM,25a2f3ea-66c9-11e1-0800-aaf7ad456b15,12) memb {
 25a2f3ea-66c9-11e1-0800-aaf7ad456b15,
 52321e04-6770-11e1-0800-39f9cf850209,
} joined {
} left {
} partitioned {
 b3157d8d-6771-11e1-0800-4c20bc75ca8d,
})
120307 23:53:54 [Note] WSREP: forgetting b3157d8d-6771-11e1-0800-4c20bc75ca8d (tcp://46.23.72.59:4567)
120307 23:53:54 [Note] WSREP: declaring 25a2f3ea-66c9-11e1-0800-aaf7ad456b15 stable
120307 23:53:56 [Warning] WSREP: discarding established (time wait) b3157d8d-6771-11e1-0800-4c20bc75ca8d (tcp://46.23.72.59:4567)
120307 23:53:58 [Warning] WSREP: discarding established (time wait) b3157d8d-6771-11e1-0800-4c20bc75ca8d (tcp://46.23.72.59:4567)
120307 23:53:59 [Note] WSREP: cleaning up b3157d8d-6771-11e1-0800-4c20bc75ca8d (tcp://46.23.72.59:4567)
120307 23:59:53 [Warning] WSREP: evs::proto(52321e04-6770-11e1-0800-39f9cf850209, GATHER, view_id(REG,25a2f3ea-66c9-11e1-0800-aaf7ad456b15,12)) install timer expired
evs::proto(evs::proto(52321e04-6770-11e1-0800-39f9cf850209, GATHER, view_id(REG,25a2f3ea-66c9-11e1-0800-aaf7ad456b15,12)), GATHER) {
current_view=view(view_id(REG,25a2f3ea-66c9-11e1-0800-aaf7ad456b15,12) memb {
 25a2f3ea-66c9-11e1-0800-aaf7ad456b15,
 52321e04-6770-11e1-0800-39f9cf850209,
} joined {
} left {
} partitioned {
}),
input_map=evs::input_map: {aru_seq=1,safe_seq=1,node_index=node: {idx=0,range=[2,1],safe_seq=1} node: {idx=1,range=[2,1],safe_seq=1} ,msg_index=,recovery_index= (0,1),evs::msg{version=0,type=1,user_type=255,order=4,seq=1,seq_range=0,aru_seq=0,flags=4,source=25a2f3ea-66c9-11e1-0800-aaf7ad456b15,source_view_id=view_id(REG,25a2f3ea-66c9-11e1-0800-aaf7ad456b15,12),range_uuid=00000000-0000-0000-0000-000000000000,range=[-1,-1],fifo_seq=484011,node_list=()
}
 (1,1),evs::msg{version=0,type=1,user_type=255,order=0,seq=1,seq_range=0,aru_seq=0,flags=0,source=52321e04-6770-11e1-0800-39f9cf850209,source_view_id=view_id(REG,25a2f3ea-66c9-11e1-0800-aaf7ad456b15,12),range_uuid=00000000-0000-0000-0000-000000000000,range=[-1,-1],fifo_seq=322373,node_list=()
}
},
fifo_seq=323557,
last_sent=1,
known={
 25a2f3ea-66c9-11e1-0800-aaf7ad456b15,evs::node{operational=1,suspected=0,installed=0,fifo_seq=485200,join_message=
evs::msg{version=0,type=4,user_type=255,order=1,seq=1,seq_range=-1,aru_seq=1,flags=4,source=25a2f3ea-66c9-11e1-0800-aaf7ad456b15,source_view_id=view_id(REG,25a2f3ea-66c9-11e1-0800-aaf7ad456b15,12),range_uuid=00000000-0000-0000-0000-000000000000,range=[-1,-1],fifo_seq=485200,node_list=( 25a2f3ea-66c9-11e1-0800-aaf7ad456b15,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,25a2f3ea-66c9-11e1-0800-aaf7ad456b15,12),safe_seq=1,im_range=[2,1],}
 52321e04-6770-11e1-0800-39f9cf850209,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,25a2f3ea-66c9-11e1-0800-aaf7ad456b15,12),safe_seq=1,im_range=[2,1],}
 b3157d8d-6771-11e1-0800-4c20bc75ca8d,node: {operational=1,suspected=1,leave_seq=-1,view_id=view_id(REG,00000000-0000-0000-0000-000000000000,0),safe_seq=-1,im_range=[-1,-1],}
)
},
}
 52321e04-6770-11e1-0800-39f9cf850209,evs::node{operational=1,suspected=0,installed=0,fifo_seq=-1,join_message=
evs::msg{version=0,type=4,user_type=255,order=1,seq=1,seq_range=-1,aru_seq=1,flags=0,source=52321e04-6770-11e1-0800-39f9cf850209,source_view_id=view_id(REG,25a2f3ea-66c9-11e1-0800-aaf7ad456b15,12),range_uuid=00000000-0000-0000-0000-000000000000,range=[-1,-1],fifo_seq=323557,node_list=( 25a2f3ea-66c9-11e1-0800-aaf7ad456b15,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,25a2f3ea-66c9-11e1-0800-aaf7ad456b15,12),safe_seq=1,im_range=[2,1],}
 52321e04-6770-11e1-0800-39f9cf850209,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,25a2f3ea-66c9-11e1-0800-aaf7ad456b15,12),safe_seq=1,im_range=[2,1],}
 b3157d8d-6771-11e1-0800-4c20bc75ca8d,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,b3157d8d-6771-11e1-0800-4c20bc75ca8d,12),safe_seq=0,im_range=[1,0],}
)
},
}
 b3157d8d-6771-11e1-0800-4c20bc75ca8d,evs::node{operational=1,suspected=0,installed=0,fifo_seq=320489,join_message=
evs::msg{version=0,type=4,user_type=255,order=1,seq=0,seq_range=-1,aru_seq=0,flags=4,source=b3157d8d-6771-11e1-0800-4c20bc75ca8d,source_view_id=view_id(REG,b3157d8d-6771-11e1-0800-4c20bc75ca8d,12),range_uuid=00000000-0000-0000-0000-000000000000,range=[-1,-1],fifo_seq=320489,node_list=( 25a2f3ea-66c9-11e1-0800-aaf7ad456b15,node: {operational=1,suspected=1,leave_seq=-1,view_id=view_id(REG,00000000-0000-0000-0000-000000000000,0),safe_seq=-1,im_range=[-1,-1],}
 52321e04-6770-11e1-0800-39f9cf850209,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,25a2f3ea-66c9-11e1-0800-aaf7ad456b15,12),safe_seq=1,im_range=[2,1],}
 b3157d8d-6771-11e1-0800-4c20bc75ca8d,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,b3157d8d-6771-11e1-0800-4c20bc75ca8d,12),safe_seq=0,im_range=[1,0],}
)
},
}
 }
 }
120307 23:59:53 [Note] WSREP: no install message received
120307 23:59:53 [Note] WSREP: New COMPONENT: primary = no, my_idx = 0, memb_num = 1
120307 23:59:53 [Note] WSREP: Flow-control interval: [8, 16]
120307 23:59:53 [Note] WSREP: Received NON-PRIMARY.
120307 23:59:53 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 126389)
120307 23:59:53 [Note] WSREP: GMCast::handle_stable_view: view(view_id(NON_PRIM,25a2f3ea-66c9-11e1-0800-aaf7ad456b15,12) memb {
 52321e04-6770-11e1-0800-39f9cf850209,
} joined {
} left {
} partitioned {
 25a2f3ea-66c9-11e1-0800-aaf7ad456b15,
})
120307 23:59:53 [Note] WSREP: New COMPONENT: primary = no, my_idx = 0, memb_num = 1
120307 23:59:53 [Note] WSREP: Flow-control interval: [8, 16]
120307 23:59:53 [Note] WSREP: Received NON-PRIMARY.
120307 23:59:53 [Note] WSREP: GMCast::handle_stable_view: view(view_id(NON_PRIM,52321e04-6770-11e1-0800-39f9cf850209,13) memb {
 52321e04-6770-11e1-0800-39f9cf850209,
} joined {
} left {
} partitioned {
 25a2f3ea-66c9-11e1-0800-aaf7ad456b15,
})
120307 23:59:53 [Warning] WSREP: evs::proto(52321e04-6770-11e1-0800-39f9cf850209, GATHER, view_id(REG,52321e04-6770-11e1-0800-39f9cf850209,13)) source b3157d8d-6771-11e1-0800-4c20bc75ca8d is not supposed to be representative
120307 23:59:54 [Note] WSREP: New cluster view: global state: 00d090b7-66c3-11e1-0800-23e68969619b:126389, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 1
120307 23:59:54 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
120307 23:59:54 [Note] WSREP: New cluster view: global state: 00d090b7-66c3-11e1-0800-23e68969619b:126389, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 1
120307 23:59:54 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
120308 0:05:53 [Warning] WSREP: evs::proto(52321e04-6770-11e1-0800-39f9cf850209, GATHER, view_id(REG,52321e04-6770-11e1-0800-39f9cf850209,13)) install timer expired
evs::proto(evs::proto(52321e04-6770-11e1-0800-39f9cf850209, GATHER, view_id(REG,52321e04-6770-11e1-0800-39f9cf850209,13)), GATHER) {
current_view=view(view_id(REG,52321e04-6770-11e1-0800-39f9cf850209,13) memb {
 52321e04-6770-11e1-0800-39f9cf850209,
} joined {
} left {
} partitioned {
}),
input_map=evs::input_map: {aru_seq=0,safe_seq=0,node_index=node: {idx=0,range=[1,0],safe_seq=0} ,msg_index=,recovery_index= (0,0),evs::msg{version=0,type=1,user_type=255,order=4,seq=0,seq_range=0,aru_seq=-1,flags=0,source=52321e04-6770-11e1-0800-39f9cf850209,source_view_id=view_id(REG,52321e04-6770-11e1-0800-39f9cf850209,13),range_uuid=00000000-0000-0000-0000-000000000000,range=[-1,-1],fifo_seq=323562,node_list=()
}
},
fifo_seq=324755,
last_sent=0,
known={
 25a2f3ea-66c9-11e1-0800-aaf7ad456b15,evs::node{operational=1,suspected=0,installed=0,fifo_seq=486395,join_message=
evs::msg{version=0,type=4,user_type=255,order=1,seq=0,seq_range=-1,aru_seq=0,flags=4,source=25a2f3ea-66c9-11e1-0800-aaf7ad456b15,source_view_id=view_id(REG,25a2f3ea-66c9-11e1-0800-aaf7ad456b15,13),range_uuid=00000000-0000-0000-0000-000000000000,range=[-1,-1],fifo_seq=486395,node_list=( 25a2f3ea-66c9-11e1-0800-aaf7ad456b15,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,25a2f3ea-66c9-11e1-0800-aaf7ad456b15,13),safe_seq=0,im_range=[1,0],}
 52321e04-6770-11e1-0800-39f9cf850209,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,52321e04-6770-11e1-0800-39f9cf850209,13),safe_seq=0,im_range=[1,0],}
 b3157d8d-6771-11e1-0800-4c20bc75ca8d,node: {operational=1,suspected=1,leave_seq=-1,view_id=view_id(REG,00000000-0000-0000-0000-000000000000,0),safe_seq=-1,im_range=[-1,-1],}
)
},
}
 52321e04-6770-11e1-0800-39f9cf850209,evs::node{operational=1,suspected=0,installed=0,fifo_seq=-1,join_message=
evs::msg{version=0,type=4,user_type=255,order=1,seq=0,seq_range=-1,aru_seq=0,flags=0,source=52321e04-6770-11e1-0800-39f9cf850209,source_view_id=view_id(REG,52321e04-6770-11e1-0800-39f9cf850209,13),range_uuid=00000000-0000-0000-0000-000000000000,range=[-1,-1],fifo_seq=324755,node_list=( 25a2f3ea-66c9-11e1-0800-aaf7ad456b15,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,25a2f3ea-66c9-11e1-0800-aaf7ad456b15,13),safe_seq=0,im_range=[1,0],}
 52321e04-6770-11e1-0800-39f9cf850209,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,52321e04-6770-11e1-0800-39f9cf850209,13),safe_seq=0,im_range=[1,0],}
 b3157d8d-6771-11e1-0800-4c20bc75ca8d,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,b3157d8d-6771-11e1-0800-4c20bc75ca8d,13),safe_seq=0,im_range=[1,0],}
)
},
}
 b3157d8d-6771-11e1-0800-4c20bc75ca8d,evs::node{operational=1,suspected=0,installed=0,fifo_seq=321686,join_message=
evs::msg{version=0,type=4,user_type=255,order=1,seq=0,seq_range=-1,aru_seq=0,flags=4,source=b3157d8d-6771-11e1-0800-4c20bc75ca8d,source_view_id=view_id(REG,b3157d8d-6771-11e1-0800-4c20bc75ca8d,13),range_uuid=00000000-0000-0000-0000-000000000000,range=[-1,-1],fifo_seq=321686,node_list=( 25a2f3ea-66c9-11e1-0800-aaf7ad456b15,node: {operational=0,suspected=1,leave_seq=-1,view_id=view_id(REG,00000000-0000-0000-0000-000000000000,0),safe_seq=-1,im_range=[-1,-1],}
 52321e04-6770-11e1-0800-39f9cf850209,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,52321e04-6770-11e1-0800-39f9cf850209,13),safe_seq=0,im_range=[1,0],}
 b3157d8d-6771-11e1-0800-4c20bc75ca8d,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,b3157d8d-6771-11e1-0800-4c20bc75ca8d,13),safe_seq=0,im_range=[1,0],}
)
},
}
 }
 }
120308 0:05:54 [Note] WSREP: no install message received
120308 0:05:54 [Warning] WSREP: subsequent views have same members, prev view view(view_id(REG,52321e04-6770-11e1-0800-39f9cf850209,13) memb {
 52321e04-6770-11e1-0800-39f9cf850209,
} joined {
} left {
} partitioned {
}) current view view(view_id(REG,52321e04-6770-11e1-0800-39f9cf850209,14) memb {
 52321e04-6770-11e1-0800-39f9cf850209,
} joined {
} left {
} partitioned {
})
120308 0:05:54 [Note] WSREP: New COMPONENT: primary = no, my_idx = 0, memb_num = 1
120308 0:05:54 [Note] WSREP: Flow-control interval: [8, 16]
120308 0:05:54 [Note] WSREP: Received NON-PRIMARY.
120308 0:05:54 [Note] WSREP: New cluster view: global state: 00d090b7-66c3-11e1-0800-23e68969619b:126389, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 1
120308 0:05:54 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
120308 0:05:54 [Note] WSREP: GMCast::handle_stable_view: view(view_id(NON_PRIM,52321e04-6770-11e1-0800-39f9cf850209,14) memb {
 52321e04-6770-11e1-0800-39f9cf850209,
} joined {
} left {
} partitioned {
 25a2f3ea-66c9-11e1-0800-aaf7ad456b15,
})
120308 0:05:54 [Warning] WSREP: evs::proto(52321e04-6770-11e1-0800-39f9cf850209, GATHER, view_id(REG,52321e04-6770-11e1-0800-39f9cf850209,14)) source b3157d8d-6771-11e1-0800-4c20bc75ca8d is not supposed to be representative
120308 0:11:54 [Warning] WSREP: evs::proto(52321e04-6770-11e1-0800-39f9cf850209, GATHER, view_id(REG,52321e04-6770-11e1-0800-39f9cf850209,14)) source b3157d8d-6771-11e1-0800-4c20bc75ca8d is not supposed to be representative
120308 0:17:54 [Warning] WSREP: evs::proto(52321e04-6770-11e1-0800-39f9cf850209, GATHER, view_id(REG,52321e04-6770-11e1-0800-39f9cf850209,14)) install timer expired
evs::proto(evs::proto(52321e04-6770-11e1-0800-39f9cf850209, GATHER, view_id(REG,52321e04-6770-11e1-0800-39f9cf850209,14)), GATHER) {
current_view=view(view_id(REG,52321e04-6770-11e1-0800-39f9cf850209,14) memb {
 52321e04-6770-11e1-0800-39f9cf850209,
} joined {
} left {
} partitioned {
}),
input_map=evs::input_map: {aru_seq=0,safe_seq=0,node_index=node: {idx=0,range=[1,0],safe_seq=0} ,msg_index=,recovery_index= (0,0),evs::msg{version=0,type=1,user_type=255,order=4,seq=0,seq_range=0,aru_seq=-1,flags=0,source=52321e04-6770-11e1-0800-39f9cf850209,source_view_id=view_id(REG,52321e04-6770-11e1-0800-39f9cf850209,14),range_uuid=00000000-0000-0000-0000-000000000000,range=[-1,-1],fifo_seq=324760,node_list=()
}
},
fifo_seq=327140,
last_sent=0,
known={
 25a2f3ea-66c9-11e1-0800-aaf7ad456b15,evs::node{operational=1,suspected=0,installed=0,fifo_seq=488775,join_message=
evs::msg{version=0,type=4,user_type=255,order=1,seq=0,seq_range=-1,aru_seq=0,flags=4,source=25a2f3ea-66c9-11e1-0800-aaf7ad456b15,source_view_id=view_id(REG,25a2f3ea-66c9-11e1-0800-aaf7ad456b15,14),range_uuid=00000000-0000-0000-0000-000000000000,range=[-1,-1],fifo_seq=488775,node_list=( 25a2f3ea-66c9-11e1-0800-aaf7ad456b15,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,25a2f3ea-66c9-11e1-0800-aaf7ad456b15,14),safe_seq=0,im_range=[1,0],}
 52321e04-6770-11e1-0800-39f9cf850209,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,52321e04-6770-11e1-0800-39f9cf850209,14),safe_seq=0,im_range=[1,0],}
 b3157d8d-6771-11e1-0800-4c20bc75ca8d,node: {operational=1,suspected=1,leave_seq=-1,view_id=view_id(REG,00000000-0000-0000-0000-000000000000,0),safe_seq=-1,im_range=[-1,-1],}
)
},
}
 52321e04-6770-11e1-0800-39f9cf850209,evs::node{operational=1,suspected=0,installed=0,fifo_seq=-1,join_message=
evs::msg{version=0,type=4,user_type=255,order=1,seq=0,seq_range=-1,aru_seq=0,flags=0,source=52321e04-6770-11e1-0800-39f9cf850209,source_view_id=view_id(REG,52321e04-6770-11e1-0800-39f9cf850209,14),range_uuid=00000000-0000-0000-0000-000000000000,range=[-1,-1],fifo_seq=327140,node_list=( 25a2f3ea-66c9-11e1-0800-aaf7ad456b15,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,25a2f3ea-66c9-11e1-0800-aaf7ad456b15,14),safe_seq=0,im_range=[1,0],}
 52321e04-6770-11e1-0800-39f9cf850209,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,52321e04-6770-11e1-0800-39f9cf850209,14),safe_seq=0,im_range=[1,0],}
 b3157d8d-6771-11e1-0800-4c20bc75ca8d,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,b3157d8d-6771-11e1-0800-4c20bc75ca8d,15),safe_seq=0,im_range=[1,0],}
)
},
}
 b3157d8d-6771-11e1-0800-4c20bc75ca8d,evs::node{operational=1,suspected=0,installed=0,fifo_seq=324079,join_message=
evs::msg{version=0,type=4,user_type=255,order=1,seq=0,seq_range=-1,aru_seq=0,flags=4,source=b3157d8d-6771-11e1-0800-4c20bc75ca8d,source_view_id=view_id(REG,b3157d8d-6771-11e1-0800-4c20bc75ca8d,15),range_uuid=00000000-0000-0000-0000-000000000000,range=[-1,-1],fifo_seq=324079,node_list=( 25a2f3ea-66c9-11e1-0800-aaf7ad456b15,node: {operational=1,suspected=1,leave_seq=-1,view_id=view_id(REG,00000000-0000-0000-0000-000000000000,0),safe_seq=-1,im_range=[-1,-1],}
 52321e04-6770-11e1-0800-39f9cf850209,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,52321e04-6770-11e1-0800-39f9cf850209,14),safe_seq=0,im_range=[1,0],}
 b3157d8d-6771-11e1-0800-4c20bc75ca8d,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,b3157d8d-6771-11e1-0800-4c20bc75ca8d,15),safe_seq=0,im_range=[1,0],}
)
},
}
 }
 }
120308 0:17:54 [Note] WSREP: no install message received
120308 0:17:54 [Warning] WSREP: subsequent views have same members, prev view view(view_id(REG,52321e04-6770-11e1-0800-39f9cf850209,14) memb {
 52321e04-6770-11e1-0800-39f9cf850209,
} joined {
} left {
} partitioned {
}) current view view(view_id(REG,52321e04-6770-11e1-0800-39f9cf850209,15) memb {
 52321e04-6770-11e1-0800-39f9cf850209,
} joined {
} left {
} partitioned {
})
120308 0:17:54 [Note] WSREP: New COMPONENT: primary = no, my_idx = 0, memb_num = 1
120308 0:17:54 [Note] WSREP: Flow-control interval: [8, 16]
120308 0:17:54 [Note] WSREP: Received NON-PRIMARY.
120308 0:17:54 [Note] WSREP: New cluster view: global state: 00d090b7-66c3-11e1-0800-23e68969619b:126389, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 1
120308 0:17:54 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
120308 0:17:54 [Note] WSREP: GMCast::handle_stable_view: view(view_id(NON_PRIM,52321e04-6770-11e1-0800-39f9cf850209,15) memb {
 52321e04-6770-11e1-0800-39f9cf850209,
} joined {
} left {
} partitioned {
 25a2f3ea-66c9-11e1-0800-aaf7ad456b15,
})
120308 0:17:54 [Warning] WSREP: evs::proto(52321e04-6770-11e1-0800-39f9cf850209, GATHER, view_id(REG,52321e04-6770-11e1-0800-39f9cf850209,15)) source b3157d8d-6771-11e1-0800-4c20bc75ca8d is not supposed to be representative
120308 0:19:13 [Note] /opt/galera/mysql/sbin/mysqld: Normal shutdown

120308 0:19:13 [Note] WSREP: Stop replication
120308 0:19:13 [Note] WSREP: Closing send monitor...
120308 0:19:13 [Note] WSREP: Closed send monitor.
120308 0:19:13 [Note] WSREP: gcomm: terminating thread
120308 0:19:13 [Note] WSREP: gcomm: joining thread
120308 0:19:13 [Note] WSREP: gcomm: closing backend
120308 0:19:13 [Note] WSREP: evs::proto(52321e04-6770-11e1-0800-39f9cf850209, LEAVING, view_id(REG,52321e04-6770-11e1-0800-39f9cf850209,15)) uuid 25a2f3ea-66c9-11e1-0800-aaf7ad456b15 missing from install message, assuming partitioned
120308 0:19:13 [Note] WSREP: evs::proto(52321e04-6770-11e1-0800-39f9cf850209, LEAVING, view_id(REG,52321e04-6770-11e1-0800-39f9cf850209,15)) uuid b3157d8d-6771-11e1-0800-4c20bc75ca8d missing from install message, assuming partitioned
120308 0:19:13 [Note] WSREP: GMCast::handle_stable_view: view((empty))
120308 0:19:13 [Note] WSREP: Received self-leave message.
120308 0:19:13 [Note] WSREP: gcomm: closed
120308 0:19:13 [Note] WSREP: Flow-control interval: [0, 0]
120308 0:19:13 [Note] WSREP: Received SELF-LEAVE. Closing connection.
120308 0:19:13 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 126389)
120308 0:19:13 [Note] WSREP: RECV thread exiting 0: Success
120308 0:19:13 [Note] WSREP: recv_thread() joined.
120308 0:19:13 [Note] WSREP: New cluster view: global state: 00d090b7-66c3-11e1-0800-23e68969619b:126389, view# -1: non-Primary, number of nodes: 0, my index: -1, protocol version 1
120308 0:19:13 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
120308 0:19:13 [Note] WSREP: Closing slave action queue.
120308 0:19:13 [Note] WSREP: applier thread exiting (code:0)
120308 0:19:15 [Note] WSREP: rollbacker thread exiting
120308 0:19:15 [Note] Event Scheduler: Purging the queue. 0 events
120308 0:19:15 [Note] WSREP: dtor state: CLOSED
120308 0:19:15 [Note] WSREP: apply mon: entered 0
120308 0:19:15 [Note] WSREP: apply mon: entered 0
120308 0:19:15 [Note] WSREP: mon: entered 3744 oooe fraction 0 oool fraction 0
120308 0:19:15 [Note] WSREP: cert index usage at exit 630
120308 0:19:15 [Note] WSREP: cert trx map usage at exit 678
120308 0:19:15 [Note] WSREP: deps set usage at exit 0
120308 0:19:15 [Note] WSREP: avg deps dist 600.994
120308 0:19:15 [Note] WSREP: wsdb trx map usage 0 conn query map usage 0
120308 0:19:15 [Note] WSREP: Shifting CLOSED -> DESTROYED (TO: 126389)
120308 0:19:15 [Note] WSREP: Flushing memory map to disk...
120308 0:19:15 InnoDB: Starting shutdown...
120308 0:19:18 InnoDB: Shutdown completed; log sequence number 62771429
120308 0:19:18 [Note] /opt/galera/mysql/sbin/mysqld: Shutdown complete

120308 0:19:30 [Warning] The syntax '--log-slow-queries' is deprecated and will be removed in a future release. Please use '--slow-query-log'/'--slow-query-log-file' instead.
120308 0:19:30 [Warning] Ignoring user change to 'root' because the user was set to 'mysql' earlier on the command line

120308 0:19:30 InnoDB: The InnoDB memory heap is disabled
120308 0:19:30 InnoDB: Mutexes and rw_locks use GCC atomic builtins
120308 0:19:30 InnoDB: Compressed tables use zlib 1.2.3
120308 0:19:30 InnoDB: Using Linux native AIO
120308 0:19:30 InnoDB: Initializing buffer pool, size = 420.0M
120308 0:19:30 InnoDB: Completed initialization of buffer pool
120308 0:19:30 InnoDB: highest supported file format is Barracuda.
120308 0:19:32 InnoDB: Waiting for the background threads to start
120308 0:19:33 InnoDB: 1.1.8 started; log sequence number 62771429
120308 0:19:34 [Note] Event Scheduler: Loaded 0 events
120308 0:19:34 [Note] WSREP: wsrep_load(): loading provider library '/opt/galera/galera/lib/libgalera_smm.so'
120308 0:19:34 [Note] WSREP: wsrep_load(): Galera 23.2.0(r120) by Codership Oy <email address hidden> loaded succesfully.
120308 0:19:34 [Note] WSREP: Reusing existing '/opt/galera/mysql/var//galera.cache'.
120308 0:19:34 [Note] WSREP: Passing config to GCS: evs.consensus_timeout = PT7M; evs.inactive_timeout = PT6M; evs.suspect_timeout = PT5M; gcache.dir = /opt/galera/mysql/var/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /opt/galera/mysql/var//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 0.5; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; replicator.causal_read_timeout = PT30S; replicator.commit_order = 3
120308 0:19:34 [Note] WSREP: Start replication
120308 0:19:34 [Note] WSREP: Found saved state: 00d090b7-66c3-11e1-0800-23e68969619b:126389
120308 0:19:34 [Note] WSREP: Assign initial position for certification: 126389, protocol version: -1
120308 0:19:34 [Note] WSREP: Setting initial position to 00d090b7-66c3-11e1-0800-23e68969619b:126389
120308 0:19:34 [Note] WSREP: protonet asio version 0
120308 0:19:34 [Note] WSREP: backend: asio
120308 0:19:34 [Note] WSREP: GMCast version 0
120308 0:19:34 [Note] WSREP: (6230951e-68b4-11e1-0800-0915ace4902d, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
120308 0:19:34 [Note] WSREP: (6230951e-68b4-11e1-0800-0915ace4902d, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
120308 0:19:34 [Note] WSREP: EVS version 0
120308 0:19:34 [Note] WSREP: PC version 0
120308 0:19:34 [Note] WSREP: gcomm: connecting to group 'my_wsrep_cluster', peer ''
120308 0:19:34 [Note] WSREP: GMCast::handle_stable_view: view(view_id(PRIM,6230951e-68b4-11e1-0800-0915ace4902d,1) memb {
 6230951e-68b4-11e1-0800-0915ace4902d,
} joined {
} left {
} partitioned {
})
120308 0:19:34 [Note] WSREP: gcomm: connected
120308 0:19:34 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
120308 0:19:34 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
120308 0:19:34 [Note] WSREP: Opened channel 'my_wsrep_cluster'
120308 0:19:34 [Note] WSREP: New COMPONENT: primary = yes, my_idx = 0, memb_num = 1
120308 0:19:34 [Note] /opt/galera/mysql/sbin/mysqld: ready for connections.
Version: '5.5.20-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution, wsrep_23.4.r3713
120308 0:19:34 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 62311b54-68b4-11e1-0800-b3b65a87cee4
120308 0:19:34 [Note] WSREP: STATE EXCHANGE: sent state msg: 62311b54-68b4-11e1-0800-b3b65a87cee4
120308 0:19:34 [Note] WSREP: STATE EXCHANGE: got state msg: 62311b54-68b4-11e1-0800-b3b65a87cee4 from 0 (webserver.unlocksamsungonline.com)
120308 0:19:34 [Note] WSREP: Quorum results:
 version = 2,
 component = PRIMARY,
 conf_id = 0,
 members = 1/1 (joined/total),
 act_id = 126389,
 last_appl. = -1,
 protocols = 0/3/1 (gcs/repl/appl),
 group UUID = 00d090b7-66c3-11e1-0800-23e68969619b
120308 0:19:34 [Note] WSREP: Flow-control interval: [8, 16]
120308 0:19:34 [Note] WSREP: Restored state OPEN -> JOINED (126389)
120308 0:19:34 [Note] WSREP: Member 0 (webserver.unlocksamsungonline.com) synced with group.
120308 0:19:34 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 126389)
120308 0:19:34 [Note] WSREP: New cluster view: global state: 00d090b7-66c3-11e1-0800-23e68969619b:126389, view# 1: Primary, number of nodes: 1, my index: 0, protocol version 1
120308 0:19:34 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
120308 0:19:34 [Note] WSREP: Assign initial position for certification: 126389, protocol version: 2
120308 0:19:34 [Note] WSREP: Synchronized with group, ready for connections
120308 0:19:34 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
120308 0:24:36 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:24:36 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:24:37 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:24:37 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:24:38 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:24:38 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:24:39 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:24:39 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:24:40 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:24:40 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:24:41 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:24:41 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:24:42 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:24:43 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:24:43 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:24:44 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:24:44 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:24:45 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:24:45 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:24:46 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:24:46 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:24:47 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:24:47 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:24:48 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:24:48 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:24:49 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:24:49 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:24:50 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:24:50 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:24:51 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:24:51 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:24:52 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:24:52 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:24:53 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:24:53 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:24:54 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:24:54 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:24:55 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:24:55 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:24:56 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:24:56 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:24:57 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:24:57 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:24:58 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:24:58 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:24:59 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:24:59 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:25:00 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:25:00 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:25:01 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:25:01 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:25:02 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:25:02 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:25:03 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:25:03 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:25:04 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:25:04 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:25:05 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:25:05 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:25:06 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:25:06 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:25:07 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:25:07 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:25:08 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:25:08 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:25:09 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:25:09 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:25:10 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:25:10 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:25:11 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:25:11 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:25:12 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:25:12 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:25:13 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:25:35 [Warning] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)) install timer expired
evs::proto(evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1)), GATHER) {
current_view=view(view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1) memb {
 6230951e-68b4-11e1-0800-0915ace4902d,
} joined {
} left {
} partitioned {
}),
input_map=evs::input_map: {aru_seq=5,safe_seq=5,node_index=node: {idx=0,range=[6,5],safe_seq=5} ,msg_index= (0,5),evs::msg{version=0,type=1,user_type=255,order=0,seq=5,seq_range=0,aru_seq=4,flags=0,source=6230951e-68b4-11e1-0800-0915ace4902d,source_view_id=view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1),range_uuid=00000000-0000-0000-0000-000000000000,range=[-1,-1],fifo_seq=10,node_list=()
}
,recovery_index=},
fifo_seq=1203,
last_sent=5,
known={
 25a2f3ea-66c9-11e1-0800-aaf7ad456b15,evs::node{operational=1,suspected=0,installed=0,fifo_seq=492609,join_message=
evs::msg{version=0,type=4,user_type=255,order=1,seq=0,seq_range=-1,aru_seq=0,flags=4,source=25a2f3ea-66c9-11e1-0800-aaf7ad456b15,source_view_id=view_id(REG,25a2f3ea-66c9-11e1-0800-aaf7ad456b15,15),range_uuid=00000000-0000-0000-0000-000000000000,range=[-1,-1],fifo_seq=492609,node_list=( 25a2f3ea-66c9-11e1-0800-aaf7ad456b15,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,25a2f3ea-66c9-11e1-0800-aaf7ad456b15,15),safe_seq=0,im_range=[1,0],}
 52321e04-6770-11e1-0800-39f9cf850209,node: {operational=0,suspected=1,leave_seq=0,view_id=view_id(REG,52321e04-6770-11e1-0800-39f9cf850209,15),safe_seq=-1,im_range=[-1,-1],}
 6230951e-68b4-11e1-0800-0915ace4902d,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1),safe_seq=5,im_range=[6,5],}
 b3157d8d-6771-11e1-0800-4c20bc75ca8d,node: {operational=0,suspected=1,leave_seq=-1,view_id=view_id(REG,00000000-0000-0000-0000-000000000000,0),safe_seq=-1,im_range=[-1,-1],}
)
},
}
 52321e04-6770-11e1-0800-39f9cf850209,evs::node{operational=0,suspected=1,installed=0,fifo_seq=-1,}
 6230951e-68b4-11e1-0800-0915ace4902d,evs::node{operational=1,suspected=0,installed=0,fifo_seq=-1,join_message=
evs::msg{version=0,type=4,user_type=255,order=1,seq=5,seq_range=-1,aru_seq=5,flags=0,source=6230951e-68b4-11e1-0800-0915ace4902d,source_view_id=view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1),range_uuid=00000000-0000-0000-0000-000000000000,range=[-1,-1],fifo_seq=1203,node_list=( 25a2f3ea-66c9-11e1-0800-aaf7ad456b15,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,25a2f3ea-66c9-11e1-0800-aaf7ad456b15,15),safe_seq=0,im_range=[1,0],}
 52321e04-6770-11e1-0800-39f9cf850209,node: {operational=0,suspected=1,leave_seq=-1,view_id=view_id(REG,00000000-0000-0000-0000-000000000000,0),safe_seq=-1,im_range=[-1,-1],}
 6230951e-68b4-11e1-0800-0915ace4902d,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1),safe_seq=5,im_range=[6,5],}
 b3157d8d-6771-11e1-0800-4c20bc75ca8d,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,b3157d8d-6771-11e1-0800-4c20bc75ca8d,16),safe_seq=0,im_range=[1,0],}
)
},
}
 b3157d8d-6771-11e1-0800-4c20bc75ca8d,evs::node{operational=1,suspected=0,installed=0,fifo_seq=327904,join_message=
evs::msg{version=0,type=4,user_type=255,order=1,seq=0,seq_range=-1,aru_seq=0,flags=4,source=b3157d8d-6771-11e1-0800-4c20bc75ca8d,source_view_id=view_id(REG,b3157d8d-6771-11e1-0800-4c20bc75ca8d,16),range_uuid=00000000-0000-0000-0000-000000000000,range=[-1,-1],fifo_seq=327904,node_list=( 25a2f3ea-66c9-11e1-0800-aaf7ad456b15,node: {operational=0,suspected=1,leave_seq=-1,view_id=view_id(REG,00000000-0000-0000-0000-000000000000,0),safe_seq=-1,im_range=[-1,-1],}
 52321e04-6770-11e1-0800-39f9cf850209,node: {operational=0,suspected=1,leave_seq=0,view_id=view_id(REG,52321e04-6770-11e1-0800-39f9cf850209,15),safe_seq=-1,im_range=[-1,-1],}
 6230951e-68b4-11e1-0800-0915ace4902d,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1),safe_seq=5,im_range=[6,5],}
 b3157d8d-6771-11e1-0800-4c20bc75ca8d,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,b3157d8d-6771-11e1-0800-4c20bc75ca8d,16),safe_seq=0,im_range=[1,0],}
)
},
}
 }
 }
120308 0:25:35 [Note] WSREP: no install message received
120308 0:25:35 [Warning] WSREP: subsequent views have same members, prev view view(view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,1) memb {
 6230951e-68b4-11e1-0800-0915ace4902d,
} joined {
} left {
} partitioned {
}) current view view(view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,2) memb {
 6230951e-68b4-11e1-0800-0915ace4902d,
} joined {
} left {
} partitioned {
})
120308 0:25:35 [Note] WSREP: GMCast::handle_stable_view: view(view_id(PRIM,6230951e-68b4-11e1-0800-0915ace4902d,2) memb {
 6230951e-68b4-11e1-0800-0915ace4902d,
} joined {
} left {
} partitioned {
})
120308 0:25:35 [Note] WSREP: forgetting 25a2f3ea-66c9-11e1-0800-aaf7ad456b15 (tcp://83.170.88.228:4567)
120308 0:25:35 [Note] WSREP: New COMPONENT: primary = yes, my_idx = 0, memb_num = 1
120308 0:25:35 [Note] WSREP: forgetting b3157d8d-6771-11e1-0800-4c20bc75ca8d (tcp://46.23.72.59:4567)
120308 0:25:35 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 39724b32-68b5-11e1-0800-1bf94159c4cc
120308 0:25:35 [Note] WSREP: STATE EXCHANGE: sent state msg: 39724b32-68b5-11e1-0800-1bf94159c4cc
120308 0:25:35 [Note] WSREP: STATE EXCHANGE: got state msg: 39724b32-68b5-11e1-0800-1bf94159c4cc from 0 (webserver.unlocksamsungonline.com)
120308 0:25:35 [Note] WSREP: Quorum results:
 version = 2,
 component = PRIMARY,
 conf_id = 1,
 members = 1/1 (joined/total),
 act_id = 126389,
 last_appl. = 0,
 protocols = 0/3/1 (gcs/repl/appl),
 group UUID = 00d090b7-66c3-11e1-0800-23e68969619b
120308 0:25:35 [Note] WSREP: Flow-control interval: [8, 16]
120308 0:25:35 [Note] WSREP: New cluster view: global state: 00d090b7-66c3-11e1-0800-23e68969619b:126389, view# 2: Primary, number of nodes: 1, my index: 0, protocol version 1
120308 0:25:35 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
120308 0:25:35 [Note] WSREP: Assign initial position for certification: 126389, protocol version: 2
120308 0:25:37 [Warning] WSREP: discarding established (time wait) 25a2f3ea-66c9-11e1-0800-aaf7ad456b15 (tcp://83.170.88.228:4567)
120308 0:25:37 [Warning] WSREP: discarding established (time wait) b3157d8d-6771-11e1-0800-4c20bc75ca8d (tcp://46.23.72.59:4567)
120308 0:25:39 [Warning] WSREP: discarding established (time wait) 25a2f3ea-66c9-11e1-0800-aaf7ad456b15 (tcp://83.170.88.228:4567)
120308 0:25:39 [Warning] WSREP: discarding established (time wait) b3157d8d-6771-11e1-0800-4c20bc75ca8d (tcp://46.23.72.59:4567)
120308 0:25:40 [Warning] WSREP: discarding established (time wait) 25a2f3ea-66c9-11e1-0800-aaf7ad456b15 (tcp://83.170.88.228:4567)
120308 0:25:40 [Warning] WSREP: discarding established (time wait) b3157d8d-6771-11e1-0800-4c20bc75ca8d (tcp://46.23.72.59:4567)
120308 0:25:41 [Note] WSREP: cleaning up b3157d8d-6771-11e1-0800-4c20bc75ca8d (tcp://46.23.72.59:4567)
120308 0:25:41 [Note] WSREP: cleaning up 25a2f3ea-66c9-11e1-0800-aaf7ad456b15 (tcp://83.170.88.228:4567)
120308 0:30:42 [Note] WSREP: evs::proto(6230951e-68b4-11e1-0800-0915ace4902d, GATHER, view_id(REG,6230951e-68b4-11e1-0800-0915ace4902d,2)) suspecting node: 52321e04-6770-11e1-0800-39f9cf850209
120308 0:30:42 [Note] WSREP: node 52321e04-6770-11e1-0800-39f9cf850209 marked with nil view id and suspected in all present join messages, declaring inactive

After this the line
120308 0:30:42 [Note] WSREP: node 52321e04-6770-11e1-0800-39f9cf850209 marked with nil view id and suspected in all present join messages, declaring inactive
repeats many many times.

Could anybody shed any light on this please. Thanks in advance.

Question information

Language:
English Edit question
Status:
Expired
For:
Galera Edit question
Assignee:
No assignee Edit question
Last query:
2012-03-17
Last reply:
2012-04-02

This question was reopened

Teemu Ollakka (teemu-ollakka) said : #1

Hi,

This looks like issue in network connectivity. First symptoms appear at 120306 14:38:48, but message relaying seems to save the group here. Second incident starts at 120307 23:53:54, one of the nodes partition from group and tries to come back after it, but for some reason there seem to be lack of connectivity between two of the nodes, it could be that message relaying does not work as expected in this case. However, I'd need to see logs from all of the nodes to make further conclusions about what exactly goes wrong.

This particular server was restarted at 120308 0:19:30, but due to connectivity issue and long evs timeouts in configuration the other servers hadn't formed new group which prevented restarted server to start operating properly. Log message

120308 0:30:42 [Note] WSREP: node 52321e04-6770-11e1-0800-39f9cf850209 marked with nil view id and suspected in all present join messages, declaring inactive

is clear indication of this. Restarting all servers at once could have ended in better outcome, although it was not completely clear from this log if the connectivity issue had been resolved.

Steven (steven-7tv) said : #2

Hi Teemu,

Thank you for your reply.
According to the vps.net support (who are helping me out with this also) the db on the US server (from which the first posted log comes) is completely flushed due a yet unknown reason..

The log from the server mentioned at 120306 14:38:48 is:
120306 9:49:43 [Note] WSREP: Flow-control interval: [14, 28]
120306 9:49:43 [Note] WSREP: New cluster view: global state: 00d090b7-66c3-11e1-0800-23e68969619b:122788, view# 7: Primary, number of nodes: 3, my index: 0, protocol version 1
120306 9:49:43 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
120306 9:49:43 [Note] WSREP: Assign initial position for certification: 122788, protocol version: 2
120306 9:49:44 [Note] WSREP: Member 2 (mysql) synced with group.
120306 11:14:43 [Warning] IP address '188.245.208.128' could not be resolved: Name or service not known
120306 11:42:31 [Warning] IP address '46.100.122.248' could not be resolved: Name or service not known
120306 12:00:52 [Warning] IP address '2.146.58.80' could not be resolved: Name or service not known
120306 13:10:26 [Warning] IP address '82.115.13.250' could not be resolved: Name or service not known
120306 14:38:46 [Note] WSREP: (25a2f3ea-66c9-11e1-0800-aaf7ad456b15, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://68.169.42.153:4567
120306 14:38:47 [Note] WSREP: (25a2f3ea-66c9-11e1-0800-aaf7ad456b15, 'tcp://0.0.0.0:4567') reconnecting to 52321e04-6770-11e1-0800-39f9cf850209 (tcp://68.169.42.153:4567), attempt 0
120306 14:38:48 [Note] WSREP: (25a2f3ea-66c9-11e1-0800-aaf7ad456b15, 'tcp://0.0.0.0:4567') turning message relay requesting off
120306 16:35:11 [Warning] IP address '2.146.169.35' could not be resolved: Name or service not known
120306 17:52:54 [Warning] IP address '109.162.158.66' could not be resolved: Name or service not known
120306 18:48:07 [Warning] IP address '109.162.187.252' could not be resolved: Name or service not known
120306 19:04:05 [Warning] IP address '94.74.181.184' could not be resolved: Name or service not known
120306 19:54:29 [Warning] IP address '109.201.16.45' could not be resolved: Name or service not known
120306 21:30:55 [Warning] IP address '2.183.79.8' could not be resolved: Name or service not known
120306 21:39:17 [Warning] IP address '2.184.244.137' could not be resolved: Name or service not known
120306 21:55:17 [Warning] IP address '2.179.242.181' could not be resolved: Name or service not known
120306 22:02:04 [Warning] IP address '2.146.139.84' could not be resolved: Name or service not known
120306 22:50:45 [Warning] IP address '178.157.58.4' could not be resolved: Name or service not known
120307 5:01:51 [Note] WSREP: (25a2f3ea-66c9-11e1-0800-aaf7ad456b15, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://46.23.72.59:4567
120307 5:01:52 [Note] WSREP: (25a2f3ea-66c9-11e1-0800-aaf7ad456b15, 'tcp://0.0.0.0:4567') reconnecting to b3157d8d-6771-11e1-0800-4c20bc75ca8d (tcp://46.23.72.59:4567), attempt 0
120307 5:01:52 [Note] WSREP: (25a2f3ea-66c9-11e1-0800-aaf7ad456b15, 'tcp://0.0.0.0:4567') cleaning up established 0x7fe950070560 which is duplicate of 0x7fe94809f600
120307 5:01:52 [Note] WSREP: (25a2f3ea-66c9-11e1-0800-aaf7ad456b15, 'tcp://0.0.0.0:4567') turning message relay requesting off
120307 5:02:17 [Note] WSREP: (25a2f3ea-66c9-11e1-0800-aaf7ad456b15, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://46.23.72.59:4567
120307 5:02:19 [Note] WSREP: (25a2f3ea-66c9-11e1-0800-aaf7ad456b15, 'tcp://0.0.0.0:4567') turning message relay requesting off
120307 5:43:13 [Warning] IP address '80.191.117.106' could not be resolved: Name or service not known
120307 7:57:54 [Warning] IP address '217.218.226.220' could not be resolved: Name or service not known
120307 8:48:20 [Warning] IP address '109.238.191.50' could not be resolved: Name or service not known
120307 9:39:40 [Warning] IP address '46.100.173.181' could not be resolved: Name or service not known
120307 10:20:58 [Warning] IP address '217.218.226.224' could not be resolved: Name or service not known
120307 11:07:09 [Warning] IP address '94.74.181.246' could not be resolved: Name or service not known
120307 12:04:33 [Warning] IP address '2.178.131.76' could not be resolved: Name or service not known
120307 12:24:58 [Warning] IP address '217.218.226.182' could not be resolved: Name or service not known
120307 15:27:28 [Warning] IP address '78.39.28.17' could not be resolved: Name or service not known
120307 17:11:10 [Warning] IP address '188.159.34.94' could not be resolved: Name or service not known
120307 19:40:58 [Warning] IP address '2.187.46.147' could not be resolved: Name or service not known
120307 23:42:51 [Note] WSREP: (25a2f3ea-66c9-11e1-0800-aaf7ad456b15, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://46.23.72.59:4567
120307 23:44:45 [Note] WSREP: (25a2f3ea-66c9-11e1-0800-aaf7ad456b15, 'tcp://0.0.0.0:4567') reconnecting to b3157d8d-6771-11e1-0800-4c20bc75ca8d (tcp://46.23.72.59:4567), attempt 43170
120307 23:46:45 [Note] WSREP: (25a2f3ea-66c9-11e1-0800-aaf7ad456b15, 'tcp://0.0.0.0:4567') reconnecting to b3157d8d-6771-11e1-0800-4c20bc75ca8d (tcp://46.23.72.59:4567), attempt 43200
120307 23:46:48 [Note] WSREP: (25a2f3ea-66c9-11e1-0800-aaf7ad456b15, 'tcp://0.0.0.0:4567') turning message relay requesting off
120307 23:46:49 [Note] WSREP: cleaning up b3157d8d-6771-11e1-0800-4c20bc75ca8d (tcp://46.23.72.59:4567)
120307 23:51:52 [Note] WSREP: evs::proto(25a2f3ea-66c9-11e1-0800-aaf7ad456b15, GATHER, view_id(REG,25a2f3ea-66c9-11e1-0800-aaf7ad456b15,11)) suspecting node: b3157d8d-6771-11e1-0800-4c20bc75ca8d
last line repeated many times

120307 23:53:49 [Note] WSREP: GMCast::handle_stable_view: view(view_id(PRIM,25a2f3ea-66c9-11e1-0800-aaf7ad456b15,12) memb {
 25a2f3ea-66c9-11e1-0800-aaf7ad456b15,
 52321e04-6770-11e1-0800-39f9cf850209,
} joined {
} left {
} partitioned {
 b3157d8d-6771-11e1-0800-4c20bc75ca8d,
})
120307 23:53:49 [Note] WSREP: declaring 52321e04-6770-11e1-0800-39f9cf850209 stable
120307 23:53:49 [Note] WSREP: New COMPONENT: primary = yes, my_idx = 0, memb_num = 2
120307 23:53:49 [Warning] WSREP: Failed to send state UUID: -11 (Resource temporarily unavailable)
120307 23:58:01 [Note] WSREP: cleaning up pending addr tcp://46.23.72.59:4567
120307 23:58:49 [Note] WSREP: evs::proto(25a2f3ea-66c9-11e1-0800-aaf7ad456b15, GATHER, view_id(REG,25a2f3ea-66c9-11e1-0800-aaf7ad456b15,12)) suspecting node: b3157d8d-6771-11e1-0800-4c20bc75ca8d
last line many times

120307 23:59:49 [Note] WSREP: node b3157d8d-6771-11e1-0800-4c20bc75ca8d marked with nil view id and suspected in all present join messages, declaring inactive
120307 23:59:49 [Note] WSREP: GMCast::handle_stable_view: view(view_id(NON_PRIM,25a2f3ea-66c9-11e1-0800-aaf7ad456b15,12) memb {
 25a2f3ea-66c9-11e1-0800-aaf7ad456b15,
} joined {
} left {
} partitioned {
 52321e04-6770-11e1-0800-39f9cf850209,
})
120307 23:59:49 [Note] WSREP: New COMPONENT: primary = no, my_idx = 0, memb_num = 1
120307 23:59:49 [Note] WSREP: Flow-control interval: [8, 16]
120307 23:59:49 [Note] WSREP: Received NON-PRIMARY.
120307 23:59:49 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 126389)
120307 23:59:49 [Note] WSREP: New cluster view: global state: 00d090b7-66c3-11e1-0800-23e68969619b:126389, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 1
120307 23:59:49 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
120307 23:59:49 [Warning] WSREP: Send action {0x4d43eb0, 214, TORDERED} returned -107 (Transport endpoint is not connected)
120307 23:59:49 [Warning] WSREP: Send action {0x4f5c610, 2825, TORDERED} returned -107 (Transport endpoint is not connected)
120307 23:59:49 [Note] WSREP: GMCast::handle_stable_view: view(view_id(NON_PRIM,25a2f3ea-66c9-11e1-0800-aaf7ad456b15,13) memb {
 25a2f3ea-66c9-11e1-0800-aaf7ad456b15,
} joined {
} left {
} partitioned {
 52321e04-6770-11e1-0800-39f9cf850209,
})
120307 23:59:49 [Note] WSREP: New COMPONENT: primary = no, my_idx = 0, memb_num = 1
120307 23:59:49 [Note] WSREP: Flow-control interval: [8, 16]
120307 23:59:49 [Note] WSREP: Received NON-PRIMARY.
120307 23:59:49 [Note] WSREP: New cluster view: global state: 00d090b7-66c3-11e1-0800-23e68969619b:126389, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 1
120307 23:59:49 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
120308 0:04:49 [Note] WSREP: evs::proto(25a2f3ea-66c9-11e1-0800-aaf7ad456b15, GATHER, view_id(REG,25a2f3ea-66c9-11e1-0800-aaf7ad456b15,13)) suspecting node: b3157d8d-6771-11e1-0800-4c20bc75ca8d
120308 0:04:50 [Note] W
etc

What worries me is these messages:
120306 21:39:17 [Warning] IP address '2.184.244.137' could not be resolved: Name or service not known

None of those IP addresses I see there are known to me! Or should I see this as a lookup-route which is failing for ip's of my servers (there are 3)?

Thank you

Alex Yurchenko (ayurchen) said : #3

Hi Steven,

120306 21:39:17 [Warning] IP address '2.184.244.137' could not be resolved: Name or service not known

is not Galera-related message. If I'm correct it is a client address/hostname and mysqld is trying to resolve it for authentication purposes. It looks like someone from unauthorized (and probably fake) IPs is trying to break in to your servers. You may want to set skip_name_resolve option to avoid this (but then you'll have to use IP addresses in client hostnames).

This may or may not be related to the overall connectivity problems that Galera experiences. BTW, if you're running a WAN cluster, you may want to increase Galera timeouts which will make the cluster much more stable: http://www.codership.com/wiki/doku.php?id=configuration_tips#wan_replication

Regards,
Alex

Alex Yurchenko (ayurchen) said : #4

Steven,
Sorry for possible confusion, I realized that you have already increased timeouts and the problem must be elsewhere.
Alex

Steven (steven-7tv) said : #5

Hi Alex,

Yeah I checked and I've got it set to:
wsrep_provider_options="evs.suspect_timeout=PT5M; evs.inactive_timeout=PT6M; evs.consensus_timeout=PT7M"

In the example
wsrep_provider_options = "evs.keepalive_period = PT3S; evs.inactive_check_period = PT10S; evs.suspect_timeout = PT30S; evs.inactive_timeout = PT1M; evs.consensus_timeout = PT1M"

There are some other options set. Could that cause problems?

Also I modified my firewalls to only allow my ip's to connect to galera port to avoid those messages that worries me.

Teemu Ollakka (teemu-ollakka) said : #6

Steven,

It looks now clear that connectivity issues caused a bug in message relaying system to surface. This will be fixed in the next galera release.

Otherwise your configuration seems ok to me.

If this kind of situation appears again, it would be probably best to identify servers which don't have direct working network connection between them and take either one temporarily down until connectivity is restored.

Steven (steven-7tv) said : #7

Hi,

Well it happened again and the result, the complete database on US is flushed again! This is pretty serious, actually this is that serious I need to consider downgrading again :(

Do you have any time span an update will be released to fix this problem?

Steven (steven-7tv) said : #8

site went down at aprox 13 22:15 Gmt but seems one node (control node already left before that)
Server log US:
120308 8:47:53 [Note] WSREP: New cluster view: global state: 00d090b7-66c3-11e1-0800-23e68969619b:126389, view# 9: Primary, number of nodes: 3, my index: 1, protocol version 1
120308 8:47:53 [Warning] WSREP: Gap in state sequence. Need state transfer.
120308 8:47:55 [Note] WSREP: Prepared SST request: mysqldump|68.169.42.153:3306
120308 8:47:55 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
120308 8:47:55 [Note] WSREP: Assign initial position for certification: 126389, protocol version: 2
120308 8:47:55 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (df46403d-68f6-11e1-0800-53b8c07168ed) does not match group state UUID (00d090b7-66c3-11e1-0800-23e68969619b): 1 (Operation not permitted)
  at galera/src/replicator_str.cpp:prepare_for_IST():429. IST will be unavailable.
120308 8:47:55 [Note] WSREP: Node 1 (webserver.unlocksamsungonline.com) requested state transfer from '*any*'. Selected 0 (webserver.unlocksamsungonline.eu)(SYNCED) as donor.
120308 8:47:55 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 126389)
120308 8:47:55 [Note] WSREP: Requesting state transfer: success, donor: 0
120308 8:48:00 [ERROR] Incorrect definition of table mysql.proc: expected column 'comment' at position 15 to have type text, found type char(64).
120308 8:49:39 [Note] WSREP: wsrep_start_position var submitted: '00d090b7-66c3-11e1-0800-23e68969619b:126389'
120308 8:49:39 [Note] WSREP: Received SST: 00d090b7-66c3-11e1-0800-23e68969619b:126389
120308 8:49:39 [Note] WSREP: SST received: 00d090b7-66c3-11e1-0800-23e68969619b:126389
120308 8:49:39 [Note] WSREP: 1 (webserver.unlocksamsungonline.com): State transfer from 0 (webserver.unlocksamsungonline.eu) complete.
120308 8:49:39 [Note] WSREP: Shifting JOINER -> JOINED (TO: 126389)
120308 8:49:39 [Note] WSREP: 0 (webserver.unlocksamsungonline.eu): State transfer to 1 (webserver.unlocksamsungonline.com) complete.
120308 8:49:39 [Note] WSREP: Member 1 (webserver.unlocksamsungonline.com) synced with group.
120308 8:49:39 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 126389)
120308 8:49:39 [Note] WSREP: Synchronized with group, ready for connections
120308 8:49:39 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
120308 8:49:39 [Note] WSREP: Member 0 (webserver.unlocksamsungonline.eu) synced with group.
120308 13:10:08 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://46.23.72.59:4567
120308 13:10:09 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') reconnecting to 732cfaf8-68f8-11e1-0800-db892ab4dca5 (tcp://46.23.72.59:4567), attempt 0
120308 13:10:18 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') turning message relay requesting off
120308 13:10:18 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') cleaning up established 0x7f20b002f1f0 which is duplicate of 0x7f20b002eb40
120308 21:20:47 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://83.170.88.228:4567
120308 21:20:48 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') reconnecting to 5072c8cb-68fb-11e1-0800-de84c5727c34 (tcp://83.170.88.228:4567), attempt 0
120308 21:20:49 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') turning message relay requesting off
120308 22:56:51 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://46.23.72.59:4567
120308 22:56:52 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') cleaning up duplicate 0x7f20c000dff0 after established 0x7f20b0056700
120308 22:56:52 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') turning message relay requesting off
120308 23:14:31 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://46.23.72.59:4567
120308 23:14:35 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') turning message relay requesting off
120308 23:14:45 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://46.23.72.59:4567
120308 23:14:50 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') turning message relay requesting off
120308 23:14:50 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') cleaning up established 0x7f20c00008b0 which is duplicate of 0x7f20c0044db0
120309 2:26:34 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://46.23.72.59:4567
120309 2:26:35 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') turning message relay requesting off
120309 2:59:01 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://46.23.72.59:4567
120309 2:59:02 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') turning message relay requesting off
120309 9:07:39 [Warning] IP address '125.211.210.147' could not be resolved: Name or service not known
120310 9:38:11 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://46.23.72.59:4567
120310 9:38:12 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') cleaning up duplicate 0x7f20c005adb0 after established 0x7f20b003a7d0
120310 9:38:12 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') turning message relay requesting off
120310 9:46:43 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://83.170.88.228:4567
120310 9:46:45 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') cleaning up duplicate 0x7f20c005adb0 after established 0x7f20c0054650
120310 9:46:45 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') turning message relay requesting off
120311 2:14:49 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://46.23.72.59:4567
120311 2:16:47 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') reconnecting to 732cfaf8-68f8-11e1-0800-db892ab4dca5 (tcp://46.23.72.59:4567), attempt 43170
120311 2:18:48 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') reconnecting to 732cfaf8-68f8-11e1-0800-db892ab4dca5 (tcp://46.23.72.59:4567), attempt 43200
120311 2:18:51 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') turning message relay requesting off
120311 2:18:52 [Note] WSREP: cleaning up 732cfaf8-68f8-11e1-0800-db892ab4dca5 (tcp://46.23.72.59:4567)
120311 2:19:46 [Note] WSREP: evs::proto(6257e849-68fb-11e1-0800-628c3a9f1cba, GATHER, view_id(REG,5072c8cb-68fb-11e1-0800-de84c5727c34,10)) suspecting node: 732cfaf8-68f8-11e1-0800-db892ab4dca5
120311 2:19:47 [Note] WSREP: GMCast::handle_stable_view: view(view_id(PRIM,5072c8cb-68fb-11e1-0800-de84c5727c34,12) memb {
 5072c8cb-68fb-11e1-0800-de84c5727c34,
 6257e849-68fb-11e1-0800-628c3a9f1cba,
} joined {
} left {
} partitioned {
 732cfaf8-68f8-11e1-0800-db892ab4dca5,
})
120311 2:19:47 [Note] WSREP: declaring 5072c8cb-68fb-11e1-0800-de84c5727c34 stable
120311 2:19:47 [Note] WSREP: New COMPONENT: primary = yes, my_idx = 1, memb_num = 2
120311 2:19:47 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
120311 2:19:48 [Note] WSREP: STATE EXCHANGE: sent state msg: a73f9a67-6b20-11e1-0800-021b7fc7c983
120311 2:19:48 [Note] WSREP: STATE EXCHANGE: got state msg: a73f9a67-6b20-11e1-0800-021b7fc7c983 from 0 (webserver.unlocksamsungonline.eu)
120311 2:19:48 [Note] WSREP: STATE EXCHANGE: got state msg: a73f9a67-6b20-11e1-0800-021b7fc7c983 from 1 (webserver.unlocksamsungonline.com)
120311 2:19:48 [Note] WSREP: Quorum results:
 version = 2,
 component = PRIMARY,
 conf_id = 9,
 members = 2/2 (joined/total),
 act_id = 131399,
 last_appl. = 130444,
 protocols = 0/3/1 (gcs/repl/appl),
 group UUID = 00d090b7-66c3-11e1-0800-23e68969619b
120311 2:19:48 [Note] WSREP: Flow-control interval: [12, 23]
120311 2:19:48 [Note] WSREP: New cluster view: global state: 00d090b7-66c3-11e1-0800-23e68969619b:131399, view# 10: Primary, number of nodes: 2, my index: 1, protocol version 1
120311 2:19:48 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
120311 2:19:48 [Note] WSREP: Assign initial position for certification: 131399, protocol version: 2
120311 10:24:10 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://83.170.88.228:4567
120311 10:24:10 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') reconnecting to 5072c8cb-68fb-11e1-0800-de84c5727c34 (tcp://83.170.88.228:4567), attempt 0
120311 10:24:11 [Warning] WSREP: last inactive check more than PT1.5S ago, skipping check
120311 10:24:11 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') turning message relay requesting off
120311 10:24:11 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') cleaning up established 0x7f20b007fe10 which is duplicate of 0x7f20c00008b0
120311 22:16:58 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://83.170.88.228:4567
120311 22:16:59 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') turning message relay requesting off
120312 11:20:29 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://83.170.88.228:4567
120312 11:20:30 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') turning message relay requesting off
120312 11:20:31 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') cleaning up established 0x7f20b0043010 which is duplicate of 0x7f20b010d010
120312 16:33:30 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://83.170.88.228:4567
120312 16:33:34 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') turning message relay requesting off
120312 16:33:35 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') cleaning up duplicate 0x7f20c003d3a0 after established 0x7f20b00fa9d0
120313 10:50:08 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://83.170.88.228:4567
120313 10:50:09 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') turning message relay requesting off
120313 10:50:09 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') cleaning up established 0x7f20c003d3a0 which is duplicate of 0x7f20c00008b0
120313 22:04:44 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://83.170.88.228:4567
120313 22:06:42 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') reconnecting to 5072c8cb-68fb-11e1-0800-de84c5727c34 (tcp://83.170.88.228:4567), attempt 43170
120313 22:08:43 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') reconnecting to 5072c8cb-68fb-11e1-0800-de84c5727c34 (tcp://83.170.88.228:4567), attempt 43200
120313 22:08:46 [Note] WSREP: (6257e849-68fb-11e1-0800-628c3a9f1cba, 'tcp://0.0.0.0:4567') turning message relay requesting off
120313 22:08:47 [Note] WSREP: cleaning up 5072c8cb-68fb-11e1-0800-de84c5727c34 (tcp://83.170.88.228:4567)
120313 22:09:41 [Note] WSREP: evs::proto(6257e849-68fb-11e1-0800-628c3a9f1cba, OPERATIONAL, view_id(REG,5072c8cb-68fb-11e1-0800-de84c5727c34,12)) suspecting node: 5072c8cb-68fb-11e1-0800-de84c5727c34
120313 22:09:42 [Note] WSREP: evs::proto(6257e849-68fb-11e1-0800-628c3a9f1cba, GATHER, view_id(REG,5072c8cb-68fb-11e1-0800-de84c5727c34,12)) suspecting node: 5072c8cb-68fb-11e1-0800-de84c5727c34
-repeating above line--
120313 22:10:42 [Note] WSREP: GMCast::handle_stable_view: view(view_id(NON_PRIM,5072c8cb-68fb-11e1-0800-de84c5727c34,12) memb {
 6257e849-68fb-11e1-0800-628c3a9f1cba,
} joined {
} left {
} partitioned {
 5072c8cb-68fb-11e1-0800-de84c5727c34,
})
120313 22:10:42 [Note] WSREP: GMCast::handle_stable_view: view(view_id(NON_PRIM,6257e849-68fb-11e1-0800-628c3a9f1cba,13) memb {
 6257e849-68fb-11e1-0800-628c3a9f1cba,
} joined {
} left {
} partitioned {
 5072c8cb-68fb-11e1-0800-de84c5727c34,
})
120313 22:10:42 [Note] WSREP: New COMPONENT: primary = no, my_idx = 0, memb_num = 1
120313 22:10:42 [Note] WSREP: Flow-control interval: [8, 16]
120313 22:10:42 [Note] WSREP: Received NON-PRIMARY.
120313 22:10:42 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 135875)
120313 22:10:42 [Note] WSREP: New COMPONENT: primary = no, my_idx = 0, memb_num = 1
120313 22:10:42 [Note] WSREP: Flow-control interval: [8, 16]
120313 22:10:42 [Note] WSREP: Received NON-PRIMARY.
120313 22:10:42 [Warning] WSREP: Send action {0x4baca20, 219, TORDERED} returned -107 (Transport endpoint is not connected)
120313 22:10:42 [Note] WSREP: New cluster view: global state: 00d090b7-66c3-11e1-0800-23e68969619b:135875, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 1
120313 22:10:42 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
120313 22:10:42 [Note] WSREP: New cluster view: global state: 00d090b7-66c3-11e1-0800-23e68969619b:135875, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 1
120313 22:10:42 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

UK server log:
120308 8:47:47 [Note] WSREP: New cluster view: global state: 00d090b7-66c3-11e1-0800-23e68969619b:126389, view# 9: Primary, number of nodes: 3, my index: 0, protocol version 1
120308 8:47:47 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
120308 8:47:47 [Note] WSREP: Assign initial position for certification: 126389, protocol version: 2
120308 8:47:50 [Note] WSREP: Node 1 (webserver.unlocksamsungonline.com) requested state transfer from '*any*'. Selected 0 (webserver.unlocksamsungonline.eu)(SYNCED) as donor.
120308 8:47:50 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 126389)
120308 8:47:50 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
120308 8:49:34 [Note] WSREP: 1 (webserver.unlocksamsungonline.com): State transfer from 0 (webserver.unlocksamsungonline.eu) complete.
120308 8:49:34 [Note] WSREP: 0 (webserver.unlocksamsungonline.eu): State transfer to 1 (webserver.unlocksamsungonline.com) complete.
120308 8:49:34 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 126389)
120308 8:49:34 [Note] WSREP: Member 1 (webserver.unlocksamsungonline.com) synced with group.
120308 8:49:34 [Note] WSREP: Member 0 (webserver.unlocksamsungonline.eu) synced with group.
120308 8:49:34 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 126389)
120308 8:49:34 [Note] WSREP: Synchronized with group, ready for connections
120308 8:49:34 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
120308 9:35:35 [Warning] IP address '2.180.36.161' could not be resolved: Name or service not known
120308 10:07:40 [Warning] IP address '217.218.226.237' could not be resolved: Name or service not known
120308 10:27:54 [Warning] IP address '2.177.188.34' could not be resolved: Name or service not known
120308 14:38:32 [Warning] IP address '178.157.58.4' could not be resolved: Name or service not known
120308 16:50:32 [Warning] IP address '222.189.238.115' could not be resolved: Name or service not known
120308 16:58:29 [Warning] IP address '109.203.165.196' could not be resolved: Name or service not known
120308 17:55:01 [Warning] IP address '2.184.215.66' could not be resolved: Name or service not known
120308 18:09:51 [Warning] IP address '2.146.112.254' could not be resolved: Name or service not known
120308 20:44:57 [Warning] IP address '60.190.223.12' could not be resolved: Name or service not known
120308 21:20:42 [Note] WSREP: (5072c8cb-68fb-11e1-0800-de84c5727c34, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://68.169.42.153:4567
120308 21:20:43 [Note] WSREP: (5072c8cb-68fb-11e1-0800-de84c5727c34, 'tcp://0.0.0.0:4567') turning message relay requesting off
120308 23:14:22 [Note] WSREP: (5072c8cb-68fb-11e1-0800-de84c5727c34, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://46.23.72.59:4567
120308 23:14:23 [Note] WSREP: (5072c8cb-68fb-11e1-0800-de84c5727c34, 'tcp://0.0.0.0:4567') reconnecting to 732cfaf8-68f8-11e1-0800-db892ab4dca5 (tcp://46.23.72.59:4567), attempt 0
120308 23:14:34 [Note] WSREP: (5072c8cb-68fb-11e1-0800-de84c5727c34, 'tcp://0.0.0.0:4567') turning message relay requesting off
120308 23:14:54 [Note] WSREP: (5072c8cb-68fb-11e1-0800-de84c5727c34, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://46.23.72.59:4567
120308 23:14:55 [Note] WSREP: (5072c8cb-68fb-11e1-0800-de84c5727c34, 'tcp://0.0.0.0:4567') turning message relay requesting off
120309 2:10:54 [Warning] IP address '2.50.7.141' could not be resolved: Name or service not known
120309 4:24:42 [Warning] IP address '81.31.241.142' could not be resolved: Name or service not known
120309 6:32:39 [Warning] IP address '2.147.21.12' could not be resolved: Name or service not known
120309 19:40:47 [Warning] IP address '2.181.83.90' could not be resolved: Name or service not known
120310 0:12:48 [Warning] IP address '2.185.108.46' could not be resolved: Name or service not known
120310 5:44:03 [Warning] IP address '109.203.170.154' could not be resolved: Name or service not known
120310 7:27:44 [Warning] IP address '94.74.143.120' could not be resolved: Name or service not known
120310 9:46:35 [Note] WSREP: (5072c8cb-68fb-11e1-0800-de84c5727c34, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://68.169.42.153:4567
120310 9:46:37 [Note] WSREP: (5072c8cb-68fb-11e1-0800-de84c5727c34, 'tcp://0.0.0.0:4567') cleaning up established 0x7f9c90056020 which is duplicate of 0x7f9c8000e380
120310 9:46:37 [Note] WSREP: (5072c8cb-68fb-11e1-0800-de84c5727c34, 'tcp://0.0.0.0:4567') turning message relay requesting off
120310 10:05:18 [Warning] IP address '217.218.226.239' could not be resolved: Name or service not known
120310 10:30:17 [Warning] IP address '80.227.102.218' could not be resolved: Name or service not known
120310 14:04:37 [Warning] IP address '188.158.206.58' could not be resolved: Name or service not known
120310 14:24:01 [Warning] IP address '109.201.16.15' could not be resolved: Name or service not known
120310 14:38:54 [Warning] IP address '178.157.30.32' could not be resolved: Name or service not known
120310 17:49:59 [Warning] IP address '94.59.37.10' could not be resolved: Name or service not known
120310 18:00:18 [Warning] IP address '86.57.111.32' could not be resolved: Temporary failure in name resolution
120310 18:00:19 [Warning] IP address '86.57.111.32' could not be resolved: Temporary failure in name resolution
120310 18:00:23 [Warning] IP address '86.57.111.32' could not be resolved: Temporary failure in name resolution
120310 18:00:31 [Warning] IP address '86.57.111.32' could not be resolved: Temporary failure in name resolution
120310 18:00:44 [Warning] IP address '86.57.111.32' could not be resolved: Temporary failure in name resolution
120310 18:00:45 [Warning] IP address '86.57.111.32' could not be resolved: Temporary failure in name resolution
120310 18:00:51 [Warning] IP address '86.57.111.32' could not be resolved: Temporary failure in name resolution
120310 18:00:54 [Warning] IP address '86.57.111.32' could not be resolved: Temporary failure in name resolution
120310 18:00:56 [Warning] IP address '86.57.111.32' could not be resolved: Temporary failure in name resolution
120310 18:00:58 [Warning] IP address '86.57.111.32' could not be resolved: Temporary failure in name resolution
120310 18:01:00 [Warning] IP address '86.57.111.32' could not be resolved: Temporary failure in name resolution
120310 18:06:23 [Warning] IP address '86.57.111.32' could not be resolved: Temporary failure in name resolution
120310 18:06:24 [Warning] IP address '86.57.111.32' could not be resolved: Temporary failure in name resolution
120310 18:06:27 [Warning] IP address '86.57.111.32' could not be resolved: Temporary failure in name resolution
120310 18:06:27 [Warning] IP address '86.57.111.32' could not be resolved: Temporary failure in name resolution
120310 18:06:28 [Warning] IP address '86.57.111.32' could not be resolved: Temporary failure in name resolution
120310 18:06:29 [Warning] IP address '86.57.111.32' could not be resolved: Temporary failure in name resolution
120310 18:10:54 [Warning] IP address '80.191.203.94' could not be resolved: Name or service not known
120311 1:27:55 [Warning] IP address '78.39.101.162' could not be resolved: Name or service not known
120311 2:14:40 [Note] WSREP: (5072c8cb-68fb-11e1-0800-de84c5727c34, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://46.23.72.59:4567
120311 2:16:38 [Note] WSREP: (5072c8cb-68fb-11e1-0800-de84c5727c34, 'tcp://0.0.0.0:4567') reconnecting to 732cfaf8-68f8-11e1-0800-db892ab4dca5 (tcp://46.23.72.59:4567), attempt 43170
120311 2:18:38 [Note] WSREP: (5072c8cb-68fb-11e1-0800-de84c5727c34, 'tcp://0.0.0.0:4567') reconnecting to 732cfaf8-68f8-11e1-0800-db892ab4dca5 (tcp://46.23.72.59:4567), attempt 43200
120311 2:18:42 [Note] WSREP: (5072c8cb-68fb-11e1-0800-de84c5727c34, 'tcp://0.0.0.0:4567') turning message relay requesting off
120311 2:18:43 [Note] WSREP: cleaning up 732cfaf8-68f8-11e1-0800-db892ab4dca5 (tcp://46.23.72.59:4567)
120311 2:19:37 [Note] WSREP: evs::proto(5072c8cb-68fb-11e1-0800-de84c5727c34, OPERATIONAL, view_id(REG,5072c8cb-68fb-11e1-0800-de84c5727c34,10)) suspecting node: 732cfaf8-68f8-11e1-0800-db892ab4dca5
120311 2:19:37 [Note] WSREP: evs::proto(5072c8cb-68fb-11e1-0800-de84c5727c34, GATHER, view_id(REG,5072c8cb-68fb-11e1-0800-de84c5727c34,10)) suspecting node: 732cfaf8-68f8-11e1-0800-db892ab4dca5
120311 2:19:38 [Note] WSREP: GMCast::handle_stable_view: view(view_id(PRIM,5072c8cb-68fb-11e1-0800-de84c5727c34,12) memb {
 5072c8cb-68fb-11e1-0800-de84c5727c34,
 6257e849-68fb-11e1-0800-628c3a9f1cba,
} joined {
} left {
} partitioned {
 732cfaf8-68f8-11e1-0800-db892ab4dca5,
})
120311 2:19:38 [Note] WSREP: declaring 6257e849-68fb-11e1-0800-628c3a9f1cba stable
120311 2:19:38 [Note] WSREP: New COMPONENT: primary = yes, my_idx = 0, memb_num = 2
120311 2:19:38 [Note] WSREP: STATE_EXCHANGE: sent state UUID: a73f9a67-6b20-11e1-0800-021b7fc7c983
120311 2:19:38 [Note] WSREP: STATE EXCHANGE: sent state msg: a73f9a67-6b20-11e1-0800-021b7fc7c983
120311 2:19:38 [Note] WSREP: STATE EXCHANGE: got state msg: a73f9a67-6b20-11e1-0800-021b7fc7c983 from 0 (webserver.unlocksamsungonline.eu)
120311 2:19:39 [Note] WSREP: STATE EXCHANGE: got state msg: a73f9a67-6b20-11e1-0800-021b7fc7c983 from 1 (webserver.unlocksamsungonline.com)
120311 2:19:39 [Note] WSREP: Quorum results:
 version = 2,
 component = PRIMARY,
 conf_id = 9,
 members = 2/2 (joined/total),
 act_id = 131399,
 last_appl. = 130444,
 protocols = 0/3/1 (gcs/repl/appl),
 group UUID = 00d090b7-66c3-11e1-0800-23e68969619b
120311 2:19:39 [Note] WSREP: Flow-control interval: [12, 23]
120311 2:19:39 [Note] WSREP: New cluster view: global state: 00d090b7-66c3-11e1-0800-23e68969619b:131399, view# 10: Primary, number of nodes: 2, my index: 0, protocol version 1
120311 2:19:39 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
120311 2:19:39 [Note] WSREP: Assign initial position for certification: 131399, protocol version: 2
120311 3:56:57 [Warning] IP address '2.176.251.22' could not be resolved: Name or service not known
120311 8:51:51 [Warning] IP address '217.218.119.111' could not be resolved: Name or service not known
120311 9:06:14 [Warning] IP address '217.218.226.247' could not be resolved: Name or service not known
120311 9:34:52 [Warning] IP address '2.144.230.134' could not be resolved: Name or service not known
120311 10:19:40 [Warning] IP address '2.185.108.51' could not be resolved: Name or service not known
120311 10:23:59 [Note] WSREP: (5072c8cb-68fb-11e1-0800-de84c5727c34, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://68.169.42.153:4567
120311 10:24:00 [Note] WSREP: (5072c8cb-68fb-11e1-0800-de84c5727c34, 'tcp://0.0.0.0:4567') reconnecting to 6257e849-68fb-11e1-0800-628c3a9f1cba (tcp://68.169.42.153:4567), attempt 0
120311 10:24:01 [Note] WSREP: (5072c8cb-68fb-11e1-0800-de84c5727c34, 'tcp://0.0.0.0:4567') turning message relay requesting off
120311 10:24:01 [Note] WSREP: (5072c8cb-68fb-11e1-0800-de84c5727c34, 'tcp://0.0.0.0:4567') cleaning up established 0x7f9c80005660 which is duplicate of 0x7f9c90074290
120311 10:39:16 [Warning] IP address '2.182.41.95' could not be resolved: Name or service not known
120311 10:43:41 [Warning] IP address '2.182.37.211' could not be resolved: Name or service not known
120311 15:21:49 [Warning] IP address '188.159.209.189' could not be resolved: Name or service not known
120311 15:23:41 [Warning] IP address '217.219.117.253' could not be resolved: Name or service not known
120311 20:29:59 [Warning] IP address '2.144.232.106' could not be resolved: Name or service not known
120311 22:16:47 [Note] WSREP: (5072c8cb-68fb-11e1-0800-de84c5727c34, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://68.169.42.153:4567
120311 22:16:48 [Note] WSREP: (5072c8cb-68fb-11e1-0800-de84c5727c34, 'tcp://0.0.0.0:4567') turning message relay requesting off
120311 23:26:05 [Warning] IP address '2.182.69.207' could not be resolved: Name or service not known
120312 1:51:39 [Warning] IP address '2.182.64.247' could not be resolved: Name or service not known
120312 4:49:36 [Warning] IP address '2.184.55.230' could not be resolved: Name or service not known
120312 6:57:05 [Warning] IP address '2.144.161.70' could not be resolved: Name or service not known
120312 9:26:56 [Warning] IP address '85.133.160.58' has been resolved to the host name '85.133.160.58.pos-1-0.7tir.sepanta.net', which resembles IPv4-address itself.
120312 11:20:17 [Note] WSREP: (5072c8cb-68fb-11e1-0800-de84c5727c34, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://68.169.42.153:4567
120312 11:20:19 [Note] WSREP: (5072c8cb-68fb-11e1-0800-de84c5727c34, 'tcp://0.0.0.0:4567') turning message relay requesting off
120312 11:20:19 [Note] WSREP: (5072c8cb-68fb-11e1-0800-de84c5727c34, 'tcp://0.0.0.0:4567') cleaning up established 0x7f9c90051ea0 which is duplicate of 0x7f9c90074290
120312 11:57:38 [Warning] IP address '2.182.42.191' could not be resolved: Name or service not known
120312 12:45:43 [Warning] IP address '46.100.110.136' could not be resolved: Name or service not known
120312 12:50:52 [Warning] IP address '188.158.199.158' could not be resolved: Name or service not known
120312 15:07:53 [Warning] IP address '2.186.32.178' could not be resolved: Name or service not known
120312 16:33:19 [Note] WSREP: (5072c8cb-68fb-11e1-0800-de84c5727c34, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://68.169.42.153:4567
120312 16:33:23 [Note] WSREP: (5072c8cb-68fb-11e1-0800-de84c5727c34, 'tcp://0.0.0.0:4567') turning message relay requesting off
120312 16:33:23 [Note] WSREP: (5072c8cb-68fb-11e1-0800-de84c5727c34, 'tcp://0.0.0.0:4567') cleaning up duplicate 0x7f9c90023c40 after established 0x7f9c800893a0
120312 17:47:19 [Warning] IP address '178.157.30.4' could not be resolved: Name or service not known
120312 18:08:46 [Warning] IP address '2.185.144.15' could not be resolved: Name or service not known
120312 19:48:11 [Warning] IP address '2.181.91.105' could not be resolved: Name or service not known
120312 20:18:48 [Warning] IP address '46.100.121.211' could not be resolved: Name or service not known
120312 21:38:45 [Warning] IP address '2.145.229.154' could not be resolved: Name or service not known
120313 6:37:25 [Warning] IP address '46.100.108.18' could not be resolved: Name or service not known
120313 8:02:08 [Warning] IP address '2.147.27.250' could not be resolved: Name or service not known
120313 8:19:59 [Warning] IP address '80.191.117.106' could not be resolved: Name or service not known
120313 8:43:36 [Warning] IP address '188.158.206.142' could not be resolved: Name or service not known
120313 8:54:52 [Warning] IP address '2.180.74.175' could not be resolved: Name or service not known
120313 10:10:42 [Warning] IP address '2.184.246.252' could not be resolved: Name or service not known
120313 10:18:10 [Warning] IP address '2.146.119.97' could not be resolved: Name or service not known
120313 10:49:55 [Note] WSREP: (5072c8cb-68fb-11e1-0800-de84c5727c34, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://68.169.42.153:4567
120313 10:49:56 [Note] WSREP: (5072c8cb-68fb-11e1-0800-de84c5727c34, 'tcp://0.0.0.0:4567') turning message relay requesting off
120313 10:49:56 [Note] WSREP: (5072c8cb-68fb-11e1-0800-de84c5727c34, 'tcp://0.0.0.0:4567') cleaning up established 0x7f9c8007efb0 which is duplicate of 0x7f9c90051ea0
120313 14:47:44 [Warning] IP address '178.131.58.248' could not be resolved: Name or service not known
120313 15:23:50 [Warning] IP address '2.190.13.224' could not be resolved: Name or service not known
120313 17:01:57 [Warning] IP address '178.157.30.87' could not be resolved: Name or service not known
120313 18:36:32 [Warning] IP address '2.180.45.239' could not be resolved: Name or service not known
120313 21:47:57 [Warning] IP address '2.144.218.180' could not be resolved: Name or service not known
120313 22:04:31 [Note] WSREP: (5072c8cb-68fb-11e1-0800-de84c5727c34, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://68.169.42.153:4567
120313 22:06:28 [Note] WSREP: (5072c8cb-68fb-11e1-0800-de84c5727c34, 'tcp://0.0.0.0:4567') reconnecting to 6257e849-68fb-11e1-0800-628c3a9f1cba (tcp://68.169.42.153:4567), attempt 43170
120313 22:08:29 [Note] WSREP: (5072c8cb-68fb-11e1-0800-de84c5727c34, 'tcp://0.0.0.0:4567') reconnecting to 6257e849-68fb-11e1-0800-628c3a9f1cba (tcp://68.169.42.153:4567), attempt 43200
120313 22:08:32 [Note] WSREP: (5072c8cb-68fb-11e1-0800-de84c5727c34, 'tcp://0.0.0.0:4567') turning message relay requesting off
120313 22:08:33 [Note] WSREP: cleaning up 6257e849-68fb-11e1-0800-628c3a9f1cba (tcp://68.169.42.153:4567)
120313 22:09:28 [Note] WSREP: evs::proto(5072c8cb-68fb-11e1-0800-de84c5727c34, OPERATIONAL, view_id(REG,5072c8cb-68fb-11e1-0800-de84c5727c34,12)) suspecting node: 6257e849-68fb-11e1-0800-628c3a9f1cba
120313 22:09:28 [Note] WSREP: evs::proto(5072c8cb-68fb-11e1-0800-de84c5727c34, GATHER, view_id(REG,5072c8cb-68fb-11e1-0800-de84c5727c34,12)) suspecting node: 6257e849-68fb-11e1-0800-628c3a9f1cba
===repeat above line many times===

22:10:28 [Note] WSREP: GMCast::handle_stable_view: view(view_id(NON_PRIM,5072c8cb-68fb-11e1-0800-de84c5727c34,12) memb {
 5072c8cb-68fb-11e1-0800-de84c5727c34,
} joined {
} left {
} partitioned {
 6257e849-68fb-11e1-0800-628c3a9f1cba,
})
120313 22:10:28 [Note] WSREP: GMCast::handle_stable_view: view(view_id(NON_PRIM,5072c8cb-68fb-11e1-0800-de84c5727c34,13) memb {
 5072c8cb-68fb-11e1-0800-de84c5727c34,
} joined {
} left {
} partitioned {
 6257e849-68fb-11e1-0800-628c3a9f1cba,
})
120313 22:10:28 [Note] WSREP: New COMPONENT: primary = no, my_idx = 0, memb_num = 1
120313 22:10:28 [Note] WSREP: Flow-control interval: [8, 16]
120313 22:10:28 [Note] WSREP: Received NON-PRIMARY.
120313 22:10:28 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 135875)
120313 22:10:28 [Note] WSREP: New COMPONENT: primary = no, my_idx = 0, memb_num = 1
120313 22:10:28 [Note] WSREP: Flow-control interval: [8, 16]
120313 22:10:28 [Note] WSREP: Received NON-PRIMARY.
120313 22:10:28 [Note] WSREP: New cluster view: global state: 00d090b7-66c3-11e1-0800-23e68969619b:135875, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 1
120313 22:10:28 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
120313 22:10:28 [Note] WSREP: New cluster view: global state: 00d090b7-66c3-11e1-0800-23e68969619b:135875, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 1
120313 22:10:28 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
120313 22:12:39 [Warning] IP address '109.238.178.14' could not be resolved: Name or service not known

NL server log (control server)
120308 8:47:52 [Note] WSREP: New cluster view: global state: 00d090b7-66c3-11e1-0800-23e68969619b:126389, view# 9: Primary, number of nodes: 3, my index: 2, protocol version 1
120308 8:47:52 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
120308 8:47:52 [Note] WSREP: Assign initial position for certification: 126389, protocol version: 2
120308 8:47:54 [Note] WSREP: Node 1 (webserver.unlocksamsungonline.com) requested state transfer from '*any*'. Selected 0 (webserver.unlocksamsungonline.eu)(SYNCED) as donor.
120308 8:49:38 [Note] WSREP: 1 (webserver.unlocksamsungonline.com): State transfer from 0 (webserver.unlocksamsungonline.eu) complete.
120308 8:49:38 [Note] WSREP: 0 (webserver.unlocksamsungonline.eu): State transfer to 1 (webserver.unlocksamsungonline.com) complete.
120308 8:49:38 [Note] WSREP: Member 1 (webserver.unlocksamsungonline.com) synced with group.
120308 8:49:38 [Note] WSREP: Member 0 (webserver.unlocksamsungonline.eu) synced with group.
120308 13:10:08 [Note] WSREP: (732cfaf8-68f8-11e1-0800-db892ab4dca5, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://68.169.42.153:4567
120308 13:10:09 [Note] WSREP: (732cfaf8-68f8-11e1-0800-db892ab4dca5, 'tcp://0.0.0.0:4567') reconnecting to 6257e849-68fb-11e1-0800-628c3a9f1cba (tcp://68.169.42.153:4567), attempt 0
120308 13:10:17 [Note] WSREP: (732cfaf8-68f8-11e1-0800-db892ab4dca5, 'tcp://0.0.0.0:4567') turning message relay requesting off
120308 13:10:17 [Note] WSREP: (732cfaf8-68f8-11e1-0800-db892ab4dca5, 'tcp://0.0.0.0:4567') cleaning up established 0x3bf5210 which is duplicate of 0x3dc2f30
120308 22:56:50 [Note] WSREP: (732cfaf8-68f8-11e1-0800-db892ab4dca5, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://68.169.42.153:4567
120308 22:56:51 [Note] WSREP: (732cfaf8-68f8-11e1-0800-db892ab4dca5, 'tcp://0.0.0.0:4567') cleaning up established 0x3bc1a80 which is duplicate of 0x3d5b3b0
120308 22:56:51 [Note] WSREP: (732cfaf8-68f8-11e1-0800-db892ab4dca5, 'tcp://0.0.0.0:4567') turning message relay requesting off
120308 23:14:27 [Note] WSREP: (732cfaf8-68f8-11e1-0800-db892ab4dca5, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://83.170.88.228:4567
120308 23:14:28 [Note] WSREP: (732cfaf8-68f8-11e1-0800-db892ab4dca5, 'tcp://0.0.0.0:4567') reconnecting to 5072c8cb-68fb-11e1-0800-de84c5727c34 (tcp://83.170.88.228:4567), attempt 0
120308 23:14:31 [Note] WSREP: (732cfaf8-68f8-11e1-0800-db892ab4dca5, 'tcp://0.0.0.0:4567') reconnecting to 5072c8cb-68fb-11e1-0800-de84c5727c34 (tcp://83.170.88.228:4567), attempt 0
120308 23:14:40 [Note] WSREP: (732cfaf8-68f8-11e1-0800-db892ab4dca5, 'tcp://0.0.0.0:4567') turning message relay requesting off
120308 23:14:44 [Note] WSREP: (732cfaf8-68f8-11e1-0800-db892ab4dca5, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://68.169.42.153:4567
120308 23:14:49 [Note] WSREP: (732cfaf8-68f8-11e1-0800-db892ab4dca5, 'tcp://0.0.0.0:4567') turning message relay requesting off
120308 23:14:49 [Note] WSREP: (732cfaf8-68f8-11e1-0800-db892ab4dca5, 'tcp://0.0.0.0:4567') cleaning up established 0x3bc1a80 which is duplicate of 0x3db5390
120308 23:14:59 [Note] WSREP: (732cfaf8-68f8-11e1-0800-db892ab4dca5, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://83.170.88.228:4567
120308 23:15:00 [Note] WSREP: (732cfaf8-68f8-11e1-0800-db892ab4dca5, 'tcp://0.0.0.0:4567') turning message relay requesting off
120309 2:26:33 [Note] WSREP: (732cfaf8-68f8-11e1-0800-db892ab4dca5, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://68.169.42.153:4567
120309 2:26:34 [Note] WSREP: (732cfaf8-68f8-11e1-0800-db892ab4dca5, 'tcp://0.0.0.0:4567') turning message relay requesting off
120309 2:59:00 [Note] WSREP: (732cfaf8-68f8-11e1-0800-db892ab4dca5, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://68.169.42.153:4567
120309 2:59:01 [Note] WSREP: (732cfaf8-68f8-11e1-0800-db892ab4dca5, 'tcp://0.0.0.0:4567') turning message relay requesting off
120309 12:02:49 [Warning] IP address '222.76.215.105' could not be resolved: Name or service not known
120309 12:40:21 [Warning] IP address '31.171.2.15' could not be resolved: Name or service not known
120309 14:04:23 [Warning] IP address '178.187.31.134' has been resolved to the host name '178.187.31-134.xdsl.ab.ru', which resembles IPv4-address itself.
120310 9:38:09 [Note] WSREP: (732cfaf8-68f8-11e1-0800-db892ab4dca5, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://68.169.42.153:4567
120310 9:38:11 [Note] WSREP: (732cfaf8-68f8-11e1-0800-db892ab4dca5, 'tcp://0.0.0.0:4567') cleaning up established 0x7f3a080e5bd0 which is duplicate of 0x7f3a0809f780
120310 9:38:11 [Note] WSREP: (732cfaf8-68f8-11e1-0800-db892ab4dca5, 'tcp://0.0.0.0:4567') turning message relay requesting off

weirdly enough the log of the NL server stops the 10th.. so seems it died that as there is nothing more there..

Alex Yurchenko (ayurchen) said : #9

we will release 23.2.1beta in a day, it will be essentially 2.0 with the fix to this problem.

Teemu Ollakka (teemu-ollakka) said : #10

Steven, have you checked NL server system logs if there is anything special happening around 120311 2:14? Log ending so abruptly is very suspicious, sounds like either MySQL server was killed with -9 (OOM killer perhaps?) or node was powered down.

Alex Yurchenko (ayurchen) said : #11
Steven (steven-7tv) said : #12

@Teemu I will do that and have a look if I can find anything weird there.

@Alex thanks. Will ask vps.net to install it for me asap.

Steven (steven-7tv) said : #13

Thanks Alex Yurchenko, that solved my question.

Steven (steven-7tv) said : #14

Hi,

Apparently I can't use that release as I'm using the self-contained Galera distribution (64-bit) from https://launchpad.net/codership-mysql/5.5/5.5.20-23.4. To use this release we will need to do all manually which on a production server I'd rather not do.

Could you provide me with a aprox date it will be released in package?

Thanks

Launchpad Janitor (janitor) said : #15

This question was expired because it remained in the 'Open' state without activity for the last 15 days.

Hi!

I'm making some test on a 3-node configuration goofing around with iptables and get this log too
(marked with nil view id and suspected in all present join messages, declaring inactive, etc)

On node 1 i put a rule so all input traffic from node 2 will be dropped
iptables -A INPUT -s node2.test -j DROP

Node 1 went from synced with the original cluster to be in Initialized state for its own cluster (size 1). Original cluster went from size 3 to 2.
Everything looks fine until here.
Then, i deleted the iptable rule (iptables -F) waiting for node 1 to join again the original cluster (wsrep_cluster_address have the node 3 DNS). But
that didn't happened. Instead, i get a very verbose log like these:

120426 14:00:41 [Note] WSREP: evs::proto(afadd5ae-8fc9-11e1-0800-d0721a3dbebc, GATHER, view_id(REG,afa06e96-8fc9-11e1-0800-b5d7c7cd4386,2)) suspecting node: 064be1d6-8ee3-11e1-0800-096d704a858c
120426 14:00:41 [Note] WSREP: evs::proto(afadd5ae-8fc9-11e1-0800-d0721a3dbebc, GATHER, view_id(REG,afa06e96-8fc9-11e1-0800-b5d7c7cd4386,2)) suspecting node: 2df562ce-8ee9-11e1-0800-30bd52fa40d8
120426 14:00:41 [Note] WSREP: node 064be1d6-8ee3-11e1-0800-096d704a858c marked with nil view id and suspected in all present join messages, declaring inactive
120426 14:00:41 [Note] WSREP: node 2df562ce-8ee9-11e1-0800-30bd52fa40d8 marked with nil view id and suspected in all present join messages, declaring inactive
120426 14:00:41 [Note] WSREP: node 064be1d6-8ee3-11e1-0800-096d704a858c marked with nil view id and suspected in all present join messages, declaring inactive
120426 14:00:41 [Note] WSREP: node 2df562ce-8ee9-11e1-0800-30bd52fa40d8 marked with nil view id and suspected in all present join messages, declaring inactive
120426 14:00:41 [Note] WSREP: node 064be1d6-8ee3-11e1-0800-096d704a858c marked with nil view id and suspected in all present join messages, declaring inactive
120426 14:00:41 [Note] WSREP: node 2df562ce-8ee9-11e1-0800-30bd52fa40d8 marked with nil view id and suspected in all present join messages, declaring inactive
120426 14:00:41 [Note] WSREP: node 064be1d6-8ee3-11e1-0800-096d704a858c marked with nil view id and suspected in all present join messages, declaring inactive
120426 14:00:41 [Note] WSREP: node 2df562ce-8ee9-11e1-0800-30bd52fa40d8 marked with nil view id and suspected in all present join messages, declaring inactive
120426 14:00:41 [Note] WSREP: declaring afa06e96-8fc9-11e1-0800-b5d7c7cd4386 stable
120426 14:00:41 [Note] WSREP: declaring fc6a8240-8ee6-11e1-0800-9d69e21c4929 stable
120426 14:00:41 [Note] WSREP: view(view_id(NON_PRIM,afa06e96-8fc9-11e1-0800-b5d7c7cd4386,39) memb {
 afa06e96-8fc9-11e1-0800-b5d7c7cd4386,
 afadd5ae-8fc9-11e1-0800-d0721a3dbebc,
 fc6a8240-8ee6-11e1-0800-9d69e21c4929,
} joined {
} left {
} partitioned {
 064be1d6-8ee3-11e1-0800-096d704a858c,
 2df562ce-8ee9-11e1-0800-30bd52fa40d8,
})
120426 14:00:41 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 1, memb_num = 3
120426 14:00:41 [Note] WSREP: Flow-control interval: [14, 28]
120426 14:00:41 [Note] WSREP: Received NON-PRIMARY.
120426 14:00:41 [Note] WSREP: New cluster view: global state: :-1, view# -1: non-Primary, number of nodes: 3, my index: 1, protocol version -1
120426 14:00:41 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification

Seems that it can't handle well a recovery from a TCP connection problem or i'm doing something wrong?
Node 1 and node 3 get hang up (mysqld running but not responding even using the local client)
Node 2 still see the cluster as size 3 but is in Initialized state. (and of course, saying "Unknow command")

Hope it help