Problem moving writer role between masters

I suspect that ARP traffic doesn't flow freely between the three hosts
and that is what is causing the IP to neer get assigned to the new
host.

Walter

On Wed, Aug 11, 2010 at 03:13, Kevin
<email address hidden> wrote:
> New question #120649 on mysql-mmm:
> https://answers.launchpad.net/mysql-mmm/+question/120649
>
> I have a master master setup with a third host setup as a monitor (monitor host IP is 192.168.14.238).
>
> output for mmm_control show is as follows:
>
>  db1(192.168.14.239) master/ONLINE. Roles: reader(192.168.14.195), writer(192.168.14.194)
>  db2(192.168.14.240) master/ONLINE. Roles: reader(192.168.14.196)
>
> Problem is every time I execute a move role say from db1 to db2 the output of mmm_control show changes to the following
> which is expected
>
>  db1(192.168.14.239) master/ONLINE. Roles: reader(192.168.14.195)
>  db2(192.168.14.240) master/ONLINE. Roles: reader(192.168.14.196), writer(192.168.14.194)
>
> But the address 192.168.14.194 never shows up on 192.168.14.240.
> I can't ping it from the monitor either. But if I set it manually on the 14.240 using the confgure_ip script
>  configure_ip mmm_agent eth0 192.168.14.194 it shows up in output for the "ip addr list"  command which again is expected.
>
> Now the interesting thing is if I move the writer role back to db1 192.168.14.194 disappears from db2 but never shows up on db1.
> I tried setting db2 to offline just to check if it moves the reader role to db1 and it does that. It shows ip in the output of 'ip addr list" on db1.
>
> I am running on centos 5.5.
>
> my common config file is as follows:
>
> active_master_role      writer
>
> <host default>
>        cluster_interface       eth0
>        pid_path                /var/run/mmm_agentd.pid
>        bin_path                /usr/lib/mysql-mmm/
>        replication_user        repl
>        replication_password    replpasswd
>        agent_user              rep_agent
>        agent_password          rep1agent
> </host>
> <host db1>
>        ip                      192.168.14.239
>        mode                    master
>        peer                    db2
> </host>
> <host db2>
>        ip                      192.168.14.240
>        mode                    master
>        peer                    db1
> </host>
> <role writer>
>        hosts                   db1, db2
>        ips                     192.168.14.194
>        mode                    exclusive
> </role>
>
> <role reader>
>        hosts                   db1, db2
>        ips                     192.168.14.195, 192.168.14.196
>        mode                    balanced
> </role>
>
> The above file is the same on all 3 hosts. The monitor config file is as follows:
>
> include mmm_common.conf
>
> <monitor>
>        ip                      127.0.0.1
>        pid_path                /var/run/mmm_mond.pid
>        bin_path                /usr/lib/mysql-mmm/
>        status_path             /var/lib/misc/mmm_mond.status
>        ping_ips                192.168.14.194, 192.168.14.195, 192.168.14.196, 192.168.14.239, 192.168.14.240
> </monitor>
>
> <host default>
>        monitor_user            rep_monitor
>        monitor_password        rep1monitor
> </host>
>
> debug 0
>
> Please let me know if you know what I am doing wrong or any way i can to get your more information in order to resolve this issue.
>
> Thanks
>
>
>
>
>
>
>
>
>
>
>
>
> --
> You received this question notification because you are an answer
> contact for mysql-mmm.
>

But how do you explain the fact that the reader IP can be assigned?
Its just the writer thats having the problem.

Thought I'd update you guys.

This is the output with debug set to 1. I started the monitor and and changed the role.

This is a smalle excerpt from the log

2010/08/11 15:07:28 DEBUG Sending command 'SET_STATUS(ONLINE, reader(192.168.14.195), db1)' to db2 (192.168.14.240:9989)
2010/08/11 15:07:28 DEBUG Received Answer: OK: Status applied successfully!|UP:72411.09
2010/08/11 15:07:28 DEBUG Sending command 'SET_STATUS(ONLINE, reader(192.168.14.196),writer(192.168.14.194), db1)' to db1 (192.168.14.239:9989)
2010/08/11 15:07:28 DEBUG Received Answer: OK: Status applied successfully!|UP:72479.45
2010/08/11 15:07:31 DEBUG Sending command 'SET_STATUS(ONLINE, reader(192.168.14.195), db1)' to db2 (192.168.14.240:9989)
2010/08/11 15:07:31 DEBUG Received Answer: OK: Status applied successfully!|UP:72414.09
2010/08/11 15:07:31 DEBUG Sending command 'SET_STATUS(ONLINE, reader(192.168.14.196),writer(192.168.14.194), db1)' to db1 (192.168.14.239:9989)
2010/08/11 15:07:31 DEBUG Received Answer: OK: Status applied successfully!|UP:72482.45
2010/08/11 15:07:32 DEBUG Sending command 'PING()' to db2 (192.168.14.240:9989)
2010/08/11 15:07:32 DEBUG Received Answer: OK: Pinged!|UP:72415.50
2010/08/11 15:07:32 FATAL Admin moved role 'writer' from 'db1' to 'db2' ----> How do I get more info ?
2010/08/11 15:07:32 DEBUG Sending command 'SET_STATUS(ONLINE, reader(192.168.14.196), db2)' to db1 (192.168.14.239:9989)
2010/08/11 15:07:32 DEBUG Received Answer: OK: Status applied successfully!|UP:72484.05
2010/08/11 15:07:32 DEBUG Sending command 'SET_STATUS(ONLINE, reader(192.168.14.195),writer(192.168.14.194), db2)' to db2 (192.168.14.240:9989)
2010/08/11 15:07:32 DEBUG Received Answer: OK: Status applied successfully!|UP:72415.82
2010/08/11 15:07:32 DEBUG Sending command 'SET_STATUS(ONLINE, reader(192.168.14.195),writer(192.168.14.194), db2)' to db2 (192.168.14.240:9989)
2010/08/11 15:07:32 DEBUG Received Answer: OK: Status applied successfully!|UP:72416.00
2010/08/11 15:07:32 DEBUG Sending command 'SET_STATUS(ONLINE, reader(192.168.14.196), db2)' to db1 (192.168.14.239:9989)
2010/08/11 15:07:32 DEBUG Received Answer: OK: Status applied successfully!|UP:72484.37

There's nothign wrong with that log. The reason you see FATAL is that that is the level at which it gets logged since it is important info. This means that teh problem is elsewhere. Pascal, do you have any ideas?

Hm, I've no idea.

Maybe start the agents in debug mode and add some more debugging statements to see if/how configure_ip is invoked.

I am experiencing the exact same problem on ubuntu 10.10.

The floating IP address for the writer role (10.0.0.20) is not applying on the actual server. However according to mmm_monitor it has.

# mmm_control show (monitor node)
db1(10.0.0.2) master/ONLINE. Roles: reader(10.0.0.21), reader(10.0.0.22), writer(10.0.0.20)

# /var/log/mysql-mmm/mmm_mond.log (monitor node)
2011/01/24 08:49:50 FATAL Admin changed state of 'db1' from ONLINE to ADMIN_OFFLINE
2011/01/24 08:49:50 INFO Removing all roles from host 'db1':
2011/01/24 08:49:50 INFO Removed role 'reader(10.0.0.22)' from host 'db1'
2011/01/24 08:49:50 INFO Removed role 'reader(10.0.0.21)' from host 'db1'
2011/01/24 08:49:50 INFO Removed role 'writer(10.0.0.20)' from host 'db1'
2011/01/24 08:51:07 FATAL Admin changed state of 'db1' from ADMIN_OFFLINE to ONLINE
2011/01/24 08:51:07 INFO Orphaned role 'writer(10.0.0.20)' has been assigned to 'db1'
2011/01/24 08:51:07 INFO Orphaned role 'reader(10.0.0.22)' has been assigned to 'db1'
2011/01/24 08:51:07 INFO Orphaned role 'reader(10.0.0.21)' has been assigned to 'db1'

# ip addr list eth1 (db1 node)
2: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 4096 qdisc pfifo_fast state UNKNO
WN qlen 1000
    link/ether 00:14:d1:17:57:8f brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.2/24 brd 10.0.0.255 scope global eth1
    inet 10.0.0.21/32 scope global eth1
    inet 10.0.0.22/32 scope global eth1
    inet6 fe80::214:d1ff:fe17:578f/64 scope link
       valid_lft forever preferred_lft forever

--------------
MMM should be checking to make sure the ARP applies on the target server, to apply the floating ip. If the ip does not apply it should try again.

I had the exact same problem, and it turned out to be mysql permissions. Check your /var/log/mysql-mmm/* logs on each box to make sure there are no errors popping up. My mmm-mon logs looked fine but the agent logs on the db's had errors.