neutron

Quantum setup broken after reboot

Asked by dunkel_licht

Hey guys,

I'm working on an OpenStack setup and followed the basic 3 node installation example on Ubuntu 12.04 using the from
http://ubuntu-cloud.archive.canonical.com/ubuntu precise-updates/folsom repository.
I set up network with the recommended quantum-networking.sh. So far everything worked at first. Today I found that the external router-ip wasn't reachable anymore. So I rebootet the networking node and it became worse than before. So here is my status after reboot:

root@network:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:50:56:9d:b1:67 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.232/24 brd 192.168.0.255 scope global eth0
    inet6 fe80::250:56ff:fe9d:b167/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:50:56:9d:e6:75 brd ff:ff:ff:ff:ff:ff
    inet 10.10.10.232/24 brd 10.10.10.255 scope global eth1
    inet6 fe80::250:56ff:fe9d:e675/64 scope link
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:50:56:9d:8c:95 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::250:56ff:fe9d:8c95/64 scope link
       valid_lft forever preferred_lft forever
6: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
    link/ether aa:7a:ae:66:32:46 brd ff:ff:ff:ff:ff:ff
7: br-ex: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
    link/ether 00:50:56:9d:8c:95 brd ff:ff:ff:ff:ff:ff
8: tap66178edc-18: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
    link/ether c2:18:3b:96:24:f8 brd ff:ff:ff:ff:ff:ff
    inet 10.5.5.2/24 brd 10.5.5.255 scope global tap66178edc-18
9: qr-27a2e49b-d0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
    link/ether 8a:a3:c3:cc:71:8a brd ff:ff:ff:ff:ff:ff
    inet 10.5.5.1/24 brd 10.5.5.255 scope global qr-27a2e49b-d0
10: qg-9ba11a5d-c3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
    link/ether c6:d5:37:3a:e0:97 brd ff:ff:ff:ff:ff:ff
    inet 80.237.213.208/26 brd 80.237.213.255 scope global qg-9ba11a5d-c3
11: br-tun: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
    link/ether 4e:68:12:ca:e8:42 brd ff:ff:ff:ff:ff:ff

What I miss in the ip configuration is the external ip 80.237.213.232 on br-ex that had been there before reboot.

root@network:~# ip r
default via 192.168.0.225 dev eth0 metric 100
10.10.10.0/24 dev eth1 proto kernel scope link src 10.10.10.232
192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.232

No routes are set besides the one that are brought up by linux's network stack.
In this case is 192.168.0.0/24 my management lan and 10.10.10.0/24 my data-lan. I use GRE tunnel to connect compute nodes to quantum. Please let me know if you need additional information. You'll find some output from quantum commands below.

root@network:~# quantum net-list
+--------------------------------------+----------+--------------------------------------+
| id | name | subnets |
+--------------------------------------+----------+--------------------------------------+
| 09d13045-56c0-4747-9aa1-1ae04c98f230 | demo-net | 2eac3bbd-0547-49a3-9696-42b16a338ca9 |
| f665454d-8159-4d13-a40f-d3d1f0bc1149 | ext_net | 133a7656-bdff-4814-8b6b-ecdf7b372973 |
+--------------------------------------+----------+--------------------------------------+
root@network:~# quantum subnet-show 2eac3bbd-0547-49a3-9696-42b16a338ca9
+------------------+--------------------------------------------+
| Field | Value |
+------------------+--------------------------------------------+
| allocation_pools | {"start": "10.5.5.2", "end": "10.5.5.254"} |
| cidr | 10.5.5.0/24 |
| dns_nameservers | 8.8.8.8 |
| enable_dhcp | True |
| gateway_ip | 10.5.5.1 |
| host_routes | |
| id | 2eac3bbd-0547-49a3-9696-42b16a338ca9 |
| ip_version | 4 |
| name | |
| network_id | 09d13045-56c0-4747-9aa1-1ae04c98f230 |
| tenant_id | cf4250b268194c4dbbd9c85ae8f1ceba |
+------------------+--------------------------------------------+
root@network:~# quantum subnet-show 133a7656-bdff-4814-8b6b-ecdf7b372973
+------------------+------------------------------------------------------+
| Field | Value |
+------------------+------------------------------------------------------+
| allocation_pools | {"start": "80.237.213.208", "end": "80.237.213.223"} |
| cidr | 80.237.213.192/26 |
| dns_nameservers | |
| enable_dhcp | False |
| gateway_ip | 80.237.213.193 |
| host_routes | |
| id | 133a7656-bdff-4814-8b6b-ecdf7b372973 |
| ip_version | 4 |
| name | |
| network_id | f665454d-8159-4d13-a40f-d3d1f0bc1149 |
| tenant_id | 09a14c7334094f86b75938978db58d99 |
+------------------+------------------------------------------------------+
root@network:~# quantum router-list
+--------------------------------------+-----------------+--------------------------------------------------------+
| id | name | external_gateway_info |
+--------------------------------------+-----------------+--------------------------------------------------------+
| cc2cd248-ce60-47ff-841a-6cbf6915b86c | provider-router | {"network_id": "f665454d-8159-4d13-a40f-d3d1f0bc1149"} |
+--------------------------------------+-----------------+--------------------------------------------------------+
root@network:~# quantum port-list -- --device_id cc2cd248-ce60-47ff-841a-6cbf6915b86c --device_owner network:router_gateway+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------------+
| id | name | mac_address | fixed_ips |
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------------+
| 9ba11a5d-c371-4560-8f6d-edeccc90d8ef | | fa:16:3e:3f:2d:3a | {"subnet_id": "133a7656-bdff-4814-8b6b-ecdf7b372973", "ip_address": "80.237.213.208"} |
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------------+

Right now I don't know where to start debugging, so any help is appreciated.

Cheers

Question information

Language:
English Edit question
Status:
Solved
For:
neutron Edit question
Assignee:
No assignee Edit question
Solved by:
dunkel_licht
Solved:
Last query:
Last reply:
Revision history for this message
Aaron Rosen (arosen) said : 		#1

are all the agents running?

Revision history for this message
dunkel_licht (markus-sendingthesea) said : 		#2

Thanks a lot for your reply. I have this services and processes running. Would you expect anything else?

root@network:~# service openvswitch-switch status
ovsdb-server is running with pid 1103
ovs-vswitchd is running with pid 1130

root@network:~# service quantum-dhcp-agent status
quantum-dhcp-agent start/running, process 1015

root@network:~# service quantum-l3-agent status
quantum-l3-agent start/running, process 1016

root@network:~# service quantum-plugin-openvswitch-agent status
quantum-plugin-openvswitch-agent start/running, process 1013

root@network:~# ps -ef|grep -i quantum
quantum 1013 1 0 10:58 ? 00:06:08 python /usr/bin/quantum-openvswitch-agent --config-file=/etc/quantum/quantum.conf --config-file=/etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini --log-file=/var/log/quantum/openvswitch-agent.log
quantum 1015 1 0 10:58 ? 00:00:00 python /usr/bin/quantum-dhcp-agent --config-file=/etc/quantum/quantum.conf --config-file=/etc/quantum/dhcp_agent.ini --log-file=/var/log/quantum/dhcp-agent.log
quantum 1016 1 0 10:58 ? 00:02:44 python /usr/bin/quantum-l3-agent --config-file=/etc/quantum/quantum.conf --config-file=/etc/quantum/l3_agent.ini --log-file=/var/log/quantum/l3-agent.log
nobody 1988 1 0 10:58 ? 00:00:00 /usr/sbin/dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tap66178edc-18 --except-interface=lo --domain=openstacklocal --pid-file=/var/lib/quantum/dhcp/09d13045-56c0-4747-9aa1-1ae04c98f230/pid --dhcp-hostsfile=/var/lib/quantum/dhcp/09d13045-56c0-4747-9aa1-1ae04c98f230/host --dhcp-optsfile=/var/lib/quantum/dhcp/09d13045-56c0-4747-9aa1-1ae04c98f230/opts --dhcp-script=/usr/bin/quantum-dhcp-agent-dnsmasq-lease-update --leasefile-ro --dhcp-range=set:tag0,10.5.5.0,static,120s
root 1989 1988 0 10:58 ? 00:00:00 /usr/sbin/dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tap66178edc-18 --except-interface=lo --domain=openstacklocal --pid-file=/var/lib/quantum/dhcp/09d13045-56c0-4747-9aa1-1ae04c98f230/pid --dhcp-hostsfile=/var/lib/quantum/dhcp/09d13045-56c0-4747-9aa1-1ae04c98f230/host --dhcp-optsfile=/var/lib/quantum/dhcp/09d13045-56c0-4747-9aa1-1ae04c98f230/opts --dhcp-script=/usr/bin/quantum-dhcp-agent-dnsmasq-lease-update --leasefile-ro --dhcp-range=set:tag0,10.5.5.0,static,120s
root 16281 13916 0 22:30 pts/0 00:00:00 grep --color=auto -i quantum

root@network:~# ps -ef|grep -i openvswitch
quantum 1013 1 0 10:58 ? 00:06:09 python /usr/bin/quantum-openvswitch-agent --config-file=/etc/quantum/quantum.conf --config-file=/etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini --log-file=/var/log/quantum/openvswitch-agent.log
root 1103 1102 0 10:58 ? 00:00:30 ovsdb-server /etc/openvswitch/conf.db -vANY:CONSOLE:EMER -vANY:SYSLOG:ERR -vANY:FILE:INFO --remote=punix:/var/run/openvswitch/db.sock --remote=db:Open_vSwitch,manager_options --private-key=db:SSL,private_key --certificate=db:SSL,certificate --bootstrap-ca-cert=db:SSL,ca_cert --no-chdir --log-file=/var/log/openvswitch/ovsdb-server.log --pidfile=/var/run/openvswitch/ovsdb-server.pid --detach --monitor
root 1130 1129 0 10:58 ? 00:00:23 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vANY:CONSOLE:EMER -vANY:SYSLOG:ERR -vANY:FILE:INFO --mlockall --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor
root 16555 13916 0 22:31 pts/0 00:00:00 grep --color=auto -i openvswitch
root@network:~#

Revision history for this message
dunkel_licht (markus-sendingthesea) said : 		#3

I checked again today and found two network interfaces twisted, so that data and api-network weren't connected correctly.