neutron

Quantum Multi-node setup instances cannot ping each other from different nodes

Asked by Eoghan

I’ve been trying to get Quantum multi-node with openvswitch working but cannot get the instances to be able to ping each other across nodes. I have tried with the milestone-proposed as well as the master branch of Quantum and in neither case can instances on different nodes ping each other.

I tried the devstack and used the flags indicated in the Quantum wiki for services on the second node so stackrc on the second node has this:
ENABLED_SERVICES=n-cpu,rabbit,g-api,quantum,q-agt,q-dhcp

Localrc on the controller node (running all services) has this
ENABLE_TENANT_VLAN=True
TENANT_VLAN_RANGE=1:1000
PHYSICAL_NETWORK=eth0
OVS_PHYSICAL_BRIDGE=br-eth0

The physical host networking is eth0.160 as the primary and eth1.820 for the floating IP’s.

Ovs-vsctl on the controller node
8d3731cf-1ac2-4df8-86e5-47ed0a5106ac
    Bridge br-ex
        Port "qg-fe7b62d0-51"
            Interface "qg-fe7b62d0-51"
                type: internal
        Port br-ex
            Interface br-ex
                type: internal
    Bridge "br-eth0"
        Port "br-eth0"
            Interface "br-eth0"
                type: internal
        Port "phy-br-eth0"
            Interface "phy-br-eth0"
    Bridge br-int
        Port "tap55d1e5e8-ab"
            tag: 1
            Interface "tap55d1e5e8-ab"
                type: internal
        Port "qr-4b50a17d-3c"
            tag: 1
            Interface "qr-4b50a17d-3c"
                type: internal
        Port br-int
            Interface br-int
                type: internal
        Port "int-br-eth0"
            Interface "int-br-eth0"
    ovs_version: "1.4.0+build0"

Ovs-vsctl on the second node
38dbbac7-f643-4bdb-8aa1-83f06e612ec4
    Bridge br-int
        Port br-int
            Interface br-int
                type: internal
        Port "tap26583155-34"
            tag: 1
            Interface "tap26583155-34"
                type: internal
    ovs_version: "1.4.0+build0"

Instances come up and can be pinged in the namespace of the server where they are spun up but cannot ping from instances on the controller node to the second node.

I also tried the steps in this question https://answers.launchpad.net/quantum/+question/208822 but that did not resolve the problem either.

I checked the router namespace and I see the SNAT and DNAT rules are set correctly. What else needs to be added/corrected here to make multi-node work?

Thanks
Eoghan

Question information

Language:
English Edit question
Status:
Solved
For:
neutron Edit question
Assignee:
No assignee Edit question
Solved by:
Eoghan
Solved:
Last query:
Last reply:
Revision history for this message
Salvatore Orlando (salvatore-orlando) said : 		#1

Hi Eoghan,

the first thing that struck me is that br-int on the second node does not have a port towards the physical bridge.
As you're using the OVS plugin in VLAN mode, this port is required for sending packets over the physical network.

On the second node, is the ovs agent starting with the same settings as the controller node?
In particular, I would check that the following flag is set:

bridge_mappings=eth0:br-eth0

Revision history for this message
Eoghan (eoghank) said : 		#2

Hi Salvatore

I edited the ovs_quantum_plugin.ini file on the second node to include this

[OVS]
bridge_mappings = eth0:br-eth0

Then restarted the agent but I had to manually create br-eth0 before it restarted and when it did start this is what ovs-vsctl looks like on the second node but instances on different nodes still could not ping each other.

38dbbac7-f643-4bdb-8aa1-83f06e612ec4
    Bridge "br-eth0"
        Port "br-eth0"
            Interface "br-eth0"
                type: internal
        Port "phy-br-eth0"
            Interface "phy-br-eth0"
    Bridge br-int
        Port "int-br-eth0"
            Interface "int-br-eth0"
        Port "tap6d40cc67-9b"
            tag: 1
            Interface "tap6d40cc67-9b"
        Port "tap06b6c317-bf"
            tag: 1
            Interface "tap06b6c317-bf"
        Port "tapd59af817-c0"
            tag: 1
            Interface "tapd59af817-c0"
        Port br-int
            Interface br-int
                type: internal
        Port "tap26583155-34"
            tag: 1
            Interface "tap26583155-34"
                type: internal
    ovs_version: "1.4.0+build0"

Revision history for this message
yong sheng gong (gongysh) said : 		#3

u need add physical eth device into br-eth0:
for example ovs-vsctl add-port br-eth0 eth0

Revision history for this message
Eoghan (eoghank) said : 		#4

I added eth0 to br-eth0 on both nodes but instances on nodeA still cannot ping instances on nodeB and vice versa. ovs-vsctl show for br-eth0 is the same on both nodes:

Bridge "br-eth0"
        Port "br-eth0"
            Interface "br-eth0"
                type: internal
        Port "eth0"
            Interface "eth0"
        Port "phy-br-eth0"
            Interface "phy-br-eth0"

When I looked at tcpdump I see the requests with the instance IP going out of int-br-eth0 and out of phy-br-eth0 but they do not appear to get up to eth0. I do see traffic form phy-br-eth0 to etho but nothing that indicates any traffic from the instance IP. Are there any specific paramaters required for setting up the ovs bridges other that what is being used in stack.sh?

Revision history for this message
Eoghan (eoghank) said : 		#5

Thanks for the help Salvator and Yong.

I got the answer to the rest of this problem in this thread https://answers.launchpad.net/quantum/+question/209168

Eoghan