neutron

dhcp reply lost between int-br1 and br-int

Asked by veronesp

Hi,
first of all, here my configuration:
O.S. CentOS 6.4 x86_64 (SELinux disabled)
Grizzly from RDO repos;
Quantum with ovs plugin and vlan (3 vlan id configured at switch level, from 3501 to 3503);
cloudctrl01 acts as keystone, glance, quantum-*, rabbitmq, mysql, nova-* (except network and compute); three interfaces: eth0 - management; eth1 - data; eth2 external;
nova01 acts as nova-compute; two interfaces: eth0 - management, eth1 - data.

As far i understand, the path in the compute node should be:
vm:demo01:eth0 <---> tapafa41705-77 <---> qbrafa41705-77 <---> qvbafa41705-77 <---> qvoafa41705-77 <---> br-int <---> int-br1 <---> phy-br1 <---> br1 <---> eth1

I suppose there something wrong on the compute node (nova01): with tcpdump i saw that dhcp reply arrive at int-br1 but not at br-int.

Additional info:
# uname -r
2.6.32-358.111.1.openstack.el6.x86_64
# rpm -qa |grep openvs
openvswitch-1.10.0-1.el6.x86_64
openstack-quantum-openvswitch-2013.1.2-1.el6.noarch
kmod-openvswitch-1.10.0-1.el6.x86_64

# ip li
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether e0:cb:4e:55:37:c9 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether e0:cb:4e:55:36:79 brd ff:ff:ff:ff:ff:ff
4: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
    link/ether a2:ae:a7:4e:81:6c brd ff:ff:ff:ff:ff:ff
43: br-int: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether 0a:4c:e0:02:1d:4c brd ff:ff:ff:ff:ff:ff
45: br1: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether e0:cb:4e:55:36:79 brd ff:ff:ff:ff:ff:ff
46: phy-br1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether e6:98:6d:71:2b:24 brd ff:ff:ff:ff:ff:ff
47: int-br1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 56:e1:03:b6:07:e5 brd ff:ff:ff:ff:ff:ff
60: qbrafa41705-77: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether 4e:47:72:51:fe:7e brd ff:ff:ff:ff:ff:ff
61: qvoafa41705-77: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether a6:c9:e2:a5:5e:52 brd ff:ff:ff:ff:ff:ff
62: qvbafa41705-77: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 4e:47:72:51:fe:7e brd ff:ff:ff:ff:ff:ff
63: tapafa41705-77: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
    link/ether fe:16:3e:0f:0b:f3 brd ff:ff:ff:ff:ff:ff

# ovs-vsctl show
cef7bfff-248f-438c-82f9-499505f51197
    Bridge br-int
        Port "int-br1"
            Interface "int-br1"
        Port "qvoafa41705-77"
            tag: 2
            Interface "qvoafa41705-77"
        Port br-int
            Interface br-int
                type: internal
    Bridge "br1"
        Port "br1"
            Interface "br1"
                type: internal
        Port "phy-br1"
            Interface "phy-br1"
        Port "eth1"
            Interface "eth1"
    ovs_version: "1.10.0"

# iptables-save -c
# Generated by iptables-save v1.4.7 on Tue Jul 9 09:41:01 2013
*filter
:INPUT ACCEPT [1550565:1487071658]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [1218790:339095333]
:fail2ban-SSH - [0:0]
:quantum-filter-top - [0:0]
:quantum-openvswi-FORWARD - [0:0]
:quantum-openvswi-INPUT - [0:0]
:quantum-openvswi-OUTPUT - [0:0]
:quantum-openvswi-iafa41705-7 - [0:0]
:quantum-openvswi-local - [0:0]
:quantum-openvswi-oafa41705-7 - [0:0]
:quantum-openvswi-sg-chain - [0:0]
:quantum-openvswi-sg-fallback - [0:0]
[1582086:1489030022] -A INPUT -j quantum-openvswi-INPUT
[0:0] -A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
[0:0] -A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
[0:0] -A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
[0:0] -A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
[31521:1958364] -A INPUT -p tcp -m multiport --dports 5900:5999 -m comment --comment "001 nova compute incoming" -j ACCEPT
[0:0] -A INPUT -p gre -j ACCEPT
[13649:906676] -A INPUT -p tcp -m tcp --dport 22 -j fail2ban-SSH
[86:29068] -A FORWARD -j quantum-filter-top
[86:29068] -A FORWARD -j quantum-openvswi-FORWARD
[0:0] -A FORWARD -d 192.168.122.0/24 -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT
[0:0] -A FORWARD -s 192.168.122.0/24 -i virbr0 -j ACCEPT
[0:0] -A FORWARD -i virbr0 -o virbr0 -j ACCEPT
[0:0] -A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
[0:0] -A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable
[1218790:339095333] -A OUTPUT -j quantum-filter-top
[1218790:339095333] -A OUTPUT -j quantum-openvswi-OUTPUT
[13649:906676] -A fail2ban-SSH -j RETURN
[1218876:339124401] -A quantum-filter-top -j quantum-openvswi-local
[43:14534] -A quantum-openvswi-FORWARD -m physdev --physdev-out tapafa41705-77 --physdev-is-bridged -j quantum-openvswi-sg-chain
[43:14534] -A quantum-openvswi-FORWARD -m physdev --physdev-in tapafa41705-77 --physdev-is-bridged -j quantum-openvswi-sg-chain
[0:0] -A quantum-openvswi-INPUT -m physdev --physdev-in tapafa41705-77 --physdev-is-bridged -j quantum-openvswi-oafa41705-7
[0:0] -A quantum-openvswi-iafa41705-7 -m state --state INVALID -j DROP
[0:0] -A quantum-openvswi-iafa41705-7 -m state --state RELATED,ESTABLISHED -j RETURN
[0:0] -A quantum-openvswi-iafa41705-7 -s 10.50.1.3/32 -p udp -m udp --sport 67 --dport 68 -j RETURN
[43:14534] -A quantum-openvswi-iafa41705-7 -j quantum-openvswi-sg-fallback
[0:0] -A quantum-openvswi-oafa41705-7 -m mac ! --mac-source FA:16:3E:0F:0B:F3 -j DROP
[43:14534] -A quantum-openvswi-oafa41705-7 -p udp -m udp --sport 68 --dport 67 -j RETURN
[0:0] -A quantum-openvswi-oafa41705-7 ! -s 10.50.1.2/32 -j DROP
[0:0] -A quantum-openvswi-oafa41705-7 -p udp -m udp --sport 67 --dport 68 -j DROP
[0:0] -A quantum-openvswi-oafa41705-7 -m state --state INVALID -j DROP
[0:0] -A quantum-openvswi-oafa41705-7 -m state --state RELATED,ESTABLISHED -j RETURN
[0:0] -A quantum-openvswi-oafa41705-7 -j RETURN
[0:0] -A quantum-openvswi-oafa41705-7 -j quantum-openvswi-sg-fallback
[43:14534] -A quantum-openvswi-sg-chain -m physdev --physdev-out tapafa41705-77 --physdev-is-bridged -j quantum-openvswi-iafa41705-7
[43:14534] -A quantum-openvswi-sg-chain -m physdev --physdev-in tapafa41705-77 --physdev-is-bridged -j quantum-openvswi-oafa41705-7
[43:14534] -A quantum-openvswi-sg-chain -j ACCEPT
[43:14534] -A quantum-openvswi-sg-fallback -j DROP
COMMIT
# Completed on Tue Jul 9 09:41:01 2013
# Generated by iptables-save v1.4.7 on Tue Jul 9 09:41:01 2013
*mangle
:PREROUTING ACCEPT [8663792:7351594978]
:INPUT ACCEPT [7791107:7256827150]
:FORWARD ACCEPT [294:97842]
:OUTPUT ACCEPT [6081439:1698102046]
:POSTROUTING ACCEPT [6084802:1698250054]
[0:0] -A POSTROUTING -o virbr0 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill
[0:0] -A POSTROUTING -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill
COMMIT
# Completed on Tue Jul 9 09:41:01 2013
# Generated by iptables-save v1.4.7 on Tue Jul 9 09:41:01 2013
*nat
:PREROUTING ACCEPT [168879:18761154]
:POSTROUTING ACCEPT [10339:1134658]
:OUTPUT ACCEPT [10336:1133644]
:quantum-openvswi-OUTPUT - [0:0]
:quantum-openvswi-POSTROUTING - [0:0]
:quantum-openvswi-PREROUTING - [0:0]
:quantum-openvswi-float-snat - [0:0]
:quantum-openvswi-snat - [0:0]
:quantum-postrouting-bottom - [0:0]
[168879:18761154] -A PREROUTING -j quantum-openvswi-PREROUTING
[10339:1134658] -A POSTROUTING -j quantum-openvswi-POSTROUTING
[10339:1134658] -A POSTROUTING -j quantum-postrouting-bottom
[0:0] -A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
[0:0] -A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
[0:0] -A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE
[10336:1133644] -A OUTPUT -j quantum-openvswi-OUTPUT
[10339:1134658] -A quantum-openvswi-snat -j quantum-openvswi-float-snat
[10339:1134658] -A quantum-postrouting-bottom -j quantum-openvswi-snat
COMMIT
# Completed on Tue Jul 9 09:41:01 2013

Thanks in advance and kind regards,
  Paolo

Question information

Language:
English Edit question
Status:
Solved
For:
neutron Edit question
Assignee:
No assignee Edit question
Solved by:
veronesp
Solved:
Last query:
Last reply:
Revision history for this message
veronesp (paolo-veronesi) said : 		#1

I forgot to put the pluing.ini:
# cat /etc/quantum/plugin.ini |grep -v '#'
[DATABASE]
sql_connection = hidden
sql_max_retries = 10
reconnect_interval = 2

[OVS]
tenant_network_type=vlan
network_vlan_ranges=physnet1:3501:3503
integration_bridge=br-int
bridge_mappings=physnet1:br1
enable_tunneling=False

[AGENT]
polling_interval = 2

[SECURITYGROUP]
firewall_driver = quantum.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver

Revision history for this message
veronesp (paolo-veronesi) said : 		#2

I manually assigned the ip to the vm (its mac address is fa:16:3e:0f:0b:f3):
ifconfig eth0 10.50.1.3 netmask 255.255.255.0 up

And i tried to ping the dnsmasq ip
ping -c 1 10.50.1.3 (dnsmasq ip)

I have the same behaviour: replies get lost between int-br1 -> br-int on the compute node:

tcpdump -nne -i br-int ether host fa:16:3e:0f:0b:f3
10:55:31.618919 fa:16:3e:0f:0b:f3 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 1, p 0, ethertype ARP, Request who-has 10.50.1.3 tell 10.50.1.2, length 28
10:55:32.618834 fa:16:3e:0f:0b:f3 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 1, p 0, ethertype ARP, Request who-has 10.50.1.3 tell 10.50.1.2, length 28
10:55:33.618751 fa:16:3e:0f:0b:f3 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 1, p 0, ethertype ARP, Request who-has 10.50.1.3 tell 10.50.1.2, length 28

tcpdump -nne -i int-br1 ether host fa:16:3e:0f:0b:f3
10:55:31.618922 fa:16:3e:0f:0b:f3 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 1, p 0, ethertype ARP, Request who-has 10.50.1.3 tell 10.50.1.2, length 28
10:55:31.619411 fa:16:3e:64:b6:fb > fa:16:3e:0f:0b:f3, ethertype ARP (0x0806), length 60: Reply 10.50.1.3 is-at fa:16:3e:64:b6:fb, length 46
10:55:32.618868 fa:16:3e:0f:0b:f3 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 1, p 0, ethertype ARP, Request who-has 10.50.1.3 tell 10.50.1.2, length 28
10:55:32.619128 fa:16:3e:64:b6:fb > fa:16:3e:0f:0b:f3, ethertype ARP (0x0806), length 60: Reply 10.50.1.3 is-at fa:16:3e:64:b6:fb, length 46
10:55:33.618780 fa:16:3e:0f:0b:f3 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 1, p 0, ethertype ARP, Request who-has 10.50.1.3 tell 10.50.1.2, length 28
10:55:33.619054 fa:16:3e:64:b6:fb > fa:16:3e:0f:0b:f3, ethertype ARP (0x0806), length 60: Reply 10.50.1.3 is-at fa:16:3e:64:b6:fb, length 46

Revision history for this message
veronesp (paolo-veronesi) said : 		#3

Hi,
problem solved. There was an issue in the configuration of two switches between cloudctrl01 and nova-compute.

We have two DELL switches, but different model and o.s. Their configuration was the same:
switchport mode general
switchport general allowed vlan add 3501-3510

This configuration works with "PowerConnect 7048R-RA, 4.2.1.3, VxWorks 6.6" (!System Software Version 4.2.1.3), but not on "PowerConnect 6224, 3.3.5.5, VxWorks 6.5" (!System Software Version 3.3.5.5). The right configuration for the PowerConnect 6224 is the following:
switchport mode trunk
switchport trunk allowed vlan 3501-3510

kind regards