neutron

Also Nova+Quantum+Openvswitch VLAN Problem!

Asked by Havent

My environment includes two physical hosts.
One of them (domain name cc201) installed all of nova components and Glance, Quantum, Keystone, Horizon, Open-vSwitch as controller and network node; the other installed only nova-compute, Quantum, Open-vSwitch as compute node.

also I run quantum-agent, load 8021q module.etc
Everything seems runs well.
But I found a curious problem!

On cc201, I create networks 192.168.153.0/24(network3, its vlan ID is 6)、192.168.155.0/24(network5, its vlan ID is 8)
I run nova-manage on host cc201 to create VMs.
the VMs are:
192.168.153.2 (on host cc201)
192.168.153.4 (on host cc202)
192.168.153.5 (on host cc202)
192.168.153.6 (on host cc202)
192.168.155.2 (on host cc201)
192.168.155.3 (on host cc201)
192.168.155.4 (on host cc202)

I log on one of them to ping another of them and capture packets through eth1(eth1 interface the openvswitch port on both of my hosts cc201 and cc202, by using command ovs-vsctl add-port eth1 br-int), results are as below:

I run tcpdump -i eth1 -v -w to capture and save packets
when I log on 192.168.153.2 (on cc201, in vlan6):
ping 192.168.153.4 (on cc202, in vlan6): (result is they connected)
on eth1 of cc201:I can see vlan 6 tag in ICMP request and reply, it is what I expected
on eth1 of cc202:I can see vlan 6 tag in ICMP request and reply, it is what I expected

when I log on 192.168.153.2 (on cc201, in vlan 6):
ping 192.168.155.2 (on cc201, in vlan8): (result is they are not connected)
this result is also what I expect

But when I log on 192.168.153.2 (on cc201, in vlan 6)
ping 192.168.155.4 (on cc202, in vlan 8): (result is they connected!!!)
on eth1 of cc201: I cannot see vlan 6 tag in ICMP, instead, I can see vlan 8 tag in ICMP request and reply!!!
on eth1 of cc202: I cannot see vlan 6 tag in ICMP, instead, I can see vlan 8 tag in ICMP request and reply!!!
another words, it has the wrong vlan tag! (expect 6 but actually 8!)

so what happened ? Is this a known bug of quantum or openvswitch?

more details is as below:
1)nova config on cc201: nova.conf--http://paste.openstack.org/show/18588/
nova-compute.conf--http://paste.openstack.org/show/18589/
2)nova config on cc202: nova.conf--http://paste.openstack.org/show/18590/
nova-compute.conf--http://paste.openstack.org/show/18591/
3)other command results on cc201--http://paste.openstack.org/show/18592/
4)other command results on cc202--http://paste.openstack.org/show/18593/

Question information

Language:
English Edit question
Status:
Solved
For:
neutron Edit question
Assignee:
No assignee Edit question
Solved by:
mizumoto
Solved:
Last query:
Last reply:
Revision history for this message
Havent (guestly) said : 		#1

output of the second capture of packets (ping from 192.168.153.2 in vlan 6 on cc201 to 192.168.155.4 in vlan 8 on cc202):

in_port(7),eth(src=fa:16:3e:1a:95:5b,dst=fa:16:3e:7f:6e:64),eth_type(0x0800),ipv4(src=192.168.153.2,dst=192.168.155.4,proto=1,tos=0,ttl=64,frag=no),icmp(type=8,code=0), packets:20, bytes:1960, used:0.192s, actions:6
in_port(1),eth(src=24:b6:57:f3:56:0b,dst=01:00:0c:cc:cc:cd),eth_type(0x8100),vlan(vid=26,pcp=7),encap(), packets:4549, bytes:291136, used:1.396s, actions:0
in_port(1),eth(src=54:52:48:00:08:13,dst=33:33:ff:00:08:13),eth_type(0x86dd),ipv6(src=fe80::5652:48ff:fe00:813,dst=ff02::1:ff00:813,label=0,proto=58,tclass=0,hlimit=1,frag=no),icmpv6(type=131,code=0), packets:0, bytes:0, used:never, actions:0
in_port(19),eth(src=fa:16:3e:22:d7:40,dst=fa:16:3e:1b:cf:e2),eth_type(0x0800),ipv4(src=192.168.153.2,dst=192.168.155.4,proto=1,tos=0,ttl=63,frag=no),icmp(type=8,code=0), packets:20, bytes:1960, used:0.192s, actions:push_vlan(vid=8,pcp=0),1

It says that operation push_vlan put the packets with tag 8 not tag 6!!
I suppose it is the reason why two VMs from different vlan can ping connected!

But why???

Revision history for this message
Best mizumoto (mizumotoda) said : 		#2

Hi,

It seems to be connected through the default gateway of VM:192.168.153.2, that is, gw-xxx of host cc201 interface. It should be 192.168.153.1 and this gw-xxx interface forward to the other gw-yyy of host cc201 interface internally. it's address should be 192.168.155.1 for vlan ID 8.
Then, br-int of cc201 send out from eth1 of cc201 to eth1 of cc202 with vlan ID 8 because its broadcast domain were 192.168.155.x.
The br-int of cc201 just forward to tap device of VM:192.168.155.4.

I think it was normal behaviour.
Please check each VM's routing table, and cc201/cc202 ifconfig, and the output of ovs-vsctl show.

mizumoto

Revision history for this message
Havent (guestly) said : 		#3

Thanks to mizumoto for your reply?
But why I ping from 192.168.153.2(on cc201, vlan 6) to 192.168.155.2(on cc201, vlan 8), it is not connected?
Is it because they are on the same physical host?
If so, how can I make VLAN work? What should I do?

Because I the image I use is simple tty-linux, so the information may be not detail. If you need more informations, just tell me!
Routing tables of VMs are as below:
1) on 192.168.153.2:
$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.153.0 * 255.255.255.0 U 0 0 0 eth0
default 192.168.153.1 0.0.0.0 UG 0 0 0 eth0

2)on 192.168.155.4:
$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.155.0 * 255.255.255.0 U 0 0 0 eth0
default 192.168.155.1 0.0.0.0 UG 0 0 0 eth0

3) cc201 ovs-vsctl:
# ovs-vsctl show
974585a6-2b5b-4bea-be4a-3127f71007bf
    Bridge vlan-test-br
        Port vlan-test-br
            Interface vlan-test-br
                type: internal
    Bridge br-int
        Port "gw-87a6f352-35"
            tag: 4
            Interface "gw-87a6f352-35"
                type: internal
        Port "gw-e7863e30-1c"
            tag: 6
            Interface "gw-e7863e30-1c"
                type: internal
        Port "tap908304d1-d1"
            tag: 8
            Interface "tap908304d1-d1"
        Port "gw-6d53af57-d9"
            tag: 8
            Interface "gw-6d53af57-d9"
                type: internal
        Port "tap1585a714-10"
            tag: 6
            Interface "tap1585a714-10"
        Port br-int
            Interface br-int
                type: internal
        Port "eth1"
            Interface "eth1"
        Port "gw-3faffe59-5d"
            tag: 5
            Interface "gw-3faffe59-5d"
                type: internal
        Port "tap09ae4556-2b"
            tag: 8
            Interface "tap09ae4556-2b"
        Port "tap77918099-5a"
            tag: 6
            Interface "tap77918099-5a"
        Port "tap6011d326-33"
            tag: 6
            Interface "tap6011d326-33"
        Port "gw-9a2757f4-2d"
            tag: 7
            Interface "gw-9a2757f4-2d"
                type: internal
    ovs_version: "1.4.0+build0"

4) cc201 ifconfig:
# ifconfig
br-int Link encap:Ethernet HWaddr 00:1b:21:c6:35:a6
          inet6 addr: fe80::21b:21ff:fec6:35a6/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:10524930 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9110 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:714364025 (714.3 MB) TX bytes:783412 (783.4 KB)

eth0 Link encap:Ethernet HWaddr d0:67:e5:23:96:d4
          inet addr:10.131.0.31 Bcast:10.131.255.255 Mask:255.252.0.0
          inet6 addr: fe80::d267:e5ff:fe23:96d4/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:3473405 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2372097 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:694315585 (694.3 MB) TX bytes:1224320432 (1.2 GB)
          Interrupt:43 Base address:0x6000

eth1 Link encap:Ethernet HWaddr 00:1b:21:c6:35:a6
          inet6 addr: fe80::21b:21ff:fec6:35a6/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
          RX packets:10906571 errors:0 dropped:0 overruns:0 frame:0
          TX packets:99047 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:814630415 (814.6 MB) TX bytes:21598782 (21.5 MB)
          Interrupt:18 Memory:e10c0000-e10e0000

eth1.25 Link encap:Ethernet HWaddr 00:1b:21:c6:35:a6
          inet addr:192.168.200.10 Bcast:192.168.200.255 Mask:255.255.255.0
          inet6 addr: fe80::21b:21ff:fec6:35a6/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:95905 errors:0 dropped:0 overruns:0 frame:0
          TX packets:138 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:4799632 (4.7 MB) TX bytes:13124 (13.1 KB)

gw-3faffe59-5d Link encap:Ethernet HWaddr fa:16:3e:34:03:72
          inet addr:192.168.152.1 Bcast:192.168.152.255 Mask:255.255.255.0
          inet6 addr: fe80::f816:3eff:fe34:372/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
          RX packets:231581 errors:0 dropped:0 overruns:0 frame:0
          TX packets:11756 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:17957763 (17.9 MB) TX bytes:2282431 (2.2 MB)

gw-6d53af57-d9 Link encap:Ethernet HWaddr fa:16:3e:22:d7:40
          inet addr:192.168.155.1 Bcast:192.168.155.255 Mask:255.255.255.0
          inet6 addr: fe80::f816:3eff:fe22:d740/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
          RX packets:80548 errors:0 dropped:0 overruns:0 frame:0
          TX packets:19101 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:8413833 (8.4 MB) TX bytes:2921376 (2.9 MB)

gw-87a6f352-35 Link encap:Ethernet HWaddr fa:16:3e:44:9c:9b
          inet addr:192.168.151.1 Bcast:192.168.151.255 Mask:255.255.255.0
          inet6 addr: fe80::f816:3eff:fe44:9c9b/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
          RX packets:268056 errors:0 dropped:0 overruns:0 frame:0
          TX packets:35932 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:26677418 (26.6 MB) TX bytes:6863883 (6.8 MB)

gw-9a2757f4-2d Link encap:Ethernet HWaddr fa:16:3e:16:12:4e
          inet addr:192.168.154.1 Bcast:192.168.154.255 Mask:255.255.255.0
          inet6 addr: fe80::f816:3eff:fe16:124e/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
          RX packets:84896 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1275 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:5781594 (5.7 MB) TX bytes:227757 (227.7 KB)

gw-e7863e30-1c Link encap:Ethernet HWaddr fa:16:3e:7f:6e:64
          inet addr:192.168.153.1 Bcast:192.168.153.255 Mask:255.255.255.0
          inet6 addr: fe80::f816:3eff:fe7f:6e64/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
          RX packets:394653 errors:0 dropped:0 overruns:0 frame:0
          TX packets:78912 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:43450135 (43.4 MB) TX bytes:11888884 (11.8 MB)

lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:16436 Metric:1
          RX packets:12419775 errors:0 dropped:0 overruns:0 frame:0
          TX packets:12419775 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:5098372204 (5.0 GB) TX bytes:5098372204 (5.0 GB)

tap09ae4556-2b Link encap:Ethernet HWaddr ca:95:1e:48:33:89
          inet6 addr: fe80::c895:1eff:fe48:3389/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:5387 errors:0 dropped:0 overruns:0 frame:0
          TX packets:60301 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:1323149 (1.3 MB) TX bytes:5239956 (5.2 MB)

tap1585a714-10 Link encap:Ethernet HWaddr 86:fc:eb:6b:36:9e
          inet6 addr: fe80::84fc:ebff:fe6b:369e/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:129863 errors:0 dropped:0 overruns:0 frame:0
          TX packets:274698 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:16792199 (16.7 MB) TX bytes:24268329 (24.2 MB)

tap6011d326-33 Link encap:Ethernet HWaddr d2:56:1e:77:52:21
          inet6 addr: fe80::d056:1eff:fe77:5221/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:5143 errors:0 dropped:0 overruns:0 frame:0
          TX packets:60777 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:1196755 (1.1 MB) TX bytes:7073594 (7.0 MB)

tap77918099-5a Link encap:Ethernet HWaddr be:b5:63:97:aa:73
          inet6 addr: fe80::bcb5:63ff:fe97:aa73/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:4697 errors:0 dropped:0 overruns:0 frame:0
          TX packets:60320 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:1154503 (1.1 MB) TX bytes:7029312 (7.0 MB)

tap908304d1-d1 Link encap:Ethernet HWaddr 42:e7:2f:38:21:91
          inet6 addr: fe80::40e7:2fff:fe38:2191/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:5408 errors:0 dropped:0 overruns:0 frame:0
          TX packets:63295 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:1328382 (1.3 MB) TX bytes:5450809 (5.4 MB)

virbr0 Link encap:Ethernet HWaddr ee:96:fd:10:c7:53
          inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0
          UP BROADCAST MULTICAST MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

5) cc201 route:
# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default ccs00.cloud.dhc 0.0.0.0 UG 100 0 0 eth0
10.128.0.0 * 255.252.0.0 U 0 0 0 eth0
192.168.122.0 * 255.255.255.0 U 0 0 0 virbr0
192.168.151.0 * 255.255.255.0 U 0 0 0 gw-87a6f352-35
192.168.152.0 * 255.255.255.0 U 0 0 0 gw-3faffe59-5d
192.168.153.0 * 255.255.255.0 U 0 0 0 gw-e7863e30-1c
192.168.154.0 * 255.255.255.0 U 0 0 0 gw-9a2757f4-2d
192.168.155.0 * 255.255.255.0 U 0 0 0 gw-6d53af57-d9
192.168.200.0 * 255.255.255.0 U 0 0 0 eth1.25

6) cc201 iptables-save:
http://paste.openstack.org/show/18631/

7) cc202 ovs-vsctl:
# ovs-vsctl show
ffa556ea-e355-4bf1-b652-fb0b6d7120b7
    Bridge br-int
        Port br-int
            Interface br-int
                type: internal
        Port "eth1"
            Interface "eth1"
        Port "tap13bcaa60-3a"
            tag: 6
            Interface "tap13bcaa60-3a"
        Port "tape682c82f-0c"
            tag: 6
            Interface "tape682c82f-0c"
        Port "tapb697f556-23"
            tag: 6
            Interface "tapb697f556-23"
        Port "tapc636bfdf-a1"
            tag: 6
            Interface "tapc636bfdf-a1"
        Port "tap3f1e7fe8-5d"
            tag: 8
            Interface "tap3f1e7fe8-5d"
    ovs_version: "1.4.0+build0"

8) cc202 ifconfig:
# ifconfig
br-int Link encap:Ethernet HWaddr 00:1b:21:c6:35:b4
          inet6 addr: fe80::21b:21ff:fec6:35b4/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:4785437 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2749 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:339951419 (339.9 MB) TX bytes:236366 (236.3 KB)

eth0 Link encap:Ethernet HWaddr d0:67:e5:23:96:ab
          inet addr:10.131.0.244 Bcast:10.131.255.255 Mask:255.252.0.0
          inet6 addr: fe80::d267:e5ff:fe23:96ab/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:2553715 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2636818 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1160731658 (1.1 GB) TX bytes:604610655 (604.6 MB)
          Interrupt:40 Base address:0x4000

eth1 Link encap:Ethernet HWaddr 00:1b:21:c6:35:b4
          inet6 addr: fe80::21b:21ff:fec6:35b4/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
          RX packets:4969577 errors:0 dropped:0 overruns:0 frame:0
          TX packets:68991 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:371072963 (371.0 MB) TX bytes:14618515 (14.6 MB)
          Interrupt:18 Memory:e10c0000-e10e0000

eth1.25 Link encap:Ethernet HWaddr 00:1b:21:c6:35:b4
          inet addr:192.168.200.11 Bcast:192.168.200.255 Mask:255.255.255.0
          inet6 addr: fe80::21b:21ff:fec6:35b4/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:95884 errors:0 dropped:0 overruns:0 frame:0
          TX packets:138 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:4798498 (4.7 MB) TX bytes:13124 (13.1 KB)

lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:16436 Metric:1
          RX packets:1116 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1116 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:327986 (327.9 KB) TX bytes:327986 (327.9 KB)

tap13bcaa60-3a Link encap:Ethernet HWaddr 2a:b0:9e:55:45:8b
          inet6 addr: fe80::28b0:9eff:fe55:458b/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:4847 errors:0 dropped:0 overruns:0 frame:0
          TX packets:60892 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:1172829 (1.1 MB) TX bytes:7117880 (7.1 MB)

tap3f1e7fe8-5d Link encap:Ethernet HWaddr 1e:ef:54:90:3b:b7
          inet6 addr: fe80::1cef:54ff:fe90:3bb7/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:13508 errors:0 dropped:0 overruns:0 frame:0
          TX packets:68710 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:2118307 (2.1 MB) TX bytes:6078996 (6.0 MB)

tapb697f556-23 Link encap:Ethernet HWaddr a2:da:d6:7d:1c:63
          inet6 addr: fe80::a0da:d6ff:fe7d:1c63/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:33911 errors:0 dropped:0 overruns:0 frame:0
          TX packets:170700 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:10107665 (10.1 MB) TX bytes:28282735 (28.2 MB)

tapc636bfdf-a1 Link encap:Ethernet HWaddr 3a:04:ff:4c:5f:c7
          inet6 addr: fe80::3804:ffff:fe4c:5fc7/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:36931 errors:0 dropped:0 overruns:0 frame:0
          TX packets:173664 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:10444839 (10.4 MB) TX bytes:28618769 (28.6 MB)

tape682c82f-0c Link encap:Ethernet HWaddr 76:eb:44:9e:25:36
          inet6 addr: fe80::74eb:44ff:fe9e:2536/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:46940 errors:0 dropped:0 overruns:0 frame:0
          TX packets:180542 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:12607864 (12.6 MB) TX bytes:29152040 (29.1 MB)

9) cc202 iptables-save:
http://paste.openstack.org/show/18632/

Revision history for this message
Havent (guestly) said : 		#4

Thanks to mizumoto for your reply?
But why I ping from 192.168.153.2(on cc201, vlan 6) to 192.168.155.2(on cc201, vlan 8), it is not connected?
Is it because they are on the same physical host?
If so, how can I make VLAN work? What should I do?

Because I the image I use is simple tty-linux, so the information may be not detail. If you need more informations, just tell me!
Routing tables of VMs are as below:
1) on 192.168.153.2:
$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.153.0 * 255.255.255.0 U 0 0 0 eth0
default 192.168.153.1 0.0.0.0 UG 0 0 0 eth0

2)on 192.168.155.4:
$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.155.0 * 255.255.255.0 U 0 0 0 eth0
default 192.168.155.1 0.0.0.0 UG 0 0 0 eth0

3) cc201 ovs-vsctl:
# ovs-vsctl show
974585a6-2b5b-4bea-be4a-3127f71007bf
    Bridge vlan-test-br
        Port vlan-test-br
            Interface vlan-test-br
                type: internal
    Bridge br-int
        Port "gw-87a6f352-35"
            tag: 4
            Interface "gw-87a6f352-35"
                type: internal
        Port "gw-e7863e30-1c"
            tag: 6
            Interface "gw-e7863e30-1c"
                type: internal
        Port "tap908304d1-d1"
            tag: 8
            Interface "tap908304d1-d1"
        Port "gw-6d53af57-d9"
            tag: 8
            Interface "gw-6d53af57-d9"
                type: internal
        Port "tap1585a714-10"
            tag: 6
            Interface "tap1585a714-10"
        Port br-int
            Interface br-int
                type: internal
        Port "eth1"
            Interface "eth1"
        Port "gw-3faffe59-5d"
            tag: 5
            Interface "gw-3faffe59-5d"
                type: internal
        Port "tap09ae4556-2b"
            tag: 8
            Interface "tap09ae4556-2b"
        Port "tap77918099-5a"
            tag: 6
            Interface "tap77918099-5a"
        Port "tap6011d326-33"
            tag: 6
            Interface "tap6011d326-33"
        Port "gw-9a2757f4-2d"
            tag: 7
            Interface "gw-9a2757f4-2d"
                type: internal
    ovs_version: "1.4.0+build0"

4) cc201 ifconfig:
# ifconfig
br-int Link encap:Ethernet HWaddr 00:1b:21:c6:35:a6
          inet6 addr: fe80::21b:21ff:fec6:35a6/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:10524930 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9110 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:714364025 (714.3 MB) TX bytes:783412 (783.4 KB)

eth0 Link encap:Ethernet HWaddr d0:67:e5:23:96:d4
          inet addr:10.131.0.31 Bcast:10.131.255.255 Mask:255.252.0.0
          inet6 addr: fe80::d267:e5ff:fe23:96d4/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:3473405 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2372097 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:694315585 (694.3 MB) TX bytes:1224320432 (1.2 GB)
          Interrupt:43 Base address:0x6000

eth1 Link encap:Ethernet HWaddr 00:1b:21:c6:35:a6
          inet6 addr: fe80::21b:21ff:fec6:35a6/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
          RX packets:10906571 errors:0 dropped:0 overruns:0 frame:0
          TX packets:99047 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:814630415 (814.6 MB) TX bytes:21598782 (21.5 MB)
          Interrupt:18 Memory:e10c0000-e10e0000

eth1.25 Link encap:Ethernet HWaddr 00:1b:21:c6:35:a6
          inet addr:192.168.200.10 Bcast:192.168.200.255 Mask:255.255.255.0
          inet6 addr: fe80::21b:21ff:fec6:35a6/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:95905 errors:0 dropped:0 overruns:0 frame:0
          TX packets:138 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:4799632 (4.7 MB) TX bytes:13124 (13.1 KB)

gw-3faffe59-5d Link encap:Ethernet HWaddr fa:16:3e:34:03:72
          inet addr:192.168.152.1 Bcast:192.168.152.255 Mask:255.255.255.0
          inet6 addr: fe80::f816:3eff:fe34:372/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
          RX packets:231581 errors:0 dropped:0 overruns:0 frame:0
          TX packets:11756 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:17957763 (17.9 MB) TX bytes:2282431 (2.2 MB)

gw-6d53af57-d9 Link encap:Ethernet HWaddr fa:16:3e:22:d7:40
          inet addr:192.168.155.1 Bcast:192.168.155.255 Mask:255.255.255.0
          inet6 addr: fe80::f816:3eff:fe22:d740/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
          RX packets:80548 errors:0 dropped:0 overruns:0 frame:0
          TX packets:19101 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:8413833 (8.4 MB) TX bytes:2921376 (2.9 MB)

gw-87a6f352-35 Link encap:Ethernet HWaddr fa:16:3e:44:9c:9b
          inet addr:192.168.151.1 Bcast:192.168.151.255 Mask:255.255.255.0
          inet6 addr: fe80::f816:3eff:fe44:9c9b/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
          RX packets:268056 errors:0 dropped:0 overruns:0 frame:0
          TX packets:35932 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:26677418 (26.6 MB) TX bytes:6863883 (6.8 MB)

gw-9a2757f4-2d Link encap:Ethernet HWaddr fa:16:3e:16:12:4e
          inet addr:192.168.154.1 Bcast:192.168.154.255 Mask:255.255.255.0
          inet6 addr: fe80::f816:3eff:fe16:124e/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
          RX packets:84896 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1275 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:5781594 (5.7 MB) TX bytes:227757 (227.7 KB)

gw-e7863e30-1c Link encap:Ethernet HWaddr fa:16:3e:7f:6e:64
          inet addr:192.168.153.1 Bcast:192.168.153.255 Mask:255.255.255.0
          inet6 addr: fe80::f816:3eff:fe7f:6e64/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
          RX packets:394653 errors:0 dropped:0 overruns:0 frame:0
          TX packets:78912 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:43450135 (43.4 MB) TX bytes:11888884 (11.8 MB)

lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:16436 Metric:1
          RX packets:12419775 errors:0 dropped:0 overruns:0 frame:0
          TX packets:12419775 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:5098372204 (5.0 GB) TX bytes:5098372204 (5.0 GB)

tap09ae4556-2b Link encap:Ethernet HWaddr ca:95:1e:48:33:89
          inet6 addr: fe80::c895:1eff:fe48:3389/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:5387 errors:0 dropped:0 overruns:0 frame:0
          TX packets:60301 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:1323149 (1.3 MB) TX bytes:5239956 (5.2 MB)

tap1585a714-10 Link encap:Ethernet HWaddr 86:fc:eb:6b:36:9e
          inet6 addr: fe80::84fc:ebff:fe6b:369e/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:129863 errors:0 dropped:0 overruns:0 frame:0
          TX packets:274698 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:16792199 (16.7 MB) TX bytes:24268329 (24.2 MB)

tap6011d326-33 Link encap:Ethernet HWaddr d2:56:1e:77:52:21
          inet6 addr: fe80::d056:1eff:fe77:5221/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:5143 errors:0 dropped:0 overruns:0 frame:0
          TX packets:60777 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:1196755 (1.1 MB) TX bytes:7073594 (7.0 MB)

tap77918099-5a Link encap:Ethernet HWaddr be:b5:63:97:aa:73
          inet6 addr: fe80::bcb5:63ff:fe97:aa73/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:4697 errors:0 dropped:0 overruns:0 frame:0
          TX packets:60320 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:1154503 (1.1 MB) TX bytes:7029312 (7.0 MB)

tap908304d1-d1 Link encap:Ethernet HWaddr 42:e7:2f:38:21:91
          inet6 addr: fe80::40e7:2fff:fe38:2191/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:5408 errors:0 dropped:0 overruns:0 frame:0
          TX packets:63295 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:1328382 (1.3 MB) TX bytes:5450809 (5.4 MB)

virbr0 Link encap:Ethernet HWaddr ee:96:fd:10:c7:53
          inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0
          UP BROADCAST MULTICAST MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

5) cc201 route:
# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default ccs00.cloud.dhc 0.0.0.0 UG 100 0 0 eth0
10.128.0.0 * 255.252.0.0 U 0 0 0 eth0
192.168.122.0 * 255.255.255.0 U 0 0 0 virbr0
192.168.151.0 * 255.255.255.0 U 0 0 0 gw-87a6f352-35
192.168.152.0 * 255.255.255.0 U 0 0 0 gw-3faffe59-5d
192.168.153.0 * 255.255.255.0 U 0 0 0 gw-e7863e30-1c
192.168.154.0 * 255.255.255.0 U 0 0 0 gw-9a2757f4-2d
192.168.155.0 * 255.255.255.0 U 0 0 0 gw-6d53af57-d9
192.168.200.0 * 255.255.255.0 U 0 0 0 eth1.25

6) cc201 iptables-save:
http://paste.openstack.org/show/18631/

7) cc202 ovs-vsctl:
# ovs-vsctl show
ffa556ea-e355-4bf1-b652-fb0b6d7120b7
    Bridge br-int
        Port br-int
            Interface br-int
                type: internal
        Port "eth1"
            Interface "eth1"
        Port "tap13bcaa60-3a"
            tag: 6
            Interface "tap13bcaa60-3a"
        Port "tape682c82f-0c"
            tag: 6
            Interface "tape682c82f-0c"
        Port "tapb697f556-23"
            tag: 6
            Interface "tapb697f556-23"
        Port "tapc636bfdf-a1"
            tag: 6
            Interface "tapc636bfdf-a1"
        Port "tap3f1e7fe8-5d"
            tag: 8
            Interface "tap3f1e7fe8-5d"
    ovs_version: "1.4.0+build0"

8) cc202 ifconfig:
# ifconfig
br-int Link encap:Ethernet HWaddr 00:1b:21:c6:35:b4
          inet6 addr: fe80::21b:21ff:fec6:35b4/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:4785437 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2749 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:339951419 (339.9 MB) TX bytes:236366 (236.3 KB)

eth0 Link encap:Ethernet HWaddr d0:67:e5:23:96:ab
          inet addr:10.131.0.244 Bcast:10.131.255.255 Mask:255.252.0.0
          inet6 addr: fe80::d267:e5ff:fe23:96ab/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:2553715 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2636818 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1160731658 (1.1 GB) TX bytes:604610655 (604.6 MB)
          Interrupt:40 Base address:0x4000

eth1 Link encap:Ethernet HWaddr 00:1b:21:c6:35:b4
          inet6 addr: fe80::21b:21ff:fec6:35b4/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
          RX packets:4969577 errors:0 dropped:0 overruns:0 frame:0
          TX packets:68991 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:371072963 (371.0 MB) TX bytes:14618515 (14.6 MB)
          Interrupt:18 Memory:e10c0000-e10e0000

eth1.25 Link encap:Ethernet HWaddr 00:1b:21:c6:35:b4
          inet addr:192.168.200.11 Bcast:192.168.200.255 Mask:255.255.255.0
          inet6 addr: fe80::21b:21ff:fec6:35b4/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:95884 errors:0 dropped:0 overruns:0 frame:0
          TX packets:138 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:4798498 (4.7 MB) TX bytes:13124 (13.1 KB)

lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:16436 Metric:1
          RX packets:1116 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1116 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:327986 (327.9 KB) TX bytes:327986 (327.9 KB)

tap13bcaa60-3a Link encap:Ethernet HWaddr 2a:b0:9e:55:45:8b
          inet6 addr: fe80::28b0:9eff:fe55:458b/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:4847 errors:0 dropped:0 overruns:0 frame:0
          TX packets:60892 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:1172829 (1.1 MB) TX bytes:7117880 (7.1 MB)

tap3f1e7fe8-5d Link encap:Ethernet HWaddr 1e:ef:54:90:3b:b7
          inet6 addr: fe80::1cef:54ff:fe90:3bb7/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:13508 errors:0 dropped:0 overruns:0 frame:0
          TX packets:68710 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:2118307 (2.1 MB) TX bytes:6078996 (6.0 MB)

tapb697f556-23 Link encap:Ethernet HWaddr a2:da:d6:7d:1c:63
          inet6 addr: fe80::a0da:d6ff:fe7d:1c63/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:33911 errors:0 dropped:0 overruns:0 frame:0
          TX packets:170700 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:10107665 (10.1 MB) TX bytes:28282735 (28.2 MB)

tapc636bfdf-a1 Link encap:Ethernet HWaddr 3a:04:ff:4c:5f:c7
          inet6 addr: fe80::3804:ffff:fe4c:5fc7/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:36931 errors:0 dropped:0 overruns:0 frame:0
          TX packets:173664 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:10444839 (10.4 MB) TX bytes:28618769 (28.6 MB)

tape682c82f-0c Link encap:Ethernet HWaddr 76:eb:44:9e:25:36
          inet6 addr: fe80::74eb:44ff:fe9e:2536/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:46940 errors:0 dropped:0 overruns:0 frame:0
          TX packets:180542 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:12607864 (12.6 MB) TX bytes:29152040 (29.1 MB)

9) cc202 iptables-save:
http://paste.openstack.org/show/18632/

Revision history for this message
mizumoto (mizumotoda) said : 		#5

Let me clear your information. You mentioned 3VMs and just 1 VM on VLAN tag: 6, i.e. 192.168.153.2, on cc201 but,

> I run nova-manage on host cc201 to create VMs.
> the VMs are:
> 192.168.153.2 (on host cc201)
> 192.168.153.4 (on host cc202)
> 192.168.153.5 (on host cc202)
> 192.168.153.6 (on host cc202)
> 192.168.155.2 (on host cc201)
> 192.168.155.3 (on host cc201)
> 192.168.155.4 (on host cc202)

the cc201 OVS have 3 tap device and 1 gw on VLAN tag: 6:
--
        Port "tap6011d326-33"
            tag: 6
        Port "gw-e7863e30-1c"
            tag: 6
        Port "tap1585a714-10"
            tag: 6
        Port "tap77918099-5a"
            tag: 6
--

And 2 VM on cc201 with VLAN tag: 8, but this don't have difference:
        Port "tap09ae4556-2b"
            tag: 8
        Port "tap908304d1-d1"
            tag: 8
        Port "gw-6d53af57-d9"
            tag: 8
--

Did you just add more 2 VM on VLAN tag 6 on OVS of cc201, so have 2 more tap on VLAN tag 6, even not mention before?

mizumoto

Revision history for this message
Havent (guestly) said : 		#6

Yes, after that, I created moure VM. But I think it is not concern with the problem. Do you think so?

Revision history for this message
Havent (guestly) said : 		#7

Network 192.168.153.0/24 has vlan6;
Network 192.168.155.0/24 has vlan8.

Revision history for this message
mizumoto (mizumotoda) said : 		#8

I created similar environment with 2 compute node with OVS quantum. Among 2 nova-network communication is controlled by iptables. And your output don't have ICMP for each instance. So expected behaviour, each VM just communicate with same network. I changed security group setting to pass ICMP, each VM could communicate with other network.

However, I could duplicated your behaviour. So something wrong and have to investigate more.

http://paste.openstack.org/show/18639/

Yes, adding more VM with correct tap, there is no problem. I asked why I had some problem with old tap/gw device had odd behaviour before.

mizumoto

Revision history for this message
Havent (guestly) said : 		#9

mizumoto:
 Thanks for your reply!
From your paste, you created two networks (two vlans), one is 172.15.3.0, the other is 172.15.5.0.
If vlan does works, these two networks should not be connected!
In other words, if you log on 172.15.5.3 to ping 172.15.5.5, it is connected-----It is right!
If you log on 172.15.5.3 to ping 172.15.3.3 or 172.15.3.2, both of the two result should be not connected!
But your test result is similar with mine.
I don't know why~~

Revision history for this message
Havent (guestly) said : 		#10

And, "Among 2 nova-network communication is controlled by iptables. " May I ask how do you set your iptables configuration?
And, "I changed security group setting to pass ICMP, each VM could communicate with other network." What does your security group setting look like?

Revision history for this message
mizumoto (mizumotoda) said : 		#11

My understanding is tap device and OVS vlan tag works correct, however, iptables or forwarding between gw interfaces for outside interface not work. I see the tcpdump of routed gw interface, source IP address has changed (NAT or routing_source_ip of nova.conf?) not nova network but KVM's physical IP address.

localadmin@kvm155:~$ sudo tcpdump -i gw-0abd2a02-ed -n icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on gw-0abd2a02-ed, link-type EN10MB (Ethernet), capture size 65535 bytes
00:41:20.463956 IP 172.15.5.3 > 172.15.3.3: ICMP echo request, id 48129, seq 0, length 64
00:41:20.464752 IP 172.15.3.3 > 172.15.5.3: ICMP echo reply, id 48129, seq 0, length 64
^C
2 packets captured
2 packets received by filter
0 packets dropped by kernel
localadmin@kvm155:~$ sudo tcpdump -i gw-a9689dbb-dd -n icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on gw-a9689dbb-dd, link-type EN10MB (Ethernet), capture size 65535 bytes
00:41:46.571095 IP 10.127.1.155 > 172.15.3.3: ICMP echo request, id 48385, seq 0, length 64
00:41:46.571866 IP 172.15.3.3 > 10.127.1.155: ICMP echo reply, id 48385, seq 0, length 64

^C
2 packets captured
2 packets received by filter
0 packets dropped by kernel
localadmin@kvm155:~$ sudo tcpdump -i eth1 -n icmp
tcpdump: WARNING: eth1: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
00:42:26.610185 IP 10.127.1.155 > 172.15.3.3: ICMP echo request, id 50689, seq 0, length 64
00:42:26.610858 IP 172.15.3.3 > 10.127.1.155: ICMP echo reply, id 50689, seq 0, length 64

To change security group setting, just add ICMP rules for applied security group of the VM. In my case, default security group.

http://paste.openstack.org/show/18649/

mizumoto

Revision history for this message
mizumoto (mizumotoda) said : 		#12

I checked adding iptable entry manually, it could block the packet among networks.
However, this is not compatible security group and drop all communication among networks, so this is just information and testing purpose but this is the rule:

[Append rule]
iptables -t filter -A nova-network-FORWARD --in-interface gw-+ --out-interface gw-+ -j DROP

[Delete rule]
iptables -t filter -D nova-network-FORWARD --in-interface gw-+ --out-interface gw-+ -j DROP

mizumoto

Revision history for this message
Havent (guestly) said : 		#13

"I checked adding iptable entry manually, it could block the packet among networks."
Hmmm, I understand what you do. But should it be quantum's work? If we must add iptable entry to solve this problem manually, it will be a few entries when there are many networks! So I think there must somthing wrong with Quantum or Openvswitch. But I've never seen otherone reporting them:-(

But why I ping from networkA(on cc201) to networkB(also on cc201), the vlan can block the ping packets?

this morning, I checked the command iptables-save, and found these rules, do they have some relation with this problem?

-A nova-compute-local -d 192.168.153.2/32 -j nova-compute-inst-6
-A nova-compute-local -d 192.168.155.2/32 -j nova-compute-inst-22
-A nova-compute-local -d 192.168.155.3/32 -j nova-compute-inst-23
-A nova-compute-local -d 192.168.153.8/32 -j nova-compute-inst-26
-A nova-compute-local -d 192.168.153.9/32 -j nova-compute-inst-27
-A nova-compute-sg-fallback -j DROP
-A nova-filter-top -j nova-compute-local
-A nova-filter-top -j nova-network-local
-A nova-filter-top -j nova-manage-local
-A nova-filter-top -j nova-api-local
-A nova-network-FORWARD -i br-int -j ACCEPT
-A nova-network-FORWARD -o br-int -j ACCEPT
-A nova-network-INPUT -i gw-e7863e30-1c -p udp -m udp --dport 67 -j ACCEPT
-A nova-network-INPUT -i gw-e7863e30-1c -p tcp -m tcp --dport 67 -j ACCEPT
-A nova-network-INPUT -i gw-e7863e30-1c -p udp -m udp --dport 53 -j ACCEPT
-A nova-network-INPUT -i gw-e7863e30-1c -p tcp -m tcp --dport 53 -j ACCEPT
-A nova-network-INPUT -i gw-87a6f352-35 -p udp -m udp --dport 67 -j ACCEPT
-A nova-network-INPUT -i gw-87a6f352-35 -p tcp -m tcp --dport 67 -j ACCEPT
-A nova-network-INPUT -i gw-87a6f352-35 -p udp -m udp --dport 53 -j ACCEPT
-A nova-network-INPUT -i gw-87a6f352-35 -p tcp -m tcp --dport 53 -j ACCEPT
-A nova-network-INPUT -i gw-3faffe59-5d -p udp -m udp --dport 67 -j ACCEPT
-A nova-network-INPUT -i gw-3faffe59-5d -p tcp -m tcp --dport 67 -j ACCEPT
-A nova-network-INPUT -i gw-3faffe59-5d -p udp -m udp --dport 53 -j ACCEPT
-A nova-network-INPUT -i gw-3faffe59-5d -p tcp -m tcp --dport 53 -j ACCEPT
-A nova-network-INPUT -i gw-6d53af57-d9 -p udp -m udp --dport 67 -j ACCEPT
-A nova-network-INPUT -i gw-6d53af57-d9 -p tcp -m tcp --dport 67 -j ACCEPT
-A nova-network-INPUT -i gw-6d53af57-d9 -p udp -m udp --dport 53 -j ACCEPT
-A nova-network-INPUT -i gw-6d53af57-d9 -p tcp -m tcp --dport 53 -j ACCEPT
-A nova-network-INPUT -i gw-9a2757f4-2d -p udp -m udp --dport 67 -j ACCEPT
-A nova-network-INPUT -i gw-9a2757f4-2d -p tcp -m tcp --dport 67 -j ACCEPT
-A nova-network-INPUT -i gw-9a2757f4-2d -p udp -m udp --dport 53 -j ACCEPT
-A nova-network-INPUT -i gw-9a2757f4-2d -p tcp -m tcp --dport 53 -j ACCEPT

Revision history for this message
Havent (guestly) said : 		#14

to mizumoto:
"source IP address has changed (NAT or routing_source_ip of nova.conf?) "

I have something similar with you before. I think this because the nova-network add a rule like:
-A nova-network-snat -s 192.168.151.0/24 -j SNAT --to-source 10.131.0.31

I delete this rule. And thus, the source address won't be changed.

This is one of the problems, but I think it is not critical.

The critical problem also is why the vlan tag wrong or another words why it is connected between different gw*s?

Revision history for this message
mizumoto (mizumotoda) said : 		#15

Havent,

The rule I wrote for in/out gw interface is just adding to rest of normal iptables entry, the most of them generated for each instances. The original iptables are here:

http://paste.openstack.org/show/18675/
http://paste.openstack.org/show/18676/

How to know quantum work or not is to see ovs-vsctl show for tap device and agent VLAN setting for br-int with ovs_quantum db entry. Run quantum-agent with debug option.

I don't think vlan tag was wrong.

In your case:

cc201:192.168.153.2(tap)--->br-int-->(gw)192.168.153.1--->(gw)192.168.155.1--->br-int--->cc201:eth1--->
---->[tag:8]--sw--[tag:8]--->
--->cc202:eth1--->br-int--->(tap)cc202:192.168.155.2

So you watched cc201:eth1, the VLAN was tag:8.
I inserted iptables rule between in/out gateway for testing purpose in my environment.

mizumoto

Revision history for this message
Havent (guestly) said : 		#16

Yes, in my case, the packets' path is just like:
cc201:192.168.153.2(tap)--->br-int-->(gw)192.168.153.1--->(gw)192.168.155.1--->br-int--->cc201:eth1--->
---->[tag:8]--sw--[tag:8]--->
--->cc202:eth1--->br-int--->(tap)cc202:192.168.155.2

But, I mean 192.168.153.2 is under network 192.168.153.0/24, it is vlan6 not vlan8, why it has tag8(vlan8's tag) finally?
I mean if openvswitch know icmp's source is a vlan6 VM, and its destination is a vlan8 VM, the ovs should prevent the vlan6 packets to access and drop it.
Do I have the correct understanding?

Revision history for this message
mizumoto (mizumotoda) said : 		#17

The tag was attached / replaced when go out from cc201's eth1 to outside, in this case to cc202's eth1. The icmp is routed from vm of cc201 through 2 gw- interface, and finally decided to go outside. So last gw is 192.168.155.1 L2 domain, so the tag you watched on cc202 eth1 was tag: 8 because routed. That's one of reason the icmp has reached to cc202's VM even separate network.

However, it should be same behaviour when 1 vm send to another vm which is on other network even it was on other hypervisor or same hypervisor. It should be control by nova's security group or project separation of nova network setting. So in this viewpoint, nova configuration, iptables, or some other thing was wrong I think.

mizumoto

Revision history for this message
Havent (guestly) said : 		#18

"So in this viewpoint, nova configuration, iptables, or some other thing was wrong I think."
I agree with this point of view. on the same hypervisor, the result is not connected, but on the different hypervisor, it is connected.
I had reported this problem in bugs series, but I don't know who will tell us the result.

I will reinstall my environment and have a try. If this time the problem is not resolved, I think waiting for someone to analyze and resolve it is the best choice for us.

Revision history for this message
mizumoto (mizumotoda) said : 		#19

In my environment, the below routing_source_ip setting of nova.conf disabled the floating nat of iptables (nova-network-float-snat). This is you mentioned above.

--
nova.conf
--routing_source_ip=0.0.0.0
--

With changing this value, and restart nova-network will change snat of iptables.
http://paste.openstack.org/show/18677/

Then, the network had separated in my environment. That is, the vm can communicate with other vm only in same network. So floating IP setting will change behaviour.
I'm not sure which setting will be reasonable in this kind of environment, but I think you don't need reinstall for this.

mizumoto

Revision history for this message
Havent (guestly) said : 		#20

hi mizumoto,
Thanks a lot for your consecutive reply!
You mean in nova.conf you set routing_source_ip=0.0.0.0 ?

Revision history for this message
Havent (guestly) said : 		#21

I've change my setting as yours: routing_source_ip=0.0.0.0
but this problem still exists.

Can you paste your nova.conf configuration?

Revision history for this message
Havent (guestly) said : 		#22

Plus, to mizumoto:
May I ask what kind of method do you access each VM?
I found that after adding the rule: routing_source_ip=0.0.0.0, I cannot ping the VM IP address on my physical host.
So I access by
virsh list
virsh vncdisplay
and use vnc tools to access VMs, I found the problem disappear!

What about your method to access VM?

Revision history for this message
mizumoto (mizumotoda) said : 		#23

I use dashboard to access each VNC console.
But it's same situation that from controller of nova-network could not connect via ssh/ping to each VM with routing_source_ip set 0.0.0.0.

I think the problem is some mismatch between floating nat setting and iptables among multiple hypervisor.
Could you test more with specific scenario and raise this as another problem even if you set correct nova network and floating related configuration?

I will send the conf file later.

mizumoto

Revision history for this message
Havent (guestly) said : 		#24

OK. Thanks.
Next four days, I have vocation. I will update my information as soon as possible!
I think it is normal that not connect via ssh/ping, if it connects, the problem will appear again.

Revision history for this message
Havent (guestly) said : 		#25

To mizumoto,
I found that your configuration files are not the same at the parameter routing_source_ip(one is 0.0.0.0 and the other is a actual IP address).

Revision history for this message
mizumoto (mizumotoda) said : 		#26

I think that should not effect (2nd one is meaningless) because all gw- interface located on actual nova-network (1st, configured) host.

mizumoto

Revision history for this message
Havent (guestly) said : 		#27

Thanks mizumoto, that solved my question.

Revision history for this message
Trinath Somanchi (trinath-somanchi) said : 		#28

Hi Stackers-

I'm bringing up openstack with quantum and OVS.

Can any one guide me on bringing up VM with support of Quantum+OVS+VLan.

Thanking you

--
Trinath S