neutron

Quantum: When using OVS, virtual interfaces silently disappear or lose their IPs

Asked by Martin Gerhard Loschwitz

Hello all,

I'm running an OpenStack setup here with Quantum+OVS. As I run all services on one node, I need use_namespaces=true in my configuration file. With that, I can set up Quantum & OpenvSwitch nicely, but as soon as I fire up a VM, all the interfaces created by Quantum and OpenVSwitch silently disappear:

35: qbr54d531d7-f3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether 8e:7d:9b:e9:6d:87 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::3871:9ff:fef7:16f7/64 scope link
       valid_lft forever preferred_lft forever
36: qvo54d531d7-f3: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether ea:7f:8f:ca:f7:6c brd ff:ff:ff:ff:ff:ff
    inet6 fe80::e87f:8fff:feca:f76c/64 scope link
       valid_lft forever preferred_lft forever
37: qvb54d531d7-f3: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master qbr54d531d7-f3 state UP qlen 1000
    link/ether 8e:7d:9b:e9:6d:87 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::8c7d:9bff:fee9:6d87/64 scope link
       valid_lft forever preferred_lft forever
38: qbrc004d01b-cf: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether a2:08:7e:7a:5e:47 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::c072:a6ff:fea7:b450/64 scope link
       valid_lft forever preferred_lft forever
39: qvoc004d01b-cf: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether f6:28:36:fc:8a:c0 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::f428:36ff:fefc:8ac0/64 scope link
       valid_lft forever preferred_lft forever
40: qvbc004d01b-cf: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master qbrc004d01b-cf state UP qlen 1000
    link/ether a2:08:7e:7a:5e:47 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a008:7eff:fe7a:5e47/64 scope link
       valid_lft forever preferred_lft forever
41: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master qbr54d531d7-f3 state UNKNOWN qlen 500
    link/ether fe:16:3e:c4:70:a8 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc16:3eff:fec4:70a8/64 scope link
       valid_lft forever preferred_lft forever
42: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master qbrc004d01b-cf state UNKNOWN qlen 500
    link/ether fe:16:3e:8e:52:dc brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc16:3eff:fe8e:52dc/64 scope link
       valid_lft forever preferred_lft forever

With use_namespaces=False in my configuration, this doesn't happen.

My configuration mainly follows Emilien Macchi's howto; it's using GRE.

I hope anyone can help me on this one!

Best regards
Martin G. Loschwitz

Question information

Language:
English Edit question
Status:
Solved
For:
neutron Edit question
Assignee:
No assignee Edit question
Solved by:
Martin Gerhard Loschwitz
Solved:
Last query:
Last reply:
Revision history for this message
Martin Gerhard Loschwitz (martin-loschwitz) said : 		#1

Also interfaces apparently disappear silently, quantum shows a lot of messages like this one:

Nov 15 23:07:09|01029|netdev_linux|INFO|ioctl(SIOCGIFHWADDR) on tap1935fdda-34 device failed: No such device

Revision history for this message
Martin Gerhard Loschwitz (martin-loschwitz) said : 		#2

Here is what "ip netns | xargs -i ip netns exec {} ip a" returns -- there, the devices do have their IPs ...

root@alice:/etc/init# ip netns | xargs -i ip netns exec {} ip a
8: tap1935fdda-34: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether fa:16:3e:46:72:69 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.2/24 brd 10.0.0.255 scope global tap1935fdda-34
    inet6 fe80::f816:3eff:fe46:7269/64 scope link
       valid_lft forever preferred_lft forever
16: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
15: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
47: qr-0dafbc88-a4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether fa:16:3e:f8:97:36 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.1/24 brd 10.0.0.255 scope global qr-0dafbc88-a4
    inet6 fe80::f816:3eff:fef8:9736/64 scope link
       valid_lft forever preferred_lft forever
48: qg-94813a40-e8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether fa:16:3e:e6:cd:3d brd ff:ff:ff:ff:ff:ff
    inet 192.168.144.100/24 brd 192.168.144.255 scope global qg-94813a40-e8
    inet6 fe80::f816:3eff:fee6:cd3d/64 scope link
       valid_lft forever preferred_lft forever

And what's also interesting is that for the ports of my br-int bridge, they're all down:

root@alice:/etc/init# sudo ovs-ofctl show br-int
OFPT_FEATURES_REPLY (xid=0x1): ver:0x1, dpid:0000966d76133843
n_tables:255, n_buffers:256
features: capabilities:0xc7, actions:0xfff
 1(tap1935fdda-34): addr:88:00:00:00:00:00
     config: PORT_DOWN
     state: LINK_DOWN
 3(tap37ab6888-42): addr:76:de:d5:87:27:ce
     config: PORT_DOWN
     state: LINK_DOWN
 14(patch-tun): addr:1e:41:08:d4:e5:ea
     config: 0
     state: 0
 15(qr-0dafbc88-a4): addr:89:00:00:00:00:00
     config: PORT_DOWN
     state: LINK_DOWN
 LOCAL(br-int): addr:96:6d:76:13:38:43
     config: PORT_DOWN
     state: LINK_DOWN
OFPT_GET_CONFIG_REPLY (xid=0x3): frags=normal miss_send_len=0

Revision history for this message
dan wendlandt (danwent) said : 		#3

can you show a "before" and "after" of the devices disappearing?

the list of devices in your initial post are only the devices associated with VMs, so those would be in the root namespace.

the list of devices in post update #2 are the devices within the router namespace, is that correct?

Revision history for this message
yong sheng gong (gongysh) said : 		#4

I think
in #2:
device 8 and 16 is in dhcp namespace,
others are in router namespace.

in #1, all are in root namespace.

If we deploy in multiple nodes env:
all interfaces in #1 should be on compute node
all interfaces in #2 should be on network node

Revision history for this message
yong sheng gong (gongysh) said : 		#5

What hypervisor are u using? Xen?

Revision history for this message
Martin Gerhard Loschwitz (martin-loschwitz) said : 		#6

This is a one node setup only.
Am 16.11.2012 01:11 schrieb "yong sheng gong" <
<email address hidden>>:

> Your question #214317 on quantum changed:
> https://answers.launchpad.net/quantum/+question/214317
>
> Status: Needs information => Answered
>
> yong sheng gong proposed the following answer:
> I think
> in #2:
> device 8 and 16 is in dhcp namespace,
> others are in router namespace.
>
> in #1, all are in root namespace.
>
>
> If we deploy in multiple nodes env:
> all interfaces in #1 should be on compute node
> all interfaces in #2 should be on network node
>
> --
> If this answers your question, please go to the following page to let us
> know that it is solved:
> https://answers.launchpad.net/quantum/+question/214317/+confirm?answer_id=3
>
> If you still need help, you can reply to this email or go to the
> following page to enter your feedback:
> https://answers.launchpad.net/quantum/+question/214317
>
> You received this question notification because you asked the question.
>

Revision history for this message
Martin Gerhard Loschwitz (martin-loschwitz) said : 		#7

kvm
Am 16.11.2012 01:11 schrieb "yong sheng gong" <
<email address hidden>>:

> Your question #214317 on quantum changed:
> https://answers.launchpad.net/quantum/+question/214317
>
> yong sheng gong requested more information:
> What hypervisor are u using? Xen?
>
> --
> To answer this request for more information, you can either reply to
> this email or enter your reply at the following page:
> https://answers.launchpad.net/quantum/+question/214317
>
> You received this question notification because you asked the question.
>

Revision history for this message
Martin Gerhard Loschwitz (martin-loschwitz) said : 		#8

Dan, that is correct. posting before and after is somewhat difficult as the devices disappear the same second they are created.

Revision history for this message
dan wendlandt (danwent) said : 		#9

by "disappear", do you mean they are moved from the root namespace to a different namespace? If so, that is expected. Which actually devices in your example above disappear?

Revision history for this message
Martin Gerhard Loschwitz (martin-loschwitz) said : 		#10

What I mean by "disappear" is that they simply are not present in "ip a" anymore, and it looks like openvswitch can't handle them properly:

root@alice:/var/run/openvswitch# grep "No such device" /var/log/openvswitch/ovs-vswitchd.log | wc -l
3420

A lot of entries like this one:

Nov 16 01:57:27|01154|netdev|WARN|failed to get flags for network device tap1935fdda-34: No such device

And numerous other device names.

Are you on Freenode by any chance? My nickname there is madkiss, so if you prefer to communicate that way, we could do that and I will post a summary in here if we manage to solve the problem.

Revision history for this message
Martin Gerhard Loschwitz (martin-loschwitz) said : 		#11

This might be helpful, too (ntp is unrelated but immediately starts to listen on a new interface when it appears and stops to do so upon interface disappearence):

Nov 15 22:58:39 alice ntpd[3148]: Listen normally on 25 qbr54d531d7-f3 fe80::3871:9ff:fef7:16f7 UDP 123
Nov 15 22:58:39 alice ntpd[3148]: Listen normally on 26 qbrc004d01b-cf fe80::c072:a6ff:fea7:b450 UDP 123
Nov 15 22:58:39 alice ntpd[3148]: Listen normally on 27 qvbc004d01b-cf fe80::a008:7eff:fe7a:5e47 UDP 123
Nov 15 22:58:39 alice ntpd[3148]: Listen normally on 28 qvb54d531d7-f3 fe80::8c7d:9bff:fee9:6d87 UDP 123
Nov 15 22:58:39 alice ntpd[3148]: Listen normally on 29 qvo54d531d7-f3 fe80::e87f:8fff:feca:f76c UDP 123
Nov 15 22:58:39 alice ntpd[3148]: Listen normally on 30 qvoc004d01b-cf fe80::f428:36ff:fefc:8ac0 UDP 123
Nov 15 22:58:39 alice ntpd[3148]: peers refreshed
Nov 15 22:58:39 alice ntpd[3148]: new interface(s) found: waking up resolver
Nov 15 22:58:51 alice ntpd[3148]: Listen normally on 31 vnet1 fe80::fc16:3eff:fe8e:52dc UDP 123
Nov 15 22:58:51 alice ntpd[3148]: Listen normally on 32 vnet0 fe80::fc16:3eff:fec4:70a8 UDP 123
Nov 15 22:58:51 alice ntpd[3148]: peers refreshed
Nov 15 22:58:51 alice ntpd[3148]: new interface(s) found: waking up resolver
Nov 15 23:09:44 alice ntpd[3148]: Deleting interface #32 vnet0, fe80::fc16:3eff:fec4:70a8#123, interface stats: received=0, sent=0, dropped=0, active_time=653 secs
Nov 15 23:09:44 alice ntpd[3148]: Deleting interface #31 vnet1, fe80::fc16:3eff:fe8e:52dc#123, interface stats: received=0, sent=0, dropped=0, active_time=653 secs
Nov 15 23:09:44 alice ntpd[3148]: peers refreshed
Nov 15 23:09:49 alice ntpd[3148]: Deleting interface #30 qvoc004d01b-cf, fe80::f428:36ff:fefc:8ac0#123, interface stats: received=0, sent=0, dropped=0, active_time=670 secs
Nov 15 23:09:49 alice ntpd[3148]: Deleting interface #29 qvo54d531d7-f3, fe80::e87f:8fff:feca:f76c#123, interface stats: received=0, sent=0, dropped=0, active_time=670 secs
Nov 15 23:09:49 alice ntpd[3148]: Deleting interface #28 qvb54d531d7-f3, fe80::8c7d:9bff:fee9:6d87#123, interface stats: received=0, sent=0, dropped=0, active_time=670 secs
Nov 15 23:09:49 alice ntpd[3148]: Deleting interface #27 qvbc004d01b-cf, fe80::a008:7eff:fe7a:5e47#123, interface stats: received=0, sent=0, dropped=0, active_time=670 secs
Nov 15 23:09:49 alice ntpd[3148]: Deleting interface #26 qbrc004d01b-cf, fe80::c072:a6ff:fea7:b450#123, interface stats: received=0, sent=0, dropped=0, active_time=670 secs
Nov 15 23:09:49 alice ntpd[3148]: Deleting interface #25 qbr54d531d7-f3, fe80::3871:9ff:fef7:16f7#123, interface stats: received=0, sent=0, dropped=0, active_time=670 secs

Revision history for this message
Martin Gerhard Loschwitz (martin-loschwitz) said : 		#12

Here are some messages that look suspicious -- I see these when firing up the quantum OVS agent:

Nov 16 02:07:32 alice ovs-vsctl: 00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl --timeout=2 add-port br-int patch-tun
Nov 16 02:07:32 alice ovs-vsctl: 00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl --timeout=2 set Interface patch-tun type=patch
Nov 16 02:07:32 alice ovs-vswitchd: 00035|netdev_vport|ERR|patch-tun: patch type requires valid 'peer' argument
Nov 16 02:07:32 alice ovs-vsctl: 00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl --timeout=2 set Interface patch-tun options:peer=patch-int
Nov 16 02:07:32 alice ovs-vsctl: 00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl --timeout=2 add-port br-tun patch-int
Nov 16 02:07:32 alice ovs-vsctl: 00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl --timeout=2 set Interface patch-int type=patch
Nov 16 02:07:32 alice ovs-vswitchd: 00056|netdev_vport|ERR|patch-int: patch type requires valid 'peer' argument
Nov 16 02:07:32 alice ovs-vsctl: 00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl --timeout=2 set Interface patch-int options:peer=patch-tun

Revision history for this message
dan wendlandt (danwent) said : 		#13

The devices that appear and then are removed in #11 are created and destroyed by Nova when VMs are booted and destroyed. Are you saying that those specific devices (e.g., vnet0) where destroyed without deleting the VMs? That would be very surprising.

The other devices (e.g., tapXXX and qrYYY or qgwZZZZ) are not supposed to be in the main namespace if use_namespaces=True

what do you see when you run "ip netns list" ? And then for each namespace name you see, run:

ip netns exec <namespace-name> ip addr

Revision history for this message
Martin Gerhard Loschwitz (martin-loschwitz) said : 		#14

Okay, so let's bring some order into this. Here is what I can see before I start anything related to Quantum (openvswitch is running, though):

root@alice:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet 169.254.169.254/32 scope link lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:10:ab:99:1c brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.111/24 brd 192.168.122.255 scope global eth0
    inet6 fe80::5054:10ff:feab:991c/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:10:cd:a9:74 brd ff:ff:ff:ff:ff:ff
    inet 192.168.133.111/24 brd 192.168.133.255 scope global eth1
    inet6 fe80::5054:10ff:fecd:a974/64 scope link
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:10:ef:a9:74 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:10ff:feef:a974/64 scope link
       valid_lft forever preferred_lft forever
6: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
    link/ether 9e:67:7a:08:10:4d brd ff:ff:ff:ff:ff:ff
7: br-ex: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
    link/ether 6a:59:e1:58:45:40 brd ff:ff:ff:ff:ff:ff
12: br-tun: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
    link/ether 5e:6e:e0:5d:f7:41 brd ff:ff:ff:ff:ff:ff

And after starting Quantum, I see this:

root@alice:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet 169.254.169.254/32 scope link lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:10:ab:99:1c brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.111/24 brd 192.168.122.255 scope global eth0
    inet6 fe80::5054:10ff:feab:991c/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:10:cd:a9:74 brd ff:ff:ff:ff:ff:ff
    inet 192.168.133.111/24 brd 192.168.133.255 scope global eth1
    inet6 fe80::5054:10ff:fecd:a974/64 scope link
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:10:ef:a9:74 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:10ff:feef:a974/64 scope link
       valid_lft forever preferred_lft forever
6: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
    link/ether 9e:67:7a:08:10:4d brd ff:ff:ff:ff:ff:ff
7: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether 6a:59:e1:58:45:40 brd ff:ff:ff:ff:ff:ff
    inet 192.168.144.10/24 scope global br-ex
    inet6 fe80::6859:e1ff:fe58:4540/64 scope link
       valid_lft forever preferred_lft forever
18: br-tun: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
    link/ether ae:7a:8d:4b:cc:43 brd ff:ff:ff:ff:ff:ff

The output you have requested, Dan, is this:

root@alice:~# ip netns exec qrouter-dcb14759-ed47-4c67-b316-3f45fb7b1ebb ip addr
15: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
19: qr-a7316545-42: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether fa:16:3e:be:27:e4 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.1/24 brd 10.0.0.255 scope global qr-a7316545-42
    inet6 fe80::f816:3eff:febe:27e4/64 scope link
       valid_lft forever preferred_lft forever
20: qg-f39dde60-d3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether fa:16:3e:1e:a4:ed brd ff:ff:ff:ff:ff:ff
    inet 192.168.144.100/24 brd 192.168.144.255 scope global qg-f39dde60-d3
    inet6 fe80::f816:3eff:fe1e:a4ed/64 scope link
       valid_lft forever preferred_lft forever

root@alice:~# ip netns exec qdhcp-882d9ede-791d-4c7f-9720-8bdc2889ffd3 ip addr
13: tap2fe63857-4a: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether fa:16:3e:bc:95:9d brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.2/24 brd 10.0.0.255 scope global tap2fe63857-4a
    inet6 fe80::f816:3eff:febc:959d/64 scope link
       valid_lft forever preferred_lft forever
14: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever

Revision history for this message
yong sheng gong (gongysh) said : 		#15

why cannot I see the vnetxxx at #14 before starting quantum now? The VMs were deleted?
are u using fedora or ubuntu? and the version number of kernel and ovs?
Thanks

Revision history for this message
Martin Gerhard Loschwitz (martin-loschwitz) said : 		#16

Turns out I was just too stupid for this. The interfaces did not "disappear", they were just moved into different network namespaces, exactly the way it is supposed to happen with use_namespaces=true.