neutron

Router ports are down

Asked by Le Wang

After checking previous questions/answers, I still can not get my router working properly.
I have dedicated controller, compute and networking nodes running on Ubuntu12.04 with Havana. The problem is on Neutron node, when I create a router and attach a public and private network to it. I am using GRE to connect nodes.

The findings are
1. The tap and qr ports connected to br-int are always down.
# ovs-ofctl show br-int
OFPT_FEATURES_REPLY (xid=0x2): dpid:000062ac839d3848
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
 2(patch-tun): addr:0e:22:43:31:0c:b9
     config: 0
     state: 0
     speed: 0 Mbps now, 0 Mbps max
 3(tapc6385b42-b8): addr:e5:01:00:00:00:00
     config: PORT_DOWN
     state: LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
 4(qr-56fa966b-cf): addr:e3:01:00:00:00:00
     config: PORT_DOWN
     state: LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
 LOCAL(br-int): addr:62:ac:83:9d:38:48
     config: 0
     state: 0
     speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

2. It is similar to qg interface
# ovs-ofctl show br-ex
OFPT_FEATURES_REPLY (xid=0x2): dpid:0000080027d90a17
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
 1(qg-d482a3ce-b0): addr:e3:01:00:00:00:00
     config: PORT_DOWN
     state: LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
 2(eth2): addr:08:00:27:d9:0a:17
     config: 0
     state: 0
     current: 1GB-FD COPPER AUTO_NEG
     advertised: 10MB-HD 10MB-FD 100MB-HD 100MB-FD 1GB-FD COPPER AUTO_NEG
     supported: 10MB-HD 10MB-FD 100MB-HD 100MB-FD 1GB-FD COPPER AUTO_NEG
     speed: 1000 Mbps now, 1000 Mbps max
 LOCAL(br-ex): addr:08:00:27:d9:0a:17
     config: 0
     state: 0
     speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

3. However, the GRE tunnel between Networking and Compute nodes is up and running.
# ovs-ofctl show br-tun
OFPT_FEATURES_REPLY (xid=0x2): dpid:00001a4b859fd545
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
 1(patch-int): addr:32:85:96:3f:ea:e7
     config: 0
     state: 0
     speed: 0 Mbps now, 0 Mbps max
 2(gre-1): addr:ee:4e:ff:d0:dd:9d
     config: 0
     state: 0
     speed: 0 Mbps now, 0 Mbps max
 LOCAL(br-tun): addr:1a:4b:85:9f:d5:45
     config: 0
     state: 0
     speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

4. More errors found from /var/log/neutron*
# tail /var/log/neutron/dhcp-agent.log
2014-02-13 14:32:41.082 7622 ERROR neutron.common.legacy [-] Skipping unknown group key: firewall_driver

# tail /var/log/neutron/l3-agent.log
2014-02-13 14:32:41.414 7630 ERROR neutron.common.legacy [-] Skipping unknown group key: firewall_driver
2014-02-13 14:32:47.175 7630 WARNING neutron.openstack.common.loopingcall [-] task run outlasted interval by 1.702282 sec

# tail /var/log/neutron/dhcp-agent.log
2014-02-13 14:32:41.082 7622 ERROR neutron.common.legacy [-] Skipping unknown group key: firewall_driver
root@openstack-VirtualBox:/etc/init.d# tail /var/log/neutron/l3-agent.log
2014-02-13 14:32:41.414 7630 ERROR neutron.common.legacy [-] Skipping unknown group key: firewall_driver
2014-02-13 14:32:47.175 7630 WARNING neutron.openstack.common.loopingcall [-] task run outlasted interval by 1.702282 sec
root@openstack-VirtualBox:/etc/init.d# tail /var/log/neutron/openvswitch-agent.log
2014-02-13 14:32:43.512 7646 ERROR neutron.openstack.common.rpc.amqp [-] Exception during message handling
2014-02-13 14:32:43.512 7646 TRACE neutron.openstack.common.rpc.amqp Traceback (most recent call last):
2014-02-13 14:32:43.512 7646 TRACE neutron.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/amqp.py", line 438, in _process_data
2014-02-13 14:32:43.512 7646 TRACE neutron.openstack.common.rpc.amqp **args)
2014-02-13 14:32:43.512 7646 TRACE neutron.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2014-02-13 14:32:43.512 7646 TRACE neutron.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2014-02-13 14:32:43.512 7646 TRACE neutron.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 290, in port_update
2014-02-13 14:32:43.512 7646 TRACE neutron.openstack.common.rpc.amqp self.sg_agent.refresh_firewall()
2014-02-13 14:32:43.512 7646 TRACE neutron.openstack.common.rpc.amqp AttributeError: 'NoneType' object has no attribute 'refresh_firewall'
2014-02-13 14:32:43.512 7646 TRACE neutron.openstack.common.rpc.amqp

Here are my configuration files
# grep -v ^# /etc/neutron/dhcp_agent.ini | grep -v ^$
[DEFAULT]
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
dhcp_driver = neutron.agent.linux.dhcp.Dnsmasq
use_namespaces = True
auth_strategy = keystone
auth_host = controller
signing_dir = /var/cache/neutron
admin_tenant_name = service
admin_user = neutron
admin_password = NEUTRON_PASS
auth_url = http://controller:35357/v2.0
dhcp_agent_manager = neutron.agent.dhcp_agent.DhcpAgentWithStateReport
root_helper = sudo neutron-rootwrap /etc/neutron/rootwrap.conf
state_path = /var/lib/neutron

# grep -v ^# /etc/neutron/l3_agent.ini | grep -v ^$
[DEFAULT]
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
use_namespaces = True
external_network_bridge = br-ex
auth_stratepy = keystone
auth_host = controller
admin_tenant_name = service
admin_user = neutron
admin_password = NEUTRON_PASS
auth_url = http://controller:35357/v2.0
l3_agent_manager = neutron.agent.l3_agent.L3NATAgentWithStateReport
root_helper = sudo neutron-rootwrap /etc/neutron/rootwrap.conf

# grep -v ^# /etc/neutron/l3_agent.ini | grep -v ^$
[DEFAULT]
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
use_namespaces = True
external_network_bridge = br-ex
auth_stratepy = keystone
auth_host = controller
admin_tenant_name = service
admin_user = neutron
admin_password = NEUTRON_PASS
auth_url = http://controller:35357/v2.0
l3_agent_manager = neutron.agent.l3_agent.L3NATAgentWithStateReport
root_helper = sudo neutron-rootwrap /etc/neutron/rootwrap.conf
root@openstack-VirtualBox:/etc/init.d# grep -v ^# /etc/neutron/neutron.conf | grep -v ^$
[DEFAULT]
state_path = /var/lib/neutron
lock_path = $state_path/lock
core_plugin = neutron.plugins.openvswitch.ovs_neutron_plugin.OVSNeutronPluginV2
auth_strategy = keystone
allow_overlapping_ips = True
rabbit_host = controller
rabbit_userid = guest
rabbit_password = RABBIT_PASS
notification_driver = neutron.openstack.common.notifier.rpc_notifier
[quotas]
[agent]
root_helper = sudo /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf
[keystone_authtoken]
auth_host = controller
auth_port = 35357
auth_protocol = http
admin_tenant_name = service
admin_user = neutron
admin_password = NEUTRON_PASS
signing_dir = $state_path/keystone-signing
[database]
connection = mysql://neutron:NEUTRON_DBPASS@controller/neutron
[service_providers]
service_provider=LOADBALANCER:Haproxy:neutron.services.loadbalancer.drivers.haproxy.plugin_driver.HaproxyOnHostPluginDriver:default

# grep -v ^# /etc/neutron/api-paste.ini | grep -v ^$
[composite:neutron]
use = egg:Paste#urlmap
/: neutronversions
/v2.0: neutronapi_v2_0
[composite:neutronapi_v2_0]
use = call:neutron.auth:pipeline_factory
noauth = extensions neutronapiapp_v2_0
keystone = authtoken keystonecontext extensions neutronapiapp_v2_0
[filter:keystonecontext]
paste.filter_factory = neutron.auth:NeutronKeystoneContext.factory
[filter:authtoken]
paste.filter_factory = keystoneclient.middleware.auth_token:filter_factory
auth_host = controller
auth_port = 35357
auth_protocol = http
admin_tenant_name = service
admin_user = neutron
admin_password = NEUTRON_PASS
[filter:extensions]
paste.filter_factory = neutron.api.extensions:plugin_aware_extension_middleware_factory
[app:neutronversions]
paste.app_factory = neutron.api.versions:Versions.factory
[app:neutronapiapp_v2_0]
paste.app_factory = neutron.api.v2.router:APIRouter.factory

# grep -v ^# /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini | grep -v ^$
[ovs]
[agent]
[securitygroup]
firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver
[database]
connection = mysql://neutron:NEUTRON_DBPASS@controller/neutron
[OVS]
tenant_network_type = gre
tunnel_id_ranges = 1:1000
enable_tunneling = True
integration_bridge = br-int
tunnel_bridge = br-tun
local_ip = 10.20.20.2

I am running out of clue to correct this, any help is appreciated.

BR,
Le

Question information

Language:
English Edit question
Status:
Solved
For:
neutron Edit question
Assignee:
No assignee Edit question
Solved by:
Le Wang
Solved:
Last query:
Last reply:
Revision history for this message
Le Wang (le-l-wang) said : 		#1

The problem is caused by Libvirt VIF Driver. After uninstalling it, the instance can get IP address and ping to the outside. However, "ovs-ofctl show br-int" on networking node still shows tap and qr interfaces are down, even though traffic can go though them.Thanks for all the help so far.