Problem connecting to Floodlight from Quantum

We are implementing a test lab environment with OpenStack pointing to a Floodlight controller on another computer. The Floodlight controller is working and responds when a third computer running Devstack sends it requests, such as Add Network.

When we attempt to add a network from the OpenStack computer, the network is added in the MySQL quantum database on the OpenStack system, but there is no activity that shows on the Floodlight controller and the network creation fails. No subnet is created, but an entry for the network appears in the dashboard. We would expect the OpenStack computer to work like the DevStack computer and create a network with a subnet.

Here is the quantum log from the OpenStack system showing the failure:

2013-03-21 11:55:59 DEBUG [keystone.middleware.auth_token] Authenticating user token
2013-03-21 11:55:59 DEBUG [keystone.middleware.auth_token] Removing headers from request environment: X-Identity-Status,X-Tenant-Id,X-Tenant-Name,X-User-Id,X-User-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant,X-Role
2013-03-21 11:55:59 DEBUG [routes.middleware] No route matched for POST /networks.json
2013-03-21 11:55:59 DEBUG [routes.middleware] Matched POST /networks.json
2013-03-21 11:55:59 DEBUG [routes.middleware] Route path: '/networks{.format}', defaults: {'action': u'create', 'controller': wsgify(quantum.api.v2.resource.resource, RequestClass=<class 'quantum.api.v2.resource.Request'>)}
2013-03-21 11:55:59 DEBUG [routes.middleware] Match dict: {'action': u'create', 'controller': wsgify(quantum.api.v2.resource.resource, RequestClass=<class 'quantum.api.v2.resource.Request'>), 'format': u'json'}
2013-03-21 11:55:59 DEBUG [quantum.openstack.common.rpc.amqp] Sending network.create.start on notifications.info
2013-03-21 11:55:59 DEBUG [amqplib] Closed channel #1
2013-03-21 11:55:59 DEBUG [amqplib] using channel_id: 1
2013-03-21 11:55:59 DEBUG [amqplib] Channel open
2013-03-21 11:55:59 DEBUG [quantum.openstack.common.rpc.amqp] Sending network.create.start on notifications.info
2013-03-21 11:55:59 DEBUG [amqplib] Closed channel #1
2013-03-21 11:55:59 DEBUG [amqplib] using channel_id: 1
2013-03-21 11:55:59 DEBUG [amqplib] Channel open
2013-03-21 11:55:59 DEBUG [quantum.plugins.bigswitch.plugin] QuantumRestProxyV2: create_network() called
2013-03-21 11:55:59 DEBUG [quantum.plugins.bigswitch.common.lockutils] Got semaphore "rest_call" for method "rest_call"...
2013-03-21 11:55:59 DEBUG [quantum.plugins.bigswitch.common.lockutils] Attempting to grab file lock "rest_call" for method "rest_call"...
2013-03-21 11:55:59 DEBUG [quantum.plugins.bigswitch.common.lockutils] Released file lock "rest_call" at /usr/lib/python2.7/dist-packages/quantum/plugins/bigswitch/bsn-rest_call for method "rest_call"...
2013-03-21 11:55:59 ERROR [quantum.api.v2.resource] create failed
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/quantum/api/v2/resource.py", line 96, in resource
    result = method(request=request, **args)
  File "/usr/lib/python2.7/dist-packages/quantum/api/v2/base.py", line 335, in create
    obj = obj_creator(request.context, **kwargs)
  File "/usr/lib/python2.7/dist-packages/quantum/plugins/bigswitch/plugin.py", line 359, in create_network
    ret = self.servers.post(resource, data)
  File "/usr/lib/python2.7/dist-packages/quantum/plugins/bigswitch/plugin.py", line 250, in post
    return self.rest_call('POST', resource, data, headers)
  File "/usr/lib/python2.7/dist-packages/quantum/plugins/bigswitch/plugin.py", line 228, in rest_call
    ret = active_server.rest_call(action, resource, data, headers)
  File "/usr/lib/python2.7/dist-packages/quantum/plugins/bigswitch/common/lockutils.py", line 215, in inner
    with lock:
  File "/usr/lib/python2.7/dist-packages/quantum/plugins/bigswitch/common/lockutils.py", line 75, in __enter__
    self.lockfile = open(self.fname, 'w')
IOError: [Errno 13] Permission denied: '/usr/lib/python2.7/dist-packages/quantum/plugins/bigswitch/bsn-rest_call'
2013-03-21 11:55:59 DEBUG [eventlet.wsgi.server] 10.1.68.36 - - [21/Mar/2013 11:55:59] "POST //v2.0/networks.json HTTP/1.1" 500 215 0.109153
2013-03-21 11:55:59 DEBUG [keystone.middleware.auth_token] Authenticating user token

Here is the /etc/default/quantum-server file:

me@OpenStackComputer:/etc/default# cat quantum-server
# defaults for quantum-server

# path to config file corresponding to the core_plugin specified in
# quantum.conf
# QUANTUM_PLUGIN_CONFIG="/etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini"
QUANTUM_PLUGIN_CONFIG="/etc/quantum/plugins/restproxy/restproxy.ini"

Here is the restproxy.ini file:

me@OpenStackComputer:/etc/quantum/plugins/restproxy# cat restproxy.ini
[DATABASE]
sql_connection = mysql://User:Password@192.168.1.36:3306/restproxy_quantum
reconnect_interval = 2
[RESTPROXY]
servertimeout = 10
servers=192.168.1.32:8080

Here is the quantum.conf file, leaving out some commented out lines for clarity:

me@OpenStackComputer:/etc/quantum# cat quantum.conf
[DEFAULT]
# Show more verbose log output (sets INFO log level output)
verbose = True

# Show debugging output in logs (sets DEBUG log level output)
debug = True

# Address to bind the API server
bind_host = 0.0.0.0

# Port the bind the API server to
bind_port = 9696

# Path to the extensions. Note that this can be a colon-separated list of
# paths. For example:
# api_extensions_path = extensions:/path/to/more/extensions:/even/more/extensions
# The __path__ of quantum.extensions is appended to this, so if your
# extensions are in there you don't need to specify them here
# api_extensions_path =

# Quantum plugin provider module
core_plugin = quantum.plugins.bigswitch.plugin.QuantumRestProxyV2
allow_overlapping_ips = False

# Paste configuration file
api_paste_config = /etc/quantum/api-paste.ini

# The strategy to be used for auth.
# Supported values are 'keystone'(default), 'noauth'.
# auth_strategy = keystone

# Base MAC address. The first 3 octets will remain unchanged. If the
# 4h octet is not 00, it will also used. The others will be
# randomly generated.
# 3 octet
# base_mac = fa:16:3e:00:00:00
# 4 octet
# base_mac = fa:16:3e:4f:00:00

# Maximum amount of retries to generate a unique MAC address
# mac_generation_retries = 16

# DHCP Lease duration (in seconds)
# dhcp_lease_duration = 120

# RPC configuration options. Defined in rpc __init__
# The messaging module to use, defaults to kombu.
# rpc_backend = quantum.openstack.common.rpc.impl_kombu
# Size of RPC thread pool
# rpc_thread_pool_size = 64,
# Size of RPC connection pool
# rpc_conn_pool_size = 30
# Seconds to wait for a response from call or multicall
# rpc_response_timeout = 60
# Seconds to wait before a cast expires (TTL). Only supported by impl_zmq.
# rpc_cast_timeout = 30
# Modules of exceptions that are permitted to be recreated
# upon receiving exception data from an rpc call.
# allowed_rpc_exception_modules = quantum.openstack.common.exception, nova.exception
# AMQP exchange to connect to if using RabbitMQ or QPID
control_exchange = quantum

# ============ Notification System Options =====================

# Defined in notifier api
notification_driver = quantum.openstack.common.notifier.list_notifier
# default_notification_level = INFO
# myhost = myhost.com
# default_publisher_id = $myhost

# Defined in list_notifier
list_notifier_drivers = quantum.openstack.common.notifier.rabbit_notifier

I have performed these commands and they do generate a response from the Floodlight computer:

 ovs-vsctl add-br br-int
 ovs-vsctl –set-controller br-int tcp:192.168.1.32:6633
 ifconfig br-int up
 ifconfig eth0 up
 ovs-vsctl add-port br-int eth0

Also, the Floodlight controller's web interface sees

Switches (1)

DPID IP Address Vendor Packets Bytes Flows Connected Since

00:00:90:e2:ba:0a:23:b0 /192.168.1.36:50759 Nicira Networks, Inc. 0 0 0 Thursday, March 21, 2013 9:40:01 AM

Any ideas are greatly appreciated.

Thank you.