Not able to list floating IPs in ESSEX

Hi,

I have done a two node installation of ESSEX following the document http://docs.openstack.org/essex/openstack-compute/starter/content/.
Installed and configured keystone, glance and nova and dashboard.

The IP of the two servers are
===============================
server1:

auto eth0
iface eth0 inet static
address 192.168.1.1
netmask 255.255.255.0
network 192.168.1.0
broadcast 192.168.1.255

auto eth1
iface eth1 inet static
address 10.10.10.11
netmask 255.255.255.0
broadcast 10.10.10.255
gateway 10.10.10.1

Server 2:

auto eth0
iface eth0 inet static
address 192.168.1.2
netmask 255.255.255.0
network 192.168.1.0
broadcast 192.168.1.255

auto eth1
iface eth1 inet static
address 10.10.10.12
netmask 255.255.255.0
broadcast 10.10.10.255
gateway 10.10.10.1
=============================
I am using VLAN manager as networking. eth0 is my VLAN interface and eth1 is my public interface

The following packages are installed in my controller node

ii nova-api 2012.1+stable~20120612-3ee026e-0ubuntu1.1 OpenStack Compute - API frontend
ii nova-cert 2012.1+stable~20120612-3ee026e-0ubuntu1.1 OpenStack Compute - certificate management
ii nova-common 2012.1+stable~20120612-3ee026e-0ubuntu1.1 OpenStack Compute - common files
ii nova-compute 2012.1+stable~20120612-3ee026e-0ubuntu1.1 OpenStack Compute - compute node
ii nova-compute-kvm 2012.1+stable~20120612-3ee026e-0ubuntu1.1 OpenStack Compute - compute node (KVM)
ii nova-consoleauth 2012.1+stable~20120612-3ee026e-0ubuntu1.1 OpenStack Compute - Console Authenticator
ii nova-doc 2012.1+stable~20120612-3ee026e-0ubuntu1.1 OpenStack Compute - documentation
ii nova-network 2012.1+stable~20120612-3ee026e-0ubuntu1.1 OpenStack Compute - Network manager
ii nova-objectstore 2012.1+stable~20120612-3ee026e-0ubuntu1.1 OpenStack Compute - object store
ii nova-scheduler 2012.1+stable~20120612-3ee026e-0ubuntu1.1 OpenStack Compute - virtual machine scheduler
ii nova-volume 2012.1+stable~20120612-3ee026e-0ubuntu1.1 OpenStack Compute - storage
ii python-nova 2012.1+stable~20120612-3ee026e-0ubuntu1.1 OpenStack Compute Python libraries
ii python-novaclient 2012.1-0ubuntu1 client library for OpenStack Compute API

The /etc/nova/.conf file of my controller is as follows:
======================================================
[DEFAULT]
dhcpbridge_flagfile=/etc/nova/nova.conf
dhcpbridge=/usr/bin/nova-dhcpbridge
logdir=/var/log/nova
state_path=/var/lib/nova
lock_path=/run/lock/nova
verbose=true
allow_admin_api=true
use_deprecated_auth=false
auth_strategy=keystone
scheduler_driver=nova.scheduler.simple.SimpleScheduler
my_ip=10.10.10.11
s3_host=10.10.10.11
rabbit_host=10.10.10.11
cc_host=10.10.10.11
ec2_host=10.10.10.11
ec2_dmz_host=10.10.10.11
nova_url=http://10.10.10.11:8774/v1.1/
ec2_url=http://10.10.10.11:8773/services/Cloud
keystone_ec2_url=http://10.10.10.11:5000/v2.0/ec2tokens
api_paste_config=/etc/nova/api-paste.ini
libvirt_type=kvm
root_helper=sudo nova-rootwrap

#VNC configuration
novnc_enabled=False
#vncserver_listen=10.10.10.11
#novncproxy_base_url=http://10.10.10.11:6080/vnc_auto.html
#vncserver_proxyclient_address=10.10.10.11
#vncserver_listen=0.0.0.0

#Network config
network_manager=nova.network.manager.VlanManager
#floating_range=10.10.10.128/25
fixed_range=192.168.0.0/16
network_size=10
force_dhcp_release=True
#update_dhcp_on_disassociate=true
routing_source_ip=10.10.10.11
ec2_private_dns_show_ip=True
#sql_connection=postgresql://novadbadmin:novasecret@10.10.10.11/nova_essex1
sql_connection=postgresql://novadbadmin:novasecret@10.10.10.11/nova_essex
#sql_connection=postgresql://novadbadmin:novasecret@10.10.10.11/nova
iscsi_ip_prefix=192.168.1
vlan_interface=eth0
vlan_start=200
public_interface=eth1
multi_host=True
libvirt_use_virtio_for_bridges=True
connection_type=libvirt

#Image service
glance_api_servers=10.10.10.11:9292
image_service=nova.image.glance.GlanceImageService

#Volume service
#volume_driver=nexenta.volume.NexentaDriver
volume_driver=nova.volume.nexenta.volume.NexentaDriver
use_local_volumes=False
nexenta_sparse=True
nexenta_host=192.168.1.10
nexenta_rest_port=8080
nexenta_volume=nova
nexenta_user=****
nexenta_password=****

================================================

PROBLEM:

I have set the auth file as follows on controller node.

export SERVICE_TOKEN=admin
export OS_TENANT_NAME=cto_kol
export OS_USERNAME=novaadmin
export OS_PASSWORD=novaadmin
export OS_AUTH_URL="http://localhost:5000/v2.0/"
export SERVICE_ENDPOINT=http://localhost:35357/v2.0

I can list tenants,users,images using keystone and nova commands. Also glance index is running fine. Through dashboard also I can list instances and images.

But I can't list and allocate floating IPs.
When I run the command nova floating-ip-list in a terminal it returns an empty list -- usual behaviour. If I run the command on the same terminal again it hangs and gives the following error. But if I open a new terminal and run the command its working only for first time. For subsequent tries I am getting the same error.

"ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500)"

The /var/log/nova/nova-api.log file is showing the following error

log file nova-api.log

2012-07-08 18:17:23 ERROR nova.rpc.common [req-cd1bf8fa-8154-4444-8b94-1736c89d2f0b bcbca6eda62744e498ceff5660aded28 c77f5bfda1fe4f43ad59e59d3226ec92] Timed out waiting for RPC response: timed out
2012-07-08 18:17:23 TRACE nova.rpc.common Traceback (most recent call last):
2012-07-08 18:17:23 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 490, in ensure
2012-07-08 18:17:23 TRACE nova.rpc.common return method(*args, **kwargs)
2012-07-08 18:17:23 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 567, in _consume
2012-07-08 18:17:23 TRACE nova.rpc.common return self.connection.drain_events(timeout=timeout)
2012-07-08 18:17:23 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 175, in drain_events
2012-07-08 18:17:23 TRACE nova.rpc.common return self.transport.drain_events(self.connection, **kwargs)
2012-07-08 18:17:23 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 238, in drain_events
2012-07-08 18:17:23 TRACE nova.rpc.common return connection.drain_events(**kwargs)
2012-07-08 18:17:23 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 57, in drain_events
2012-07-08 18:17:23 TRACE nova.rpc.common return self.wait_multi(self.channels.values(), timeout=timeout)
2012-07-08 18:17:23 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 63, in wait_multi
2012-07-08 18:17:23 TRACE nova.rpc.common chanmap.keys(), allowed_methods, timeout=timeout)
2012-07-08 18:17:23 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 120, in _wait_multiple
2012-07-08 18:17:23 TRACE nova.rpc.common channel, method_sig, args, content = read_timeout(timeout)
2012-07-08 18:17:23 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 94, in read_timeout
2012-07-08 18:17:23 TRACE nova.rpc.common return self.method_reader.read_method()
2012-07-08 18:17:23 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py", line 221, in read_method
2012-07-08 18:17:23 TRACE nova.rpc.common raise m
2012-07-08 18:17:23 TRACE nova.rpc.common timeout: timed out
2012-07-08 18:17:23 TRACE nova.rpc.common
2012-07-08 18:17:23 ERROR nova.api.openstack [req-cd1bf8fa-8154-4444-8b94-1736c89d2f0b bcbca6eda62744e498ceff5660aded28 c77f5bfda1fe4f43ad59e59d3226ec92] Caught error: Timeout while waiting on RPC response.
2012-07-08 18:17:23 TRACE nova.api.openstack Traceback (most recent call last):
2012-07-08 18:17:23 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 41, in __call__
2012-07-08 18:17:23 TRACE nova.api.openstack return req.get_response(self.application)
2012-07-08 18:17:23 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1086, in get_response
2012-07-08 18:17:23 TRACE nova.api.openstack application, catch_exc_info=False)
2012-07-08 18:17:23 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1055, in call_application
2012-07-08 18:17:23 TRACE nova.api.openstack app_iter = application(self.environ, start_response)
2012-07-08 18:17:23 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/keystone/middleware/auth_token.py", line 176, in __call__
2012-07-08 18:17:23 TRACE nova.api.openstack return self.app(env, start_response)
2012-07-08 18:17:23 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
2012-07-08 18:17:23 TRACE nova.api.openstack return resp(environ, start_response)
2012-07-08 18:17:23 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
2012-07-08 18:17:23 TRACE nova.api.openstack return resp(environ, start_response)
2012-07-08 18:17:23 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
2012-07-08 18:17:23 TRACE nova.api.openstack return resp(environ, start_response)
2012-07-08 18:17:23 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
2012-07-08 18:17:23 TRACE nova.api.openstack response = self.app(environ, start_response)
2012-07-08 18:17:23 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
2012-07-08 18:17:23 TRACE nova.api.openstack return resp(environ, start_response)
2012-07-08 18:17:23 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
2012-07-08 18:17:23 TRACE nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
2012-07-08 18:17:23 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 210, in call_func
2012-07-08 18:17:23 TRACE nova.api.openstack return self.func(req, *args, **kwargs)
2012-07-08 18:17:23 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 803, in __call__
2012-07-08 18:17:23 TRACE nova.api.openstack content_type, body, accept)
2012-07-08 18:17:23 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 851, in _process_stack
2012-07-08 18:17:23 TRACE nova.api.openstack action_result = self.dispatch(meth, request, action_args)
2012-07-08 18:17:23 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 926, in dispatch
2012-07-08 18:17:23 TRACE nova.api.openstack return method(req=request, **action_args)
2012-07-08 18:17:23 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/floating_ips.py", line 137, in index
2012-07-08 18:17:23 TRACE nova.api.openstack floating_ips = self.network_api.get_floating_ips_by_project(context)
2012-07-08 18:17:23 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 92, in get_floating_ips_by_project
2012-07-08 18:17:23 TRACE nova.api.openstack {'method': 'get_floating_ips_by_project'})
2012-07-08 18:17:23 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/rpc/__init__.py", line 68, in call
2012-07-08 18:17:23 TRACE nova.api.openstack return _get_impl().call(context, topic, msg, timeout)
2012-07-08 18:17:23 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 674, in call
2012-07-08 18:17:23 TRACE nova.api.openstack return rpc_amqp.call(context, topic, msg, timeout, Connection.pool)
2012-07-08 18:17:23 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 343, in call
2012-07-08 18:17:23 TRACE nova.api.openstack rv = list(rv)
2012-07-08 18:17:23 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 304, in __iter__
2012-07-08 18:17:23 TRACE nova.api.openstack self.done()
2012-07-08 18:17:23 TRACE nova.api.openstack File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2012-07-08 18:17:23 TRACE nova.api.openstack self.gen.next()
2012-07-08 18:17:23 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 301, in __iter__
2012-07-08 18:17:23 TRACE nova.api.openstack self._iterator.next()
2012-07-08 18:17:23 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 572, in iterconsume
2012-07-08 18:17:23 TRACE nova.api.openstack yield self.ensure(_error_callback, _consume)
2012-07-08 18:17:23 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 503, in ensure
2012-07-08 18:17:23 TRACE nova.api.openstack error_callback(e)
2012-07-08 18:17:23 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 553, in _error_callback
2012-07-08 18:17:23 TRACE nova.api.openstack raise rpc_common.Timeout()
2012-07-08 18:17:23 TRACE nova.api.openstack Timeout: Timeout while waiting on RPC response.
2012-07-08 18:17:23 TRACE nova.api.openstack
2012-07-08 18:17:23 INFO nova.api.openstack [req-cd1bf8fa-8154-4444-8b94-1736c89d2f0b bcbca6eda62744e498ceff5660aded28 c77f5bfda1fe4f43ad59e59d3226ec92] http://10.10.10.11:8774/v2/c77f5bfda1fe4f43ad59e59d3226ec92/os-floating-ips returned with HTTP 500

Same behavior is observed from the dashboard.

Please help!!!