neutron

Instance failed network setup

Asked by Marco Colombo

Hi All,
i have a Openstack Grizzly installation with Ubuntu 12.04 and cloud archive packages update to the last version
Recently i get an error when i'm try to launch new instances.
The instance stop on state "networking", this is the nova-compute log :

2013-05-15 16:26:17.711 5588 DEBUG nova.openstack.common.rpc.amqp [-] MSG_ID is 939e148bea1644498babdfcd55759034 multicall /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py:586
2013-05-15 16:26:17.711 5588 DEBUG nova.openstack.common.rpc.amqp [-] UNIQUE_ID is ce3b99d23c0f4155a710765bfad2dd5a. _add_unique_id /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py:337
2013-05-15 16:26:22.921 DEBUG nova.openstack.common.rpc.common [req-cab25ef2-a3c6-465b-8c0d-d7f8519bf82f 20b301de7d514d808403a3c1c597eecb b7929874d1fb4ebfacb14a16aa141cd2] Timed out waiting for RPC response: timed out _error_callback /usr/lib/python2.7/dist-packages/nov
a/openstack/common/rpc/impl_kombu.py:628
2013-05-15 16:26:22.927 ERROR nova.compute.manager [req-cab25ef2-a3c6-465b-8c0d-d7f8519bf82f 20b301de7d514d808403a3c1c597eecb b7929874d1fb4ebfacb14a16aa141cd2] [instance: aa8a9b0b-b2da-4c00-a9c5-bb4e96d87298] Instance failed network setup
2013-05-15 16:26:22.927 5588 TRACE nova.compute.manager [instance: aa8a9b0b-b2da-4c00-a9c5-bb4e96d87298] Traceback (most recent call last):
2013-05-15 16:26:22.927 5588 TRACE nova.compute.manager [instance: aa8a9b0b-b2da-4c00-a9c5-bb4e96d87298] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1057, in _allocate_network
2013-05-15 16:26:22.927 5588 TRACE nova.compute.manager [instance: aa8a9b0b-b2da-4c00-a9c5-bb4e96d87298] security_groups=security_groups)
2013-05-15 16:26:22.927 5588 TRACE nova.compute.manager [instance: aa8a9b0b-b2da-4c00-a9c5-bb4e96d87298] File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 89, in wrapped
2013-05-15 16:26:22.927 5588 TRACE nova.compute.manager [instance: aa8a9b0b-b2da-4c00-a9c5-bb4e96d87298] return func(self, context, *args, **kwargs)
2013-05-15 16:26:22.927 5588 TRACE nova.compute.manager [instance: aa8a9b0b-b2da-4c00-a9c5-bb4e96d87298] File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 46, in wrapper
2013-05-15 16:26:22.927 5588 TRACE nova.compute.manager [instance: aa8a9b0b-b2da-4c00-a9c5-bb4e96d87298] res = f(self, context, *args, **kwargs)
2013-05-15 16:26:22.927 5588 TRACE nova.compute.manager [instance: aa8a9b0b-b2da-4c00-a9c5-bb4e96d87298] File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 270, in allocate_for_instance
2013-05-15 16:26:22.927 5588 TRACE nova.compute.manager [instance: aa8a9b0b-b2da-4c00-a9c5-bb4e96d87298] nw_info = self.network_rpcapi.allocate_for_instance(context, **args)
2013-05-15 16:26:22.927 5588 TRACE nova.compute.manager [instance: aa8a9b0b-b2da-4c00-a9c5-bb4e96d87298] File "/usr/lib/python2.7/dist-packages/nova/network/rpcapi.py", line 171, in allocate_for_instance
2013-05-15 16:26:22.927 5588 TRACE nova.compute.manager [instance: aa8a9b0b-b2da-4c00-a9c5-bb4e96d87298] topic=topic, version='1.9')
2013-05-15 16:26:22.927 5588 TRACE nova.compute.manager [instance: aa8a9b0b-b2da-4c00-a9c5-bb4e96d87298] File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/proxy.py", line 80, in call
2013-05-15 16:26:22.927 5588 TRACE nova.compute.manager [instance: aa8a9b0b-b2da-4c00-a9c5-bb4e96d87298] return rpc.call(context, self._get_topic(topic), msg, timeout)
2013-05-15 16:26:22.927 5588 TRACE nova.compute.manager [instance: aa8a9b0b-b2da-4c00-a9c5-bb4e96d87298] File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/__init__.py", line 140, in call
2013-05-15 16:26:22.927 5588 TRACE nova.compute.manager [instance: aa8a9b0b-b2da-4c00-a9c5-bb4e96d87298] return _get_impl().call(CONF, context, topic, msg, timeout)
2013-05-15 16:26:22.927 5588 TRACE nova.compute.manager [instance: aa8a9b0b-b2da-4c00-a9c5-bb4e96d87298] File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 798, in call
2013-05-15 16:26:22.927 5588 TRACE nova.compute.manager [instance: aa8a9b0b-b2da-4c00-a9c5-bb4e96d87298] rpc_amqp.get_connection_pool(conf, Connection))
2013-05-15 16:26:22.927 5588 TRACE nova.compute.manager [instance: aa8a9b0b-b2da-4c00-a9c5-bb4e96d87298] File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 612, in call
2013-05-15 16:26:22.927 5588 TRACE nova.compute.manager [instance: aa8a9b0b-b2da-4c00-a9c5-bb4e96d87298] rv = list(rv)
2013-05-15 16:26:22.927 5588 TRACE nova.compute.manager [instance: aa8a9b0b-b2da-4c00-a9c5-bb4e96d87298] File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 554, in __iter__
2013-05-15 16:26:22.927 5588 TRACE nova.compute.manager [instance: aa8a9b0b-b2da-4c00-a9c5-bb4e96d87298] self.done()
2013-05-15 16:26:22.927 5588 TRACE nova.compute.manager [instance: aa8a9b0b-b2da-4c00-a9c5-bb4e96d87298] File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-05-15 16:26:22.927 5588 TRACE nova.compute.manager [instance: aa8a9b0b-b2da-4c00-a9c5-bb4e96d87298] self.gen.next()
2013-05-15 16:26:22.927 5588 TRACE nova.compute.manager [instance: aa8a9b0b-b2da-4c00-a9c5-bb4e96d87298] File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 551, in __iter__
2013-05-15 16:26:22.927 5588 TRACE nova.compute.manager [instance: aa8a9b0b-b2da-4c00-a9c5-bb4e96d87298] self._iterator.next()
2013-05-15 16:26:22.927 5588 TRACE nova.compute.manager [instance: aa8a9b0b-b2da-4c00-a9c5-bb4e96d87298] File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 648, in iterconsume
2013-05-15 16:26:22.927 5588 TRACE nova.compute.manager [instance: aa8a9b0b-b2da-4c00-a9c5-bb4e96d87298] yield self.ensure(_error_callback, _consume)
2013-05-15 16:26:22.927 5588 TRACE nova.compute.manager [instance: aa8a9b0b-b2da-4c00-a9c5-bb4e96d87298] File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 566, in ensure
2013-05-15 16:26:22.927 5588 TRACE nova.compute.manager [instance: aa8a9b0b-b2da-4c00-a9c5-bb4e96d87298] error_callback(e)
2013-05-15 16:26:22.927 5588 TRACE nova.compute.manager [instance: aa8a9b0b-b2da-4c00-a9c5-bb4e96d87298] File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 629, in _error_callback
2013-05-15 16:26:22.927 5588 TRACE nova.compute.manager [instance: aa8a9b0b-b2da-4c00-a9c5-bb4e96d87298] raise rpc_common.Timeout()
2013-05-15 16:26:22.927 5588 TRACE nova.compute.manager [instance: aa8a9b0b-b2da-4c00-a9c5-bb4e96d87298] Timeout: Timeout while waiting on RPC response.
2013-05-15 16:26:22.927 5588 TRACE nova.compute.manager [instance: aa8a9b0b-b2da-4c00-a9c5-bb4e96d87298]
2013-05-15 16:26:22.931 DEBUG nova.openstack.common.lockutils [req-cab25ef2-a3c6-465b-8c0d-d7f8519bf82f 20b301de7d514d808403a3c1c597eecb b7929874d1fb4ebfacb14a16aa141cd2] Got semaphore "compute_resources" for method "abort"... inner /usr/lib/python2.7/dist-packages/nova
/openstack/common/lockutils.py:186

There is some RPC timeout. I already tried to reinstall quantum server, nova-compute and rabbitMQ.
Up to 5 days ago, i could launch instances without problems. Now, i'm using the same configuration.
Here you can find some configuration files :

http://paste.openstack.org/show/37276/

if you need other information, don't hesitate to ask

Thanks

Question information

Language:
English Edit question
Status:
Solved
For:
neutron Edit question
Assignee:
No assignee Edit question
Solved by:
yong sheng gong
Solved:
Last query:
Last reply:
Revision history for this message
Best yong sheng gong (gongysh) said : 		#1

it seems that you are using nova-network network API in nova compute:
2013-05-15 16:26:22.927 5588 TRACE nova.compute.manager [instance: aa8a9b0b-b2da-4c00-a9c5-bb4e96d87298] File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 270, in allocate_for_instance
2013-05-15 16:26:22.927 5588 TRACE nova.compute.manager [instance: aa8a9b0b-b2da-4c00-a9c5-bb4e96d87298] nw_info = self.network_rpcapi.allocate_for_instance(context, **args)
2013-05-15 16:26:22.927 5588 TRACE nova.compute.manager [instance: aa8a9b0b-b2da-4c00-a9c5-bb4e96d87298] File "/usr/lib/python2.7/dist-packages/nova/network/rpcapi.py", line 171, in allocate_for_instance

you should print out the configuration item values which is printed when nova compute starts with debug= true in nova.conf

Revision history for this message
Marco Colombo (colo90) said : 		#2

Thanks yong sheng gong, that solved my question.

Revision history for this message
Marco Colombo (colo90) said : 		#3

Thanks for support, i found the problem.
Seems that nova ignore part of my nova.conf, precisely after this line "[keystone_authtoken]".
After "[keystone_authtoken]" there was my quantum config option.
I moved all quantum configuration before this line. Now all works fine.