neutron

Launching instance failed in Multi node quantum

Asked by Neelakantam

We are trying to test the multi node setup using quantum and openstack using devstack. In our setup, we have one controller node and the other is compute node with the following configuration.

Controller configuration:
--------------------------------

localrc

HOST_IP=192.168.2.210
FLAT_INTERFACE=eth2
FIXED_RANGE=10.0.0.0/24
FIXED_NETWORK_SIZE=256
FLOATING_RANGE=10.0.1.128/28
MULTI_HOST=False
ENABLED_SERVICES="$ENABLED_SERVICES,quantum,q-svc,q-agt"

Compute Configuration:
-------------------------------

localrc

HOST_IP=192.168.2.211
FLAT_INTERFACE=eth1
FIXED_RANGE=10.0.0.0/24
FIXED_NETWORK_SIZE=256
FLOATING_RANGE=10.0.1.128/28
MULTI_HOST=False
MASTER_SERVER=192.168.2.210
ENABLED_SERVICES="n-cpu,quantum,q-agt"
MYSQL_HOST=$MASTER_SERVER
RABBIT_HOST=$MASTER_SERVER
KEYSTONE_AUTH_HOST=$MASTER_SERVER
GLANCE_HOSTPORT=$MASTER_SERVER:9292
EC2_DMZ_HOST=$MASTER_SERVER

 When we are trying to launch the instance on the compute node from the controller dashboard, it is giving error in the Spawning state. The nova compute log file is showing the following error:

Got semaphore "/opt/stack/nova/instances/_base/2acf438d92efd508d063f57861b674ab3d04da91" for method "copy_and_extend"... from (pid=3346) inner /opt/stack/nova/nova/utils.py:932
2012-03-30 14:24:02 DEBUG nova.utils [req-3e41e7f0-4f8e-4f31-bcff-929496d70028 b02b184c7f8340d58b043ad93684559b dd05174146e44739b560e56077612e9c] Running cmd (subprocess): qemu-img create -f qcow2 -o cluster_size=2M,backing_file=/opt/stack/nova/instances/_base/2acf438d92efd508d063f57861b674ab3d04da91 /opt/stack/nova/instances/instance-00000001/disk from (pid=3346) execute /opt/stack/nova/nova/utils.py:219
libvir: QEMU error : Failed to add tap interface to bridge '': No such device
2012-03-30 14:24:04 ERROR nova.compute.manager [req-3e41e7f0-4f8e-4f31-bcff-929496d70028 b02b184c7f8340d58b043ad93684559b dd05174146e44739b560e56077612e9c] [instance: e39578a5-d49a-45db-9921-a936821c42ed] Instance failed to spawn
(nova.compute.manager): TRACE: Traceback (most recent call last):
(nova.compute.manager): TRACE: File "/opt/stack/nova/nova/compute/manager.py", line 592, in _spawn
(nova.compute.manager): TRACE: self._legacy_nw_info(network_info), block_device_info)
(nova.compute.manager): TRACE: File "/opt/stack/nova/nova/exception.py", line 114, in wrapped
(nova.compute.manager): TRACE: return f(*args, **kw)
(nova.compute.manager): TRACE: File "/opt/stack/nova/nova/virt/libvirt/connection.py", line 873, in spawn
(nova.compute.manager): TRACE: self._create_new_domain(xml)
(nova.compute.manager): TRACE: File "/opt/stack/nova/nova/virt/libvirt/connection.py", line 1670, in _create_new_domain
(nova.compute.manager): TRACE: domain.createWithFlags(launch_flags)
(nova.compute.manager): TRACE: File "/usr/lib/python2.7/dist-packages/libvirt.py", line 372, in createWithFlags
(nova.compute.manager): TRACE: if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
(nova.compute.manager): TRACE: libvirtError: Failed to add tap interface to bridge '': No such device
(nova.compute.manager): TRACE:
2012-03-30 14:24:04 DEBUG nova.compute.manager [req-3e41e7f0-4f8e-4f31-bcff-929496d70028 b02b184c7f8340d58b043ad93684559b dd05174146e44739b560e56077612e9c] [instance: e39578a5-d49a-45db-9921-a936821c42ed] Deallocating network for instance from (pid=3346) _deallocate_network /opt/stack/nova/nova/compute/manager.py:616
2012-03-30 14:24:04 DEBUG nova.rpc.amqp [req-3e41e7f0-4f8e-4f31-bcff-929496d70028 b02b184c7f8340d58b043ad93684559b dd05174146e44739b560e56077612e9c] Making asynchronous cast on network... from (pid=3346) cast /opt/stack/nova/nova/rpc/amqp.py:346
2012-03-30 14:24:05 ERROR nova.rpc.amqp [req-3e41e7f0-4f8e-4f31-bcff-929496d70028 b02b184c7f8340d58b043ad93684559b dd05174146e44739b560e56077612e9c] Exception during message handling
(nova.rpc.amqp): TRACE: Traceback (most recent call last):
(nova.rpc.amqp): TRACE: File "/opt/stack/nova/nova/rpc/amqp.py", line 252, in _process_data
(nova.rpc.amqp): TRACE: rval = node_func(context=ctxt, **node_args)
(nova.rpc.amqp): TRACE: File "/opt/stack/nova/nova/exception.py", line 114, in wrapped
(nova.rpc.amqp): TRACE: return f(*args, **kw)
(nova.rpc.amqp): TRACE: File "/opt/stack/nova/nova/compute/manager.py", line 177, in decorated_function
(nova.rpc.amqp): TRACE: sys.exc_info())
(nova.rpc.amqp): TRACE: File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
(nova.rpc.amqp): TRACE: self.gen.next()
(nova.rpc.amqp): TRACE: File "/opt/stack/nova/nova/compute/manager.py", line 171, in decorated_function
(nova.rpc.amqp): TRACE: return function(self, context, instance_uuid, *args, **kwargs)
(nova.rpc.amqp): TRACE: File "/opt/stack/nova/nova/compute/manager.py", line 651, in run_instance
(nova.rpc.amqp): TRACE: do_run_instance()
(nova.rpc.amqp): TRACE: File "/opt/stack/nova/nova/utils.py", line 946, in inner
(nova.rpc.amqp): TRACE: retval = f(*args, **kwargs)
(nova.rpc.amqp): TRACE: File "/opt/stack/nova/nova/compute/manager.py", line 650, in do_run_instance
(nova.rpc.amqp): TRACE: self._run_instance(context, instance_uuid, **kwargs)
(nova.rpc.amqp): TRACE: File "/opt/stack/nova/nova/compute/manager.py", line 451, in _run_instance
(nova.rpc.amqp): TRACE: self._set_instance_error_state(context, instance_uuid)
(nova.rpc.amqp): TRACE: File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
(nova.rpc.amqp): TRACE: self.gen.next()
(nova.rpc.amqp): TRACE: File "/opt/stack/nova/nova/compute/manager.py", line 432, in _run_instance
(nova.rpc.amqp): TRACE: self._deallocate_network(context, instance)
(nova.rpc.amqp): TRACE: File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
(nova.rpc.amqp): TRACE: self.gen.next()
(nova.rpc.amqp): TRACE: File "/opt/stack/nova/nova/compute/manager.py", line 429, in _run_instance
(nova.rpc.amqp): TRACE: injected_files, admin_password)
(nova.rpc.amqp): TRACE: File "/opt/stack/nova/nova/compute/manager.py", line 592, in _spawn
(nova.rpc.amqp): TRACE: self._legacy_nw_info(network_info), block_device_info)
(nova.rpc.amqp): TRACE: File "/opt/stack/nova/nova/exception.py", line 114, in wrapped
(nova.rpc.amqp): TRACE: return f(*args, **kw)
(nova.rpc.amqp): TRACE: File "/opt/stack/nova/nova/virt/libvirt/connection.py", line 873, in spawn
(nova.rpc.amqp): TRACE: self._create_new_domain(xml)
(nova.rpc.amqp): TRACE: File "/opt/stack/nova/nova/virt/libvirt/connection.py", line 1670, in _create_new_domain
(nova.rpc.amqp): TRACE: domain.createWithFlags(launch_flags)
(nova.rpc.amqp): TRACE: File "/usr/lib/python2.7/dist-packages/libvirt.py", line 372, in createWithFlags
(nova.rpc.amqp): TRACE: if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
(nova.rpc.amqp): TRACE: libvirtError: Failed to add tap interface to bridge '': No such device
(nova.rpc.amqp): TRACE:
2012-03-30 14:24:26 DEBUG nova.manager [-] Running periodic task ComputeManager._publish_service_capabilities from (pid=3346) periodic_tasks /opt/stack/nova/nova/manager.py:152

We are using ubuntu 10.10 as host platform for both controller and compute nodes.
Has anybody done the multi host configuration using quantum (devstack) succesfully?
Please help me.

Question information

Language:
English Edit question
Status:
Answered
For:
neutron Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Neelakantam (neelugaddam) said : 		#1

Sorry, we are using ubuntu 11.10 on compute and controller nodes.

Revision history for this message
dan wendlandt (danwent) said : 		#2

Hi, it looks like the OVS bridge for libvirt to plug the VMs into does not exist.

stack.sh should default to using the OVS plugin, but you could see if explicitly setting Q_PLUGIN=openvswitch in your localrc helps.

Otherwise, can you post the output of stack.sh? I'm trying to figure out why the OVS bridge is not being created by stack.sh.

Revision history for this message
yong sheng gong (gongysh) said : 		#3

there are bugs for quantum-server to accept --config-file to specify all the configuration files. Also there is a bug for Devstack after it wants to use /etc/quantum/quantum.conf file to configuration quantum-server and its plugin

see bugs:
#965202 quantum.conf should be able to specify where the plugins.ini is
#965199 quantum config is broken after being moved to /etc
#965883 add a general funtion to read item in plugins.ini

Revision history for this message
Neelakantam (neelugaddam) said : 		#4

I solved this issue by adding the following tweaks in the stack.sh script of the compute node.

   libvirt_ovs_bridge=br-int
   libvirt_type=kvm
   libvirt_vif_type=ethernet
   libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtOpenVswitchDriver

Now the VM is launching on compute node, but the VMs are not getting the IP addresses.
Is there anything wrong in the network configuration.

Which branch of devstack is stable for testing the qunatum with multiple hosts?

Please help me.

Revision history for this message
dan wendlandt (danwent) said : 		#5

ah, of course. I only scanned the original config and had missed the fact that you are running multiple nodes.

Dave lapsley has some changes that will make devstack + quantum work for multiple hosts. He's planning on posting them very soon, right dave?

Revision history for this message
David Lapsley (dlapsley) said : 		#6

Yes. Will that out in the next few days.

Revision history for this message
Neelakantam (neelugaddam) said : 		#7

Does the manual configuration of the multi-node setup with quantum work (without devstack) ?

Revision history for this message
dan wendlandt (danwent) said : 		#8

Yes

Revision history for this message
NAVEEN REDDY (naveen-kuna) said : 		#9

Can you please point us for the detailed manual installation steps for two node set up using Quantum?

Revision history for this message
dan wendlandt (danwent) said : 		#10

Instructions for setting up Quantum depend on the plugin. The Quantum
Administrator Guide has pointers to each plugin's documentation.

For example, the Open vSwitch plugin docs are here:
http://openvswitch.org/openstack/documentation/

dan

On Sat, Mar 31, 2012 at 11:10 PM, NAVEEN REDDY <
<email address hidden>> wrote:

> Question #192137 on quantum changed:
> https://answers.launchpad.net/quantum/+question/192137
>
> NAVEEN REDDY requested more information:
> Can you please point us for the detailed manual installation steps for
> two node set up using Quantum?
>
> --
> You received this question notification because you are an answer
> contact for quantum.
>

--
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Dan Wendlandt
Nicira Networks: www.nicira.com
twitter: danwendlandt
~~~~~~~~~~~~~~~~~~~~~~~~~~~

Revision history for this message
yong sheng gong (gongysh) said : 		#11

I don't know what you mean by saying Multi node quantum.
document at
http://docs.openstack.org/incubation/openstack-network/admin/content/Limits-dle455.html
says Multi-host nova-network HA is not supported.

I think we just are able to have
1. one quantum server running,
2. one nova-network with quantum manager running
3. and agent on each compute node running

Revision history for this message
dan wendlandt (danwent) said : 		#12

Multi-node just means you are running devstack on more than one "host"
 (e.g., you may be running a "controller" node with services like
nova-scheduler, nova-api, quantum, etc) and one or more dedicated compute
nodes (i.e., those running nova-network and the quantum agent).

"Multi-host" is the somewhat confusing term given to being able to run
nova-network on each compute host, which is not what we're talking about in
this bug.

On Sun, Apr 1, 2012 at 8:25 AM, yong sheng gong <
<email address hidden>> wrote:

> Question #192137 on quantum changed:
> https://answers.launchpad.net/quantum/+question/192137
>
> yong sheng gong posted a new comment:
> I don't know what you mean by saying Multi node quantum.
> document at
>
> http://docs.openstack.org/incubation/openstack-network/admin/content/Limits-dle455.html
> says Multi-host nova-network HA is not supported.
>
> I think we just are able to have
> 1. one quantum server running,
> 2. one nova-network with quantum manager running
> 3. and agent on each compute node running
>
> --
> You received this question notification because you are an answer
> contact for quantum.
>

--
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Dan Wendlandt
Nicira Networks: www.nicira.com
twitter: danwendlandt
~~~~~~~~~~~~~~~~~~~~~~~~~~~

Revision history for this message
tiadobatima (gbaratto-3) said : 		#13

Hi Dan...

I guess I'm still confused about what "multi-node" setup is, and the quantum/openstack documentation mentioned earlier isn't very friendly:

http://docs.openstack.org/trunk/openstack-network/admin/content/Limits-dle455.html

Let's assume I have a setup with one "cloud controller" server with keystone, glance, nova-api, nova-scheduler, and quantum server (no hypervisor here). In this setup I'd also have 10 "compute nodes" (hypervisors), with everyone of them running nova-compute, nova-network and quantum-openvswitch-agent.

Is this how it's supposed to work?
Do I need to run nova-network in the "cloud manager" server too? Why?
What's the meaning of nova-network HA the doc is mentioning?

Thanks for the help!
g.

Revision history for this message
tiadobatima (gbaratto-3) said : 		#14

damn... getting confused with the names again: I meant "cloud controller", not "cloud manager"

Revision history for this message
dan wendlandt (danwent) said : 		#15

On Fri, Jun 15, 2012 at 3:06 PM, tiadobatima <
<email address hidden>> wrote:

> Question #192137 on quantum changed:
> https://answers.launchpad.net/quantum/+question/192137
>
> tiadobatima posted a new comment:
> Hi Dan...
>
> I guess I'm still confused about what "multi-node" setup is, and the
> quantum/openstack documentation mentioned earlier isn't very friendly:
>
> http://docs.openstack.org/trunk/openstack-network/admin/content/Limits-
> dle455.html
>
> Let's assume I have a setup with one "cloud controller" server with
> keystone, glance, nova-api, nova-scheduler, and quantum server (no
> hypervisor here). In this setup I'd also have 10 "compute nodes"
> (hypervisors), with everyone of them running nova-compute, nova-network
> and quantum-openvswitch-agent.
>

yes, that definitely works.

>
> Is this how it's supposed to work?
> Do I need to run nova-network in the "cloud manager" server too? Why?
> What's the meaning of nova-network HA the doc is mentioning?
>

The confusion here is that Nova created a special HA option for
nova-network which is enabled using the "multi_host" flag. This is what is
not supported in Quantum for Essex:
http://docs.openstack.org/diablo/openstack-compute/admin/content/existing-ha-networking-options.html

>
> Thanks for the help!
> g.
>
> --
> You received this question notification because you are an answer
> contact for quantum.
>

--
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Dan Wendlandt
Nicira, Inc: www.nicira.com
twitter: danwendlandt
~~~~~~~~~~~~~~~~~~~~~~~~~~~

Revision history for this message
tiadobatima (gbaratto-3) said : 		#16

Thanks!

For the configuration I described, do I need to run nova-network in the "cloud controller" as well? or just in the hypervisors?
My understanding is that we dont need the network manager, if we are using quantum?

Cheers,
g.

Revision history for this message
yong sheng gong (gongysh) said : 		#17

hi,
we need nova-network with quantum manager in essex. Quantum network manager will connect quantum to do actual work.
when nova-network is running with each compute host ( which is running hypervisor), this is multi-host network settings.

Can you help with this problem?

Provide an answer of your own, or ask Neelakantam for more information if necessary.

To post a message you must log in.