multi node instance failure

There are probably a couple of issues here. One is that you will need to use glance so that that the other node can grab images. You also may have issues with kvm module and permissions:

sudo modprobe kvm
sudo chgroup kvm /dev/kvm
sudo chmod g+w /dev/kvm

Vish

On Mar 30, 2011, at 4:06 PM, nimda wrote:

> New question #151125 on OpenStack Compute (nova):
> https://answers.launchpad.net/nova/+question/151125
>
> Hi,
>
> I've 6 machines I want to run OpenStack on. I'm planning on one controller, and the rest nodes.
>
> I've configured the controller to run instances locally, using ubuntu1010-UEC-localuser-image.tar.gz, and I'm able to ping and SSH to them, etc. Yay :D The joy.
>
> I then add a compute node. I'm manually installing parts to get a feel for what's happening. Here's what I do:-
>
> $ sudo apt-get install python-software-properties
> $ sudo add-apt-repository ppa:nova-core/trunk
> $ sudo apt-get update
> $ sudo apt-get install -y nova-common nova-compute python-mysqldb python-suds vlan
>
> I'll then scp nova.conf from the controller to the node, and restart all services on all machines, after nova-manage db sync on the controller. I don't edit network/interfaces for br100 as I'm running VlanManager, this could be a mistake. The node will then appear in the services table.
>
> When I attempt to run instances, and when they're scheduled for on the additional nodes the instances always fail. Here are 2 log files, from 2 nodes I've tried to configure:-
>
> CLOUD NODE 1
>
> 2011-03-30 17:07:34,326 WARNING nova.virt.libvirt_conn [-] instance instance-00000014: ignoring error injecting data into image 808928202 (Unexpected error whi
> le running command.
> Command: sudo tune2fs -c 0 -i 0 /dev/nbd15
> Exit code: 1
> Stdout: 'tune2fs 1.41.12 (17-May-2010)\n'
> Stderr: "tune2fs: Invalid argument while trying to open /dev/nbd15\nCouldn't find valid filesystem superblock.\n")
> 2011-03-30 17:08:05,516 ERROR nova.exception [-] Uncaught exception
> (nova.exception): TRACE: Traceback (most recent call last):
> (nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/exception.py", line 120, in _wrap
> (nova.exception): TRACE: return f(*args, **kw)
> (nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/libvirt_conn.py", line 569, in spawn
> (nova.exception): TRACE: self._conn.createXML(xml, 0)
> (nova.exception): TRACE: File "/usr/lib/python2.6/dist-packages/libvirt.py", line 1358, in createXML
> (nova.exception): TRACE: if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self)
> (nova.exception): TRACE: libvirtError: internal error process exited while connecting to monitor: char device redirected to /dev/pts/1
> (nova.exception): TRACE: qemu: could not load kernel '/var/lib/nova/instances/instance-00000014/kernel': Inappropriate ioctl for device
> (nova.exception): TRACE:
> (nova.exception): TRACE:
> 2011-03-30 17:08:05,518 ERROR nova.compute.manager [E44FOOY3XA5BTH21IXRG admin cloud] Instance '20' failed to spawn. Is virtualization enabled in the B
> IOS?
> (nova.compute.manager): TRACE: Traceback (most recent call last):
> (nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/compute/manager.py", line 234, in run_instance
> (nova.compute.manager): TRACE: self.driver.spawn(instance_ref)
> (nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/exception.py", line 126, in _wrap
> (nova.compute.manager): TRACE: raise Error(str(e))
> (nova.compute.manager): TRACE: Error: internal error process exited while connecting to monitor: char device redirected to /dev/pts/1
> (nova.compute.manager): TRACE: qemu: could not load kernel '/var/lib/nova/instances/instance-00000014/kernel': Inappropriate ioctl for device
> (nova.compute.manager): TRACE:
> (nova.compute.manager): TRACE:
>
> CLOUD NODE 2
>
> 2011-03-30 17:29:12,937 WARNING nova.virt.libvirt_conn [-] instance instance-00000017: ignoring error injecting data into image 808928202 (Unexpected error whi
> le running command.
> Command: sudo tune2fs -c 0 -i 0 /dev/nbd15
> Exit code: 1
> Stdout: 'tune2fs 1.41.12 (17-May-2010)\n'
> Stderr: "tune2fs: Invalid argument while trying to open /dev/nbd15\nCouldn't find valid filesystem superblock.\n")
> 2011-03-30 17:29:15,123 ERROR nova.exception [-] Uncaught exception
> (nova.exception): TRACE: Traceback (most recent call last):
> (nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/exception.py", line 120, in _wrap
> (nova.exception): TRACE: return f(*args, **kw)
> (nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/libvirt_conn.py", line 569, in spawn
> (nova.exception): TRACE: self._conn.createXML(xml, 0)
> (nova.exception): TRACE: File "/usr/lib/python2.6/dist-packages/libvirt.py", line 1358, in createXML
> (nova.exception): TRACE: if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self)
> (nova.exception): TRACE: libvirtError: operation failed: failed to retrieve chardev info in qemu with 'info chardev'
> (nova.exception): TRACE:
> 2011-03-30 17:29:15,125 ERROR nova.compute.manager [KO1VZT8AJ3DDN-UYB8F9 admin cloud] Instance '23' failed to spawn. Is virtualization enabled in the B
> IOS?
> (nova.compute.manager): TRACE: Traceback (most recent call last):
> (nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/compute/manager.py", line 234, in run_instance
> (nova.compute.manager): TRACE: self.driver.spawn(instance_ref)
> (nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/exception.py", line 126, in _wrap
> (nova.compute.manager): TRACE: raise Error(str(e))
> (nova.compute.manager): TRACE: Error: operation failed: failed to retrieve chardev info in qemu with 'info chardev'
> (nova.compute.manager): TRACE:
> 2011-03-30 17:29:19,186 INFO nova.compute.manager [-] Found instance 'instance-00000017' in DB but no VM. State=8, so setting state to shutoff.
> 2011-03-30 17:29:19,187 INFO nova.compute.manager [-] DB/VM state mismatch. Changing state from '8' to '5'
>
> There are error messages about virtualization not being enabled in the BIOS, but this is not the case, it is enabled.
>
> Any ideas?
>
>
> --
> You received this question notification because you are a member of Nova
> Core, which is an answer contact for OpenStack Compute (nova).

I've tested a qemu image on both nodes, and was able to execute a VM. I'm still receiving exactly the same errors in nova-compute log when trying to launch OpenStack instances though.

I'll look into setting up Glance, to see what effect that has on my experience. Are there any guides to set up Glance?

I tried to install an additional 2 node, just to see what sort of results I got. I'm starting to see a pattern now:-

NODE 3

2011-03-31 17:56:35,236 WARNING nova.virt.libvirt_conn [-] instance instance-0000001e: ignoring error injecting data into image 808928202 (Unexpected error whi
le running command.
Command: sudo tune2fs -c 0 -i 0 /dev/nbd15
Exit code: 1
Stdout: 'tune2fs 1.41.12 (17-May-2010)\n'
Stderr: "tune2fs: Invalid argument while trying to open /dev/nbd15\nCouldn't find valid filesystem superblock.\n")
2011-03-31 17:56:37,437 ERROR nova.exception [-] Uncaught exception
(nova.exception): TRACE: Traceback (most recent call last):
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/exception.py", line 120, in _wrap
(nova.exception): TRACE: return f(*args, **kw)
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/libvirt_conn.py", line 569, in spawn
(nova.exception): TRACE: self._conn.createXML(xml, 0)
(nova.exception): TRACE: File "/usr/lib/python2.6/dist-packages/libvirt.py", line 1358, in createXML
(nova.exception): TRACE: if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self)
(nova.exception): TRACE: libvirtError: operation failed: failed to retrieve chardev info in qemu with 'info chardev'
(nova.exception): TRACE:
2011-03-31 17:56:37,439 ERROR nova.compute.manager [7QVL-ILR6YXMO5S80MY9 positive posi-cloud] Instance '30' failed to spawn. Is virtualization enabled in the B
IOS?
(nova.compute.manager): TRACE: Traceback (most recent call last):
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/compute/manager.py", line 234, in run_instance
(nova.compute.manager): TRACE: self.driver.spawn(instance_ref)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/exception.py", line 126, in _wrap
(nova.compute.manager): TRACE: raise Error(str(e))
(nova.compute.manager): TRACE: Error: operation failed: failed to retrieve chardev info in qemu with 'info chardev'
(nova.compute.manager): TRACE:

NODE 4

2011-03-31 17:56:38,218 WARNING nova.virt.libvirt_conn [-] instance instance-00000021: ignoring error injecting data into image 808928202 (Unexpected error whi
le running command.
Command: sudo tune2fs -c 0 -i 0 /dev/nbd15
Exit code: 1
Stdout: 'tune2fs 1.41.12 (17-May-2010)\n'
Stderr: "tune2fs: Invalid argument while trying to open /dev/nbd15\nCouldn't find valid filesystem superblock.\n")
2011-03-31 17:56:40,447 ERROR nova.exception [-] Uncaught exception
(nova.exception): TRACE: Traceback (most recent call last):
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/exception.py", line 120, in _wrap
(nova.exception): TRACE: return f(*args, **kw)
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/libvirt_conn.py", line 569, in spawn
(nova.exception): TRACE: self._conn.createXML(xml, 0)
(nova.exception): TRACE: File "/usr/lib/python2.6/dist-packages/libvirt.py", line 1358, in createXML
(nova.exception): TRACE: if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self)
(nova.exception): TRACE: libvirtError: operation failed: failed to retrieve chardev info in qemu with 'info chardev'
(nova.exception): TRACE:
2011-03-31 17:56:40,449 ERROR nova.compute.manager [WMV3POEJTG1QY89TH31R positive posi-cloud] Instance '33' failed to spawn. Is virtualization enabled in the B
IOS?
(nova.compute.manager): TRACE: Traceback (most recent call last):
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/compute/manager.py", line 234, in run_instance
(nova.compute.manager): TRACE: self.driver.spawn(instance_ref)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/exception.py", line 126, in _wrap
(nova.compute.manager): TRACE: raise Error(str(e))
(nova.compute.manager): TRACE: Error: operation failed: failed to retrieve chardev info in qemu with 'info chardev'
(nova.compute.manager): TRACE:

I'd still like to know if there are recommended setup guides for Glance.

On Thu, Mar 31, 2011 at 12:46 PM, nimda
<email address hidden> wrote:
> I'll look into setting up Glance, to see what effect that has on my
> experience.  Are there any guides to set up Glance?

Sure thing.

http://glance.openstack.org/installing.html
http://glance.openstack.org/controllingservers.html

Working on the configuration documentation...

Cheers,
jay

I'd like to know if creating instances across multiple nodes requires setting up, past what's told in this guide:-

http://docs.openstack.org/openstack-compute/admin/content/ch01.html

I can create a single node, that's able to create instances, as mentioned above, but not onto additional nodes. Is the guide above missing anything I need to know to accomplish this?

I had started looking into Glance, as that's what similar questions have been resolved with. But, since reading about Glance, I'm now understanding I should be able to use nova's own object-store.

At the very least, is there a simple guide to create a multi-node cloud, that I can learn from?

Okay, I've had some success. I'm able to create instances on all nodes, only I'm not able to ping them. I was able to do this by copying /var/lib/nova/instances/_base/ from the controller to the node.

Additionally, I'm not able to ping any instance from the 2nd node.

This evening I managed to get instances on on controller/node and a standalone node, and was able to communicate with them.

I've tweaked many different parts so I'm not entirely sure what combination finally worked. I have tried to install another node, and that failed, I think this is because of missing /var/lib/nova/instances/_base/ files. I'll look into this.

So here's nova.conf

--dhcpbridge_flagfile=/etc/nova/nova.conf
--dhcpbridge=/usr/bin/nova-dhcpbridge
--logdir=/var/log/nova
--state_path=/var/lib/nova
--lock_path=/var/lock/nova
--verbose
--sql_connection=mysql://root:88bombs4@192.168.99.11/nova
--s3_host=192.168.99.11
--rabbit_host=192.168.99.11
--cc_host=192.168.99.11
--ec2_url=http://192.168.99.11:8773/services/Cloud
--network_manager=nova.network.manager.FlatDHCPManager
--flat_interface=eth0
--fixed_range=10.0.0.0/12
--network_size=32
--quota_instances=100

...and /etc/network/interfaces

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet dhcp

Other than that, I just followed this guide http://www.dubsquared.com. I can't yet connect to the instances publicly so that'll be my next task.

I got similar message. T.T
I use XCP(xenServer community version) as hypervisor.
Some has the answer?

2011-07-21 16:01:21,767 ERROR nova.compute.manager [X1HKIK61WUJ648ADHZZE novaadmin proj] Instance '77' failed to spawn. Is virtualization enabled in the BIO
S?
(nova.compute.manager): TRACE: Traceback (most recent call last):
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/compute/manager.py", line 234, in run_instance
(nova.compute.manager): TRACE: self.driver.spawn(instance_ref)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/xenapi_conn.py", line 188, in spawn
(nova.compute.manager): TRACE: self._vmops.spawn(instance)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/xenapi/vmops.py", line 117, in spawn
(nova.compute.manager): TRACE: vdi_uuid = self._create_disk(instance)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/xenapi/vmops.py", line 113, in _create_disk
(nova.compute.manager): TRACE: instance.image_id, user, project, disk_image_type)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/xenapi/vm_utils.py", line 386, in fetch_image
(nova.compute.manager): TRACE: access, image_type)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/xenapi/vm_utils.py", line 541, in _fetch_image_glance
(nova.compute.manager): TRACE: session, instance_id, image, access, image_type)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/xenapi/vm_utils.py", line 456, in _fetch_image_glance_disk
(nova.compute.manager): TRACE: lambda dev:
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/xenapi/vm_utils.py", line 944, in with_vdi_attached_here
(nova.compute.manager): TRACE: return f(dev)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/xenapi/vm_utils.py", line 458, in <lambda>
(nova.compute.manager): TRACE: virtual_size, image_file))
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/xenapi/vm_utils.py", line 1013, in _stream_disk
(nova.compute.manager): TRACE: _write_partition(virtual_size, dev)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/xenapi/vm_utils.py", line 1033, in _write_partition
(nova.compute.manager): TRACE: execute('sudo', 'parted', '--script', dest, 'mklabel', 'msdos')
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/xenapi/vm_utils.py", line 1031, in execute
(nova.compute.manager): TRACE: return utils.execute(*cmd, **kwargs)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/utils.py", line 173, in execute
(nova.compute.manager): TRACE: cmd=' '.join(cmd))
(nova.compute.manager): TRACE: ProcessExecutionError: Unexpected error while running command.
(nova.compute.manager): TRACE: Command: sudo parted --script /dev/xvdb mklabel msdos
(nova.compute.manager): TRACE: Exit code: 1
(nova.compute.manager): TRACE: Stdout: ''
(nova.compute.manager): TRACE: Stderr: 'sudo: no tty present and no askpass program specified\n'
(nova.compute.manager): TRACE:
(nova.compute.manager): TRACE: Traceback (most recent call last):
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/compute/manager.py", line 234, in run_instance
(nova.compute.manager): TRACE: self.driver.spawn(instance_ref)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/xenapi_conn.py", line 188, in spawn
(nova.compute.manager): TRACE: self._vmops.spawn(instance)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/xenapi/vmops.py", line 117, in spawn
(nova.compute.manager): TRACE: vdi_uuid = self._create_disk(instance)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/xenapi/vmops.py", line 113, in _create_disk
(nova.compute.manager): TRACE: instance.image_id, user, project, disk_image_type)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/xenapi/vm_utils.py", line 386, in fetch_image
(nova.compute.manager): TRACE: access, image_type)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/xenapi/vm_utils.py", line 541, in _fetch_image_glance
(nova.compute.manager): TRACE: session, instance_id, image, access, image_type)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/xenapi/vm_utils.py", line 456, in _fetch_image_glance_disk
(nova.compute.manager): TRACE: lambda dev:
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/xenapi/vm_utils.py", line 944, in with_vdi_attached_here
(nova.compute.manager): TRACE: return f(dev)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/xenapi/vm_utils.py", line 458, in <lambda>
(nova.compute.manager): TRACE: virtual_size, image_file))
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/xenapi/vm_utils.py", line 1013, in _stream_disk
(nova.compute.manager): TRACE: _write_partition(virtual_size, dev)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/xenapi/vm_utils.py", line 1033, in _write_partition
(nova.compute.manager): TRACE: execute('sudo', 'parted', '--script', dest, 'mklabel', 'msdos')
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/xenapi/vm_utils.py", line 1031, in execute
(nova.compute.manager): TRACE: return utils.execute(*cmd, **kwargs)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/utils.py", line 173, in execute
(nova.compute.manager): TRACE: cmd=' '.join(cmd))
(nova.compute.manager): TRACE: ProcessExecutionError: Unexpected error while running command.
(nova.compute.manager): TRACE: Command: sudo parted --script /dev/xvdb mklabel msdos
(nova.compute.manager): TRACE: Exit code: 1
(nova.compute.manager): TRACE: Stdout: ''
(nova.compute.manager): TRACE: Stderr: 'sudo: no tty present and no askpass program specified\n'
(nova.compute.manager): TRACE: