euca-run-instances: failed to spawn

I am seeing the same problem. I have two nodes, one running api, network and compute and one just a compute node. Both nodes are running ubuntu 10.10, and the nova packages are installed from ppa trunk:

ii nova-api 2011.2~bzr815-0ubuntu0ppa1~lucid1 OpenStack Compute - Nova - API frontend
ii nova-common 2011.2~bzr815-0ubuntu0ppa1~lucid1 OpenStack Compute - Nova - common files
ii nova-compute 2011.2~bzr815-0ubuntu0ppa1~lucid1 OpenStack Compute - Nova - compute node
ii nova-doc 2011.2~bzr815-0ubuntu0ppa1~lucid1 OpenStack Compute - Nova - documetation
ii nova-network 2011.2~bzr815-0ubuntu0ppa1~lucid1 OpenStack Compute - Nova - Network thingamajig
ii nova-objectstore 2011.2~bzr815-0ubuntu0ppa1~lucid1 OpenStack Compute - Nova - object store
ii nova-scheduler 2011.2~bzr815-0ubuntu0ppa1~lucid1 OpenStack Compute - Nova - Scheduler
ii python-nova 2011.2~bzr815-0ubuntu0ppa1~lucid1 OpenStack Compute - Nova - Python libraries
ii python-novaclient 2.4-0ubuntu1~lucid1 client library for OpenStack Compute API

When i launch an instance, if scheduled on the node running all the services (network, objectstore, scheduler) the instances boots and runs fine. On the compute only node:

2011-03-17 15:40:48,164 DEBUG nova.utils [-] Running cmd (subprocess): mkdir -p /var/lib/nova/instances/instance-00000028/ from (pid=1382) execute /usr/lib/pymodules/python2.6/nova/utils.py:148
2011-03-17 15:40:48,174 INFO nova.virt.libvirt_conn [-] instance instance-00000028: Creating image
2011-03-17 15:40:48,233 DEBUG nova.utils [-] Running cmd (subprocess): cp /var/lib/nova/instances/_base/7faf7e4d /var/lib/nova/instances/instance-00000028/kernel from (pid=1382) execute /usr/lib/pymodules/python2.6/nova/utils.py:148
2011-03-17 15:40:48,265 DEBUG nova.utils [-] Running cmd (subprocess): qemu-img create -f qcow2 -o cluster_size=2M,backing_file=/var/lib/nova/instances/_base/1b9c0a16 /var/lib/nova/instances/instance-00000028/disk from (pid=1382) execute /usr/lib/pymodules/python2.6/nova/utils.py:148
2011-03-17 15:40:48,315 DEBUG nova.utils [-] Running cmd (subprocess): qemu-img create -f qcow2 -o cluster_size=2M,backing_file=/var/lib/nova/instances/_base/local_20 /var/lib/nova/instances/instance-00000028/disk.local from (pid=1382) execute /usr/lib/pymodules/python2.6/nova/utils.py:148
2011-03-17 15:40:48,424 INFO nova.virt.libvirt_conn [-] instance instance-00000028: injecting key into image 463211030
2011-03-17 15:40:48,424 INFO nova.virt.libvirt_conn [-] instance instance-00000028: injecting net into image 463211030
2011-03-17 15:40:48,434 DEBUG nova.utils [-] Running cmd (subprocess): sudo qemu-nbd -c /dev/nbd15 /var/lib/nova/instances/instance-00000028/disk from (pid=1382) execute /usr/lib/pymodules/python2.6/nova/utils.py:148
2011-03-17 15:40:48,460 DEBUG nova.utils [-] Running cmd (subprocess): sudo tune2fs -c 0 -i 0 /dev/nbd15 from (pid=1382) execute /usr/lib/pymodules/python2.6/nova/utils.py:148
2011-03-17 15:40:48,474 DEBUG nova.utils [-] Result was 1 from (pid=1382) execute /usr/lib/pymodules/python2.6/nova/utils.py:161
2011-03-17 15:40:48,475 DEBUG nova.utils [-] Running cmd (subprocess): sudo qemu-nbd -d /dev/nbd15 from (pid=1382) execute /usr/lib/pymodules/python2.6/nova/utils.py:148
2011-03-17 15:40:48,490 WARNING nova.virt.libvirt_conn [-] instance instance-00000028: ignoring error injecting data into image 463211030 (Unexpected error while running command.
Command: sudo tune2fs -c 0 -i 0 /dev/nbd15
Exit code: 1
Stdout: 'tune2fs 1.41.12 (17-May-2010)\n'
Stderr: "tune2fs: Invalid argument while trying to open /dev/nbd15\nCouldn't find valid filesystem superblock.\n")
2011-03-17 15:41:20,229 ERROR nova.exception [-] Uncaught exception
(nova.exception): TRACE: Traceback (most recent call last):
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/exception.py", line 120, in _wrap
(nova.exception): TRACE: return f(*args, **kw)
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/libvirt_conn.py", line 425, in spawn
(nova.exception): TRACE: self._conn.createXML(xml, 0)
(nova.exception): TRACE: File "/usr/lib/python2.6/dist-packages/libvirt.py", line 1358, in createXML
(nova.exception): TRACE: if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self)
(nova.exception): TRACE: libvirtError: internal error process exited while connecting to monitor: char device redirected to /dev/pts/1
(nova.exception): TRACE: qemu: could not load kernel '/var/lib/nova/instances/instance-00000028/kernel': Inappropriate ioctl for device
(nova.exception): TRACE:
(nova.exception): TRACE:
2011-03-17 15:41:20,230 ERROR nova.compute.manager [TTBRINFUQW3TUQ2JUX53 john project1] instance 40: Failed to spawn
(nova.compute.manager): TRACE: Traceback (most recent call last):
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/compute/manager.py", line 218, in run_instance
(nova.compute.manager): TRACE: self.driver.spawn(instance_ref)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/exception.py", line 126, in _wrap
(nova.compute.manager): TRACE: raise Error(str(e))
(nova.compute.manager): TRACE: Error: internal error process exited while connecting to monitor: char device redirected to /dev/pts/1
(nova.compute.manager): TRACE: qemu: could not load kernel '/var/lib/nova/instances/instance-00000028/kernel': Inappropriate ioctl for device
(nova.compute.manager): TRACE:

and euca-describe-instances shows 'failed to spawn'

Regards
John

I am seeing something like this, and it's go to do with the 2nd node installe via packages, with my investigations it looks like the instance is not being copied over to the node. My node was missing the _base directory under /var/lib/nova/instances yet the instance itself is created under this directory. Any help would be great!

here is my compute error log

011-03-17 15:29:42,987 INFO nova [-] called setup_basic_filtering in nwfilter
2011-03-17 15:29:42,987 INFO nova [-] ensuring static filters
2011-03-17 15:29:43,136 INFO nova [-] <nova.db.sqlalchemy.models.SecurityGroupIngressRule object at 0x38ead10>
2011-03-17 15:29:43,136 INFO nova [-] <nova.db.sqlalchemy.models.SecurityGroupIngressRule object at 0x3bf1ad0>
2011-03-17 15:29:43,136 INFO nova [-] <nova.db.sqlalchemy.models.SecurityGroupIngressRule object at 0x3bf1950>
2011-03-17 15:29:43,137 INFO nova [-] <nova.db.sqlalchemy.models.SecurityGroupIngressRule object at 0x3bf1a50>
2011-03-17 15:29:43,145 DEBUG nova.utils [-] Attempting to grab iptables for method apply... from (pid=4027) inner /usr/lib/pymodules/python2.6/nova/utils.py:535
2011-03-17 15:29:43,153 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-save -t filter from (pid=4027) execute /usr/lib/pymodules/python2.6/nova/utils.py:148
2011-03-17 15:29:43,163 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-restore from (pid=4027) execute /usr/lib/pymodules/python2.6/nova/utils.py:148
2011-03-17 15:29:43,173 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-save -t nat from (pid=4027) execute /usr/lib/pymodules/python2.6/nova/utils.py:148
2011-03-17 15:29:43,182 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-restore from (pid=4027) execute /usr/lib/pymodules/python2.6/nova/utils.py:148
2011-03-17 15:29:43,201 DEBUG nova.utils [-] Running cmd (subprocess): mkdir -p /var/lib/nova/instances/instance-00000015/ from (pid=4027) execute /usr/lib/pymodules/python2.6/nova/utils.py:148
2011-03-17 15:29:43,206 INFO nova.virt.libvirt_conn [-] instance instance-00000015: Creating image
2011-03-17 15:29:43,277 ERROR nova.exception [-] Uncaught exception
(nova.exception): TRACE: Traceback (most recent call last):
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/exception.py", line 120, in _wrap
(nova.exception): TRACE: return f(*args, **kw)
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/libvirt_conn.py", line 424, in spawn
(nova.exception): TRACE: self._create_image(instance, xml)
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/libvirt_conn.py", line 609, in _create_image
(nova.exception): TRACE: fname = '%08x' % int(disk_images['kernel_id'])
(nova.exception): TRACE: ValueError: invalid literal for int() with base 10: 'ami-zmpaoemm'
(nova.exception): TRACE:
2011-03-17 15:29:43,277 ERROR nova.compute.manager [X-XU2QSS54CYE169SRCB project admin-project] instance 21: Failed to spawn
(nova.compute.manager): TRACE: Traceback (most recent call last):
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/compute/manager.py", line 218, in run_instance
(nova.compute.manager): TRACE: self.driver.spawn(instance_ref)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/exception.py", line 126, in _wrap
(nova.compute.manager): TRACE: raise Error(str(e))
(nova.compute.manager): TRACE: Error: invalid literal for int() with base 10: 'ami-zmpaoemm'

Cheers,

Wayne

I have the same problem. I tried to manually copy the files inside _base directory from cloud controller to compute host and the instance go in running mode, but now I have a problem with the lan. I don't think the 2 problems are correlated.
Bye,
Alberto

I read http://wiki.openstack.org/LiveMigrationUsage about live migration between multiple nova servers: in the first part of article they make a NFS exprot of instance directory. At this point I don't think that _base directory will be copied automatically from hosts...

Thanks John O'loughlin, that solved my question.

Looks like there is some issue with the image service you are using.

The suggestion on this bug might be helpful:

https://bugs.launchpad.net/nova/+bug/736754

You need to use GlanceImageService for multiple nodes. You also may need to convert your existing images from the old format using
nova-manage image convert /path/to/image/dir

On Mar 20, 2011, at 7:02 PM, KiRaDo wrote:

> Question #149424 on OpenStack Compute (nova) changed:
> https://answers.launchpad.net/nova/+question/149424
>
> Status: Answered => Solved
>
> KiRaDo confirmed that the question is solved:
> Thanks John O'loughlin, that solved my question.
>
> --
> You received this question notification because you are a member of Nova
> Core, which is an answer contact for OpenStack Compute (nova).

Ok thanks, I'll look at the glance service.

Previously, as per the current documentation, I had nova-objectstore installed on the 'cloud controller' and then used the --s3_host in nova.conf to point the compute nodes at it.

So the recommendation is to install glance and run this in place of nova-objectstore ?