devstack

Instance creation errors out with all flavors other than tiny

Asked by Vivekanandan Prakasam

Instance creation errors out with all flavors other than tiny. Only tiny flavor is working. I can understand there is some problem with allocating disk but I couldn't really figureout whats wrong can anyone help?

Question information

Language:
English Edit question
Status:
Answered
For:
devstack Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Bryce Kerley (bkerley) said : 		#1

I'm having the same issue, also on devstack. I'm on Ubuntu 12.04.1 LTS 64-bit.

The error from my instance_faults table in mysql nova is:

| 2013-02-07 16:30:40 | NULL | NULL | 16 | 372ab34a-3ec2-4715-84ee-63abaf77b93f | 5
00 | OSError | [Errno 2] No such file or directory
  File "/opt/stack/nova/nova/compute/manager.py", line 690, in _run_instance
    injected_files, admin_password)
  File "/opt/stack/nova/nova/compute/manager.py", line 962, in _spawn
    block_device_info)
  File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1059, in spawn
    admin_pass=admin_password)
  File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1330, in _create_image
    project_id=instance['project_id'])
  File "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 130, in cache
    *args, **kwargs)
  File "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 188, in create_image
    copy_qcow2_image(base, self.path, size)
  File "/opt/stack/nova/nova/openstack/common/lockutils.py", line 221, in inner
    retval = f(*args, **kwargs)
  File "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 184, in copy_qcow2_image
    disk.extend(target, size)
  File "/opt/stack/nova/nova/virt/disk/api.py", line 125, in extend
    resize2fs(image)
  File "/opt/stack/nova/nova/virt/disk/api.py", line 102, in resize2fs
    run_as_root=run_as_root)
  File "/opt/stack/nova/nova/utils.py", line 220, in execute
    shell=shell)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/green/subprocess.py", line 25, in __init
__
    subprocess_orig.Popen.__init__(self, args, 0, *argss, **kwds)
  File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child
    raise child_exception
 | bryce-openstack | 0 |

Revision history for this message
Bryce Kerley (bkerley) said : 		#2

Hmm, stracing the openstack-compute process and its children gives me this:

./create.strace.7035:execve("/usr/local/bin/qemu-img", ["qemu-img", "resize", "/opt/stack/data/nova/instances/instance-00000007/disk", "85899345920"], [/* 24 vars */]) = -1 ENOENT (No such file or directory)
./create.strace.7035:execve("/usr/bin/qemu-img", ["qemu-img", "resize", "/opt/stack/data/nova/instances/instance-00000007/disk", "85899345920"], [/* 24 vars */]) = 0
./create.strace.7035:write(1, "Image resized.\n", 15) = 15

The strace command for reference:

sudo strace -o create.strace -ff -p 30877 -s 128

30877 is the pid for openstack-compute. Find it using ps.

Revision history for this message
Bryce Kerley (bkerley) said : 		#3

vkmc in irc suggested using the screen log with "screen -r stack"

The tab labeled "n-cpu" is where the exciting logs are. You may need to use "ctrl-a [" and pgup to find the exciting bits.

2013-02-07 20:10:48.279 DEBUG nova.openstack.common.lockutils [req-2bc3f017-ee94-4062-9788-d6872d973412 admin demo] Got file lock "/opt/stack/data/nova/instances/_base/60e13c0dc2b35fadd35c9e77772a25a2317c172e" at /opt/stack/data/nova/instances/locks/n
ova-_opt_stack_data_nova_instances__base_60e13c0dc2b35fadd35c9e77772a25a2317c172e for method "copy_qcow2_image"... from (pid=30877) inner /opt/stack/nova/nova/openstack/common/lockutils.py:220
2013-02-07 20:10:48.280 DEBUG nova.utils [req-2bc3f017-ee94-4062-9788-d6872d973412 admin demo] Running cmd (subprocess): env LC_ALL=C LANG=C qemu-img info /opt/stack/data/nova/instances/_base/60e13c0dc2b35fadd35c9e77772a25a2317c172e from (pid=30877) execute /opt/stack/nova/nova/utils.py:206
2013-02-07 20:10:48.306 DEBUG nova.utils [req-2bc3f017-ee94-4062-9788-d6872d973412 admin demo] Result was 0 from (pid=30877) execute /opt/stack/nova/nova/utils.py:230
2013-02-07 20:10:48.307 DEBUG nova.utils [req-2bc3f017-ee94-4062-9788-d6872d973412 admin demo] Running cmd (subprocess): qemu-img create -f qcow2 -o backing_file=/opt/stack/data/nova/instances/_base/60e13c0dc2b35fadd35c9e77772a25a2317c172e /opt/stack/data/nova/instances/instance-0000000b/disk from (pid=30877) execute /opt/stack/nova/nova/utils.py:206
2013-02-07 20:10:48.499 DEBUG nova.utils [req-2bc3f017-ee94-4062-9788-d6872d973412 admin demo] Result was 0 from (pid=30877) execute /opt/stack/nova/nova/utils.py:230
2013-02-07 20:10:48.501 DEBUG nova.utils [req-2bc3f017-ee94-4062-9788-d6872d973412 admin demo] Running cmd (subprocess): env LC_ALL=C LANG=C qemu-img info /opt/stack/data/nova/instances/instance-0000000b/disk from (pid=30877) execute /opt/stack/nova/nova/utils.py:206
2013-02-07 20:10:48.603 DEBUG nova.utils [req-2bc3f017-ee94-4062-9788-d6872d973412 admin demo] Result was 0 from (pid=30877) execute /opt/stack/nova/nova/utils.py:230
2013-02-07 20:10:48.605 DEBUG nova.utils [req-2bc3f017-ee94-4062-9788-d6872d973412 admin demo] Running cmd (subprocess): qemu-img resize /opt/stack/data/nova/instances/instance-0000000b/disk 85899345920 from (pid=30877) execute /opt/stack/nova/nova/utils.py:206
2013-02-07 20:10:48.748 DEBUG nova.utils [req-2bc3f017-ee94-4062-9788-d6872d973412 admin demo] Result was 0 from (pid=30877) execute /opt/stack/nova/nova/utils.py:230
2013-02-07 20:10:48.749 DEBUG nova.utils [req-2bc3f017-ee94-4062-9788-d6872d973412 admin demo] Running cmd (subprocess): e2fsck -fp /opt/stack/data/nova/instances/instance-0000000b/disk from (pid=30877) execute /opt/stack/nova/nova/utils.py:206
2013-02-07 20:10:48.770 DEBUG nova.openstack.common.lockutils [req-2bc3f017-ee94-4062-9788-d6872d973412 admin demo] Released file lock "/opt/stack/data/nova/instances/_base/60e13c0dc2b35fadd35c9e77772a25a2317c172e" at /opt/stack/data/nova/instances/locks/nova-_opt_stack_data_nova_instances__base_60e13c0dc2b35fadd35c9e77772a25a2317c172e for method "copy_qcow2_image"... from (pid=30877) inner /opt/stack/nova/nova/openstack/common/lockutils.py:227
2013-02-07 20:10:48.772 ERROR nova.compute.manager [req-2bc3f017-ee94-4062-9788-d6872d973412 admin demo] [instance: eaeee908-16a9-4ab0-95d4-82cd662476dd] Instance failed to spawn
2013-02-07 20:10:48.772 TRACE nova.compute.manager [instance: eaeee908-16a9-4ab0-95d4-82cd662476dd] Traceback (most recent call last):
2013-02-07 20:10:48.772 TRACE nova.compute.manager [instance: eaeee908-16a9-4ab0-95d4-82cd662476dd] File "/opt/stack/nova/nova/compute/manager.py", line 984, in _spawn
2013-02-07 20:10:48.772 TRACE nova.compute.manager [instance: eaeee908-16a9-4ab0-95d4-82cd662476dd] block_device_info)
2013-02-07 20:10:48.772 TRACE nova.compute.manager [instance: eaeee908-16a9-4ab0-95d4-82cd662476dd] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1184, in spawn
2013-02-07 20:10:48.772 TRACE nova.compute.manager [instance: eaeee908-16a9-4ab0-95d4-82cd662476dd] admin_pass=admin_password)
2013-02-07 20:10:48.772 TRACE nova.compute.manager [instance: eaeee908-16a9-4ab0-95d4-82cd662476dd] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1455, in _create_image
2013-02-07 20:10:48.772 TRACE nova.compute.manager [instance: eaeee908-16a9-4ab0-95d4-82cd662476dd] project_id=instance['project_id'])
2013-02-07 20:10:48.772 TRACE nova.compute.manager [instance: eaeee908-16a9-4ab0-95d4-82cd662476dd] File "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 130, in cache
2013-02-07 20:10:48.772 TRACE nova.compute.manager [instance: eaeee908-16a9-4ab0-95d4-82cd662476dd] *args, **kwargs)
2013-02-07 20:10:48.772 TRACE nova.compute.manager [instance: eaeee908-16a9-4ab0-95d4-82cd662476dd] File "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 188, in create_image
2013-02-07 20:10:48.772 TRACE nova.compute.manager [instance: eaeee908-16a9-4ab0-95d4-82cd662476dd] copy_qcow2_image(base, self.path, size)
2013-02-07 20:10:48.772 TRACE nova.compute.manager [instance: eaeee908-16a9-4ab0-95d4-82cd662476dd] File "/opt/stack/nova/nova/openstack/common/lockutils.py", line 221, in inner
2013-02-07 20:10:48.772 TRACE nova.compute.manager [instance: eaeee908-16a9-4ab0-95d4-82cd662476dd] retval = f(*args, **kwargs)
2013-02-07 20:10:48.772 TRACE nova.compute.manager [instance: eaeee908-16a9-4ab0-95d4-82cd662476dd] File "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 184, in copy_qcow2_image
2013-02-07 20:10:48.772 TRACE nova.compute.manager [instance: eaeee908-16a9-4ab0-95d4-82cd662476dd] disk.extend(target, size)
2013-02-07 20:10:48.772 TRACE nova.compute.manager [instance: eaeee908-16a9-4ab0-95d4-82cd662476dd] File "/opt/stack/nova/nova/virt/disk/api.py", line 125, in extend
2013-02-07 20:10:48.772 TRACE nova.compute.manager [instance: eaeee908-16a9-4ab0-95d4-82cd662476dd] resize2fs(image)
2013-02-07 20:10:48.772 TRACE nova.compute.manager [instance: eaeee908-16a9-4ab0-95d4-82cd662476dd] File "/opt/stack/nova/nova/virt/disk/api.py", line 102, in resize2fs
2013-02-07 20:10:48.772 TRACE nova.compute.manager [instance: eaeee908-16a9-4ab0-95d4-82cd662476dd] run_as_root=run_as_root)
2013-02-07 20:10:48.772 TRACE nova.compute.manager [instance: eaeee908-16a9-4ab0-95d4-82cd662476dd] File "/opt/stack/nova/nova/utils.py", line 222, in execute
2013-02-07 20:10:48.772 TRACE nova.compute.manager [instance: eaeee908-16a9-4ab0-95d4-82cd662476dd] shell=shell)
2013-02-07 20:10:48.772 TRACE nova.compute.manager [instance: eaeee908-16a9-4ab0-95d4-82cd662476dd] File "/usr/local/lib/python2.7/dist-packages/eventlet/green/subprocess.py", line 25, in __init__
2013-02-07 20:10:48.772 TRACE nova.compute.manager [instance: eaeee908-16a9-4ab0-95d4-82cd662476dd] subprocess_orig.Popen.__init__(self, args, 0, *argss, **kwds)
2013-02-07 20:10:48.772 TRACE nova.compute.manager [instance: eaeee908-16a9-4ab0-95d4-82cd662476dd] File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
2013-02-07 20:10:48.772 TRACE nova.compute.manager [instance: eaeee908-16a9-4ab0-95d4-82cd662476dd] errread, errwrite)
2013-02-07 20:10:48.772 TRACE nova.compute.manager [instance: eaeee908-16a9-4ab0-95d4-82cd662476dd] File "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child
2013-02-07 20:10:48.772 TRACE nova.compute.manager [instance: eaeee908-16a9-4ab0-95d4-82cd662476dd] raise child_exception
2013-02-07 20:10:48.772 TRACE nova.compute.manager [instance: eaeee908-16a9-4ab0-95d4-82cd662476dd] OSError: [Errno 2] No such file or directory
2013-02-07 20:10:48.772 TRACE nova.compute.manager [instance: eaeee908-16a9-4ab0-95d4-82cd662476dd]
2013-02-07 20:10:48.775 DEBUG nova.openstack.common.lockutils [req-2bc3f017-ee94-4062-9788-d6872d973412 admin demo] Got semaphore "compute_resources" for method "abort"... from (pid=30877) inner /opt/stack/nova/nova/openstack/common/lockutils.py:186
2013-02-07 20:10:48.776 DEBUG nova.compute.claims [req-2bc3f017-ee94-4062-9788-d6872d973412 admin demo] [instance: eaeee908-16a9-4ab0-95d4-82cd662476dd] Aborting claim: [Claim: 8192 MB memory, 80 GB disk, 4 VCPUS] from (pid=30877) abort /opt/stack/nova/nova/compute/claims.py:94

Revision history for this message
Bryce Kerley (bkerley) said : 		#4

Nova is expecting e2fsck and resize2fs to be in the path, but they're in /sbin in Ubuntu 12.04 unless you're root. Either add /sbin to the "stack" user's path, or symlink them to somewhere that is in the "stack" user's path.

Revision history for this message
JP Espinosa (jp-a) said : 		#5

I had this problem with Openstack on Ubuntu Server and found this question, also this guy had this problem https://lists.launchpad.net/openstack/msg08388.html

I am answering here for reference:

My setup:
Ubuntu 12.04.2 LTS
Openstack 2012.2.2-dev (2012.2.2-LOCALBRANCH:LOCALREVISION)

Full trace: https://gist.github.com/jpea/5135047
--------------------------------------------
2013-03-11 11:22:07 DEBUG nova.utils [req-0918d30d-00b8-45b0-b2b8-a4743c879ac6 50398e8030bb497a9a4bbd1429576ad2 411625da55b546e2ba68b8dc9e111ab6] Result was 0 execute /usr/lib/python2.7/dist-packages/nova/utils.py:203
2013-03-11 11:22:07 DEBUG nova.utils [req-0918d30d-00b8-45b0-b2b8-a4743c879ac6 50398e8030bb497a9a4bbd1429576ad2 411625da55b546e2ba68b8dc9e111ab6] Running cmd (subprocess): qemu-img resize /var/lib/nova/instances/_base/876d0b1240b2bfa3bb04fa14f184db987604d322_40 42949672960 execute /usr/lib/python2.7/dist-packages/nova/utils.py:187
2013-03-11 11:22:07 DEBUG nova.utils [req-0918d30d-00b8-45b0-b2b8-a4743c879ac6 50398e8030bb497a9a4bbd1429576ad2 411625da55b546e2ba68b8dc9e111ab6] Result was 0 execute /usr/lib/python2.7/dist-packages/nova/utils.py:203
2013-03-11 11:22:07 DEBUG nova.utils [req-0918d30d-00b8-45b0-b2b8-a4743c879ac6 50398e8030bb497a9a4bbd1429576ad2 411625da55b546e2ba68b8dc9e111ab6] Running cmd (subprocess): e2fsck -fp /var/lib/nova/instances/_base/876d0b1240b2bfa3bb04fa14f184db987604d322_40 execute /usr/lib/python2.7/dist-packages/nova/utils.py:187
2013-03-11 11:22:07 ERROR nova.compute.manager [req-0918d30d-00b8-45b0-b2b8-a4743c879ac6 50398e8030bb497a9a4bbd1429576ad2 411625da55b546e2ba68b8dc9e111ab6] [instance: ab2bf4b8-b103-4ce6-8b7d-d883a135c4ee] Instance failed to spawn
2013-03-11 11:22:07 1693 TRACE nova.compute.manager [instance: ab2bf4b8-b103-4ce6-8b7d-d883a135c4ee] Traceback (most recent call last):
2013-03-11 11:22:07 1693 TRACE nova.compute.manager [instance: ab2bf4b8-b103-4ce6-8b7d-d883a135c4ee] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 747, in _spawn
2013-03-11 11:22:07 1693 TRACE nova.compute.manager [instance: ab2bf4b8-b103-4ce6-8b7d-d883a135c4ee] block_device_info)
2013-03-11 11:22:07 1693 TRACE nova.compute.manager [instance: ab2bf4b8-b103-4ce6-8b7d-d883a135c4ee] File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
2013-03-11 11:22:07 1693 TRACE nova.compute.manager [instance: ab2bf4b8-b103-4ce6-8b7d-d883a135c4ee] temp_level, payload)
2013-03-11 11:22:07 1693 TRACE nova.compute.manager [instance: ab2bf4b8-b103-4ce6-8b7d-d883a135c4ee] File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-03-11 11:22:07 1693 TRACE nova.compute.manager [instance: ab2bf4b8-b103-4ce6-8b7d-d883a135c4ee] self.gen.next()
2013-03-11 11:22:07 1693 TRACE nova.compute.manager [instance: ab2bf4b8-b103-4ce6-8b7d-d883a135c4ee] File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 92, in wrapped
2013-03-11 11:22:07 1693 TRACE nova.compute.manager [instance: ab2bf4b8-b103-4ce6-8b7d-d883a135c4ee] return f(*args, **kw)
2013-03-11 11:22:07 1693 TRACE nova.compute.manager [instance: ab2bf4b8-b103-4ce6-8b7d-d883a135c4ee] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1090, in spawn
2013-03-11 11:22:07 1693 TRACE nova.compute.manager [instance: ab2bf4b8-b103-4ce6-8b7d-d883a135c4ee] admin_pass=admin_password)
2013-03-11 11:22:07 1693 TRACE nova.compute.manager [instance: ab2bf4b8-b103-4ce6-8b7d-d883a135c4ee] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1356, in _create_image
2013-03-11 11:22:07 1693 TRACE nova.compute.manager [instance: ab2bf4b8-b103-4ce6-8b7d-d883a135c4ee] project_id=instance['project_id'])
2013-03-11 11:22:07 1693 TRACE nova.compute.manager [instance: ab2bf4b8-b103-4ce6-8b7d-d883a135c4ee] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 126, in cache
2013-03-11 11:22:07 1693 TRACE nova.compute.manager [instance: ab2bf4b8-b103-4ce6-8b7d-d883a135c4ee] *args, **kwargs)
2013-03-11 11:22:07 1693 TRACE nova.compute.manager [instance: ab2bf4b8-b103-4ce6-8b7d-d883a135c4ee] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 175, in create_image
2013-03-11 11:22:07 1693 TRACE nova.compute.manager [instance: ab2bf4b8-b103-4ce6-8b7d-d883a135c4ee] copy_qcow2_image(base, self.path, size)
2013-03-11 11:22:07 1693 TRACE nova.compute.manager [instance: ab2bf4b8-b103-4ce6-8b7d-d883a135c4ee] File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 796, in inner
2013-03-11 11:22:07 1693 TRACE nova.compute.manager [instance: ab2bf4b8-b103-4ce6-8b7d-d883a135c4ee] retval = f(*args, **kwargs)
2013-03-11 11:22:07 1693 TRACE nova.compute.manager [instance: ab2bf4b8-b103-4ce6-8b7d-d883a135c4ee] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 170, in copy_qcow2_image
2013-03-11 11:22:07 1693 TRACE nova.compute.manager [instance: ab2bf4b8-b103-4ce6-8b7d-d883a135c4ee] disk.extend(qcow2_base, size)
2013-03-11 11:22:07 1693 TRACE nova.compute.manager [instance: ab2bf4b8-b103-4ce6-8b7d-d883a135c4ee] File "/usr/lib/python2.7/dist-packages/nova/virt/disk/api.py", line 127, in extend
2013-03-11 11:22:07 1693 TRACE nova.compute.manager [instance: ab2bf4b8-b103-4ce6-8b7d-d883a135c4ee] resize2fs(image)
2013-03-11 11:22:07 1693 TRACE nova.compute.manager [instance: ab2bf4b8-b103-4ce6-8b7d-d883a135c4ee] File "/usr/lib/python2.7/dist-packages/nova/virt/disk/api.py", line 104, in resize2fs
2013-03-11 11:22:07 1693 TRACE nova.compute.manager [instance: ab2bf4b8-b103-4ce6-8b7d-d883a135c4ee] utils.execute('e2fsck', '-fp', image, check_exit_code=check_exit_code)
2013-03-11 11:22:07 1693 TRACE nova.compute.manager [instance: ab2bf4b8-b103-4ce6-8b7d-d883a135c4ee] File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 195, in execute
2013-03-11 11:22:07 1693 TRACE nova.compute.manager [instance: ab2bf4b8-b103-4ce6-8b7d-d883a135c4ee] shell=shell)
2013-03-11 11:22:07 1693 TRACE nova.compute.manager [instance: ab2bf4b8-b103-4ce6-8b7d-d883a135c4ee] File "/usr/lib/python2.7/dist-packages/eventlet/green/subprocess.py", line 25, in __init__
2013-03-11 11:22:07 1693 TRACE nova.compute.manager [instance: ab2bf4b8-b103-4ce6-8b7d-d883a135c4ee] subprocess_orig.Popen.__init__(self, args, 0, *argss, **kwds)
2013-03-11 11:22:07 1693 TRACE nova.compute.manager [instance: ab2bf4b8-b103-4ce6-8b7d-d883a135c4ee] File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
2013-03-11 11:22:07 1693 TRACE nova.compute.manager [instance: ab2bf4b8-b103-4ce6-8b7d-d883a135c4ee] errread, errwrite)
2013-03-11 11:22:07 1693 TRACE nova.compute.manager [instance: ab2bf4b8-b103-4ce6-8b7d-d883a135c4ee] File "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child
2013-03-11 11:22:07 1693 TRACE nova.compute.manager [instance: ab2bf4b8-b103-4ce6-8b7d-d883a135c4ee] raise child_exception
2013-03-11 11:22:07 1693 TRACE nova.compute.manager [instance: ab2bf4b8-b103-4ce6-8b7d-d883a135c4ee] OSError: [Errno 2] No such file or directory
2013-03-11 11:22:07 1693 TRACE nova.compute.manager [instance: ab2bf4b8-b103-4ce6-8b7d-d883a135c4ee]
--------------------------------------------

As suggested Bryce Kerley, this fixed it:

ln -s /sbin/e2fsck /usr/bin/e2fsck
ln -s /sbin/resize2fs /usr/bin/resize2fs

Can you help with this problem?

Provide an answer of your own, or ask Vivekanandan Prakasam for more information if necessary.

To post a message you must log in.