OpenStack Compute (nova)

euca-run-instancess produce vm in indefinite spawning state

Asked by Armando Migliaccio on 2010-07-28

I have checked out revision 182 of nova and I was folliwing instructions available at http://wiki.openstack.org/InstallFromSource.

I got everything running on a physical machine, however when I try to launch an instance, the instance is stuck in the spawning state as shown by euca-describe-instances (see below). I have copied the output of each node below, hoping that you could point me in the right direction in troubleshooting this. Many thanks!

armandomi@armandomi-cieg:~/openstack/nova/trunk182$ euca-describe-instances
RESERVATION r-wpo4fi31 novaproject default
INSTANCE i-ze0bnh1q ami-8jcw97ou 192.168.1.222 192.168.1.222 spawning (novaproject, armandomi-cieg) 0 m1.small 2010-07-28T16:11:41Z

*******************************************************************************************
******************************LOGGED INFORMATION *****************************
*******************************************************************************************
***************************************NOVA-API ***************************************
root@armandomi-cieg:~/openstack/nova# nova-api ${NOVA_API_ARGS} --nodaemonize --verbose
DEBUG:root:Full set of FLAGS:
DEBUG:root:storage_availability_zone : nova
DEBUG:root:help : None
DEBUG:root:cloudpipe_start_port : 12000
DEBUG:root:compute_topic : compute
DEBUG:root:keys_path : /home/armandomi/openstack/nova/trunk182/../data/keys
DEBUG:root:rabbit_password : guest
DEBUG:root:simple_network_bridge : br0
DEBUG:root:public_interface : vlan1
DEBUG:root:network_size : 256
DEBUG:root:vpn_start_port : 1000
DEBUG:root:simple_network_ips : ['192.168.1.220', '192.168.1.221', '192.168.1.222']
DEBUG:root:helpxml : None
DEBUG:root:simple_network : True
DEBUG:root:aoe_eth_dev : eth0
DEBUG:root:rabbit_port : 5672
DEBUG:root:logfile : None
DEBUG:root:last_shelf_id : 279
DEBUG:root:redis_host : 127.0.0.1
DEBUG:root:? : None
DEBUG:root:default_instance_type : m1.small
DEBUG:root:volume_topic : volume
DEBUG:root:public_range : 4.4.4.0/24
DEBUG:root:use_vpn : True
DEBUG:root:working_directory : ./
DEBUG:root:ec2_url : http://127.0.0.1:8773/services/Cloud
DEBUG:root:storage_dev : /dev/sdb
DEBUG:root:s3_port : 3333
DEBUG:root:fake_libvirt : False
DEBUG:root:fake_storage : False
DEBUG:root:redis_db : 0
DEBUG:root:credential_key_file : pk.pem
DEBUG:root:pidfile : nova-api.pid
DEBUG:root:rabbit_userid : guest
DEBUG:root:simple_network_gateway : 192.168.1.1
DEBUG:root:buckets_path : /home/armandomi/openstack/nova/trunk182/nova/../buckets
DEBUG:root:instances_path : /home/armandomi/openstack/nova/trunk182/nova/../instances
DEBUG:root:storage_name : armandomi-cieg
DEBUG:root:node_name : armandomi-cieg
DEBUG:root:networks_path : /home/armandomi/openstack/nova/trunk182/nova/../networks
DEBUG:root:daemonize : False
DEBUG:root:s3_host : 127.0.0.1
DEBUG:root:aoe_export_dir : /var/lib/vblade-persist/vblades
DEBUG:root:images_path : /home/armandomi/openstack/nova/trunk182/nova/../images
DEBUG:root:cloud_topic : cloud
DEBUG:root:verbose : True
DEBUG:root:default_kernel : aki-11111
DEBUG:root:vpn_key_suffix : -key
DEBUG:root:libvirt_xml_template : /home/armandomi/openstack/nova/trunk182/nova/compute/libvirt.xml.template
DEBUG:root:vpn_client_template : /home/armandomi/openstack/nova/trunk182/nova/cloudpipe/client.ovpn.template
DEBUG:root:process_pool_size : 4
DEBUG:root:slots_per_shelf : 16
DEBUG:root:volume_group : nova-volumes
DEBUG:root:simple_network_broadcast : 192.168.1.255
DEBUG:root:credential_cert_file : cert.pem
DEBUG:root:simple_network_netmask : 255.255.255.0
DEBUG:root:simple_network_dns : 8.8.4.4
DEBUG:root:vpn_end_port : 2000
DEBUG:root:report_interval : 10
DEBUG:root:helpshort : None
DEBUG:root:ca_file : cacert.pem
DEBUG:root:use_intermediate_ca : False
DEBUG:root:vpn_image_id : ami-CLOUDPIPE
DEBUG:root:public_vlan : 1
DEBUG:root:use_s3 : True
DEBUG:root:private_range : 10.0.0.0/8
DEBUG:root:credentials_template : /home/armandomi/openstack/nova/trunk182/nova/auth/novarc.template
DEBUG:root:vpn_ip : 127.0.0.1
DEBUG:root:default_image : ami-11111
DEBUG:root:credential_rc_file : novarc
DEBUG:root:fake_network : False
DEBUG:root:first_shelf_id : 270
DEBUG:root:dhcpbridge_flagfile : /etc/nova/nova-dhcpbridge.conf
DEBUG:root:vlan_end : 4093
DEBUG:root:credential_cert_subject : /C=US/ST=California/L=MountainView/O=AnsoLabs/OU=NovaDev/CN=%s-%s
DEBUG:root:auth_driver : nova.auth.ldapdriver.FakeLdapDriver
DEBUG:root:cc_port : 8773
DEBUG:root:bridge_dev : eth1
DEBUG:root:fake_rabbit : False
DEBUG:root:rabbit_host : localhost
DEBUG:root:simple_network_network : 192.168.1.0
DEBUG:root:use_syslog : True
DEBUG:root:superuser_roles : ['cloudadmin']
DEBUG:root:control_exchange : nova
DEBUG:root:simple_network_template : /home/armandomi/openstack/nova/trunk182/nova/compute/interfaces.template
DEBUG:root:default_ramdisk : ari-11111
DEBUG:root:node_availability_zone : nova
DEBUG:root:vlan_start : 100
DEBUG:root:global_roles : ['cloudadmin', 'itsec']
DEBUG:root:rabbit_virtual_host : /
DEBUG:root:ca_path : /home/armandomi/openstack/nova/trunk182/CA
DEBUG:root:cnt_vpn_clients : 5
DEBUG:root:redis_port : 6379
DEBUG:root:network_topic : network
DEBUG:root:Started HTTP server on 8773
**************************************************************************************************
***************************************NOVA-VOLUME ***************************************
root@armandomi-cieg:~/openstack/nova# nova-volume ${NOVA_VOLUME_ARGS} --verbose --nodaemon
DEBUG:root:Full set of FLAGS:
DEBUG:root:storage_availability_zone : nova
DEBUG:root:volume_topic : volume
DEBUG:root:verbose : True
DEBUG:root:savestats : None
DEBUG:root:encrypted : None
DEBUG:root:compute_topic : compute
DEBUG:root:default_kernel : aki-11111
DEBUG:root:report_profile : None
DEBUG:root:rabbit_password : guest
DEBUG:root:syslog : None
DEBUG:root:prefix : nova-volume
DEBUG:root:vpn_key_suffix : -key
DEBUG:root:ec2_url : http://127.0.0.1:8773/services/Cloud
DEBUG:root:originalname : None
DEBUG:root:rundir : .
DEBUG:root:profiler : hotshot
DEBUG:root:help : None
DEBUG:root:storage_dev : /dev/loop0
DEBUG:root:fake_rabbit : False
DEBUG:root:s3_port : 3333
DEBUG:root:help_reactors : None
DEBUG:root:rabbit_host : localhost
DEBUG:root:source : None
DEBUG:root:process_pool_size : 4
DEBUG:root:fake_libvirt : False
DEBUG:root:umask : None
DEBUG:root:nothotshot : None
DEBUG:root:debug : False
DEBUG:root:fake_storage : False
DEBUG:root:redis_db : 0
DEBUG:root:gid : None
DEBUG:root:slots_per_shelf : 16
DEBUG:root:volume_group : nova-volumes
DEBUG:root:pidfile : /home/armandomi/openstack/nova/nova-volume.pid
DEBUG:root:reactor : None
DEBUG:root:? : None
DEBUG:root:helpxml : None
DEBUG:root:rabbit_userid : guest
DEBUG:root:file : twistd.tap
DEBUG:root:default_instance_type : m1.small
DEBUG:root:report_interval : 10
DEBUG:root:helpshort : None
DEBUG:root:node_availability_zone : nova
DEBUG:root:version : None
DEBUG:root:aoe_eth_dev : eth0
DEBUG:root:rabbit_port : 5672
DEBUG:root:chroot : None
DEBUG:root:profile : None
DEBUG:root:euid : None
DEBUG:root:vpn_image_id : ami-CLOUDPIPE
DEBUG:root:logfile : nova-volume.log
DEBUG:root:uid : None
DEBUG:root:nodaemon : True
DEBUG:root:b : None
DEBUG:root:last_shelf_id : 279
DEBUG:root:storage_name : armandomi-cieg
DEBUG:root:no_save : True
DEBUG:root:aoe_export_dir : /var/lib/vblade-persist/vblades
DEBUG:root:rabbit_virtual_host : /
DEBUG:root:node_name : armandomi-cieg
DEBUG:root:redis_host : 127.0.0.1
DEBUG:root:spew : None
DEBUG:root:r : None
DEBUG:root:default_image : ami-11111
DEBUG:root:control_exchange : nova
DEBUG:root:default_ramdisk : ari-11111
DEBUG:root:redis_port : 6379
DEBUG:root:s3_host : 127.0.0.1
DEBUG:root:python : /home/armandomi/openstack/nova/trunk182/bin/nova-volume
DEBUG:root:first_shelf_id : 270
DEBUG:root:fake_network : False
DEBUG:root:network_topic : network
WARNING:root:Starting volume node
***********************************************************************************************************
***************************************NOVA-OBJECTSTORE ***************************************
root@armandomi-cieg:~/openstack/nova/data/objectstorelogs# nova-objectstore ${NOVA_OBJECTSTORE_ARGS} --verbose --nodaemon
DEBUG:root:Serving /home/armandomi/openstack/nova/trunk182/bin/nova-objectstore
DEBUG:root:Full set of FLAGS:
DEBUG:root:storage_availability_zone : nova
DEBUG:root:help : None
DEBUG:root:cloudpipe_start_port : 12000
DEBUG:root:encrypted : None
DEBUG:root:compute_topic : compute
DEBUG:root:keys_path : /home/armandomi/openstack/nova/trunk182/../data/keys
DEBUG:root:report_profile : None
DEBUG:root:rabbit_password : guest
DEBUG:root:prefix : nova-objectstore
DEBUG:root:originalname : None
DEBUG:root:simple_network_bridge : br0
DEBUG:root:public_interface : vlan1
DEBUG:root:reactor : None
DEBUG:root:network_size : 256
DEBUG:root:vpn_start_port : 1000
DEBUG:root:spew : None
DEBUG:root:umask : None
DEBUG:root:source : None
DEBUG:root:nothotshot : None
DEBUG:root:simple_network_ips : ['192.168.1.220', '192.168.1.221', '192.168.1.222']
DEBUG:root:helpxml : None
DEBUG:root:simple_network : True
DEBUG:root:python : /home/armandomi/openstack/nova/trunk182/bin/nova-objectstore
DEBUG:root:aoe_eth_dev : eth0
DEBUG:root:rabbit_port : 5672
DEBUG:root:euid : None
DEBUG:root:logfile : nova-objectstore.log
DEBUG:root:nodaemon : True
DEBUG:root:last_shelf_id : 279
DEBUG:root:redis_host : 127.0.0.1
DEBUG:root:pidfile : /home/armandomi/openstack/nova/data/objectstorelogs/nova-objectstore.pid
DEBUG:root:default_instance_type : m1.small
DEBUG:root:volume_topic : volume
DEBUG:root:public_range : 4.4.4.0/24
DEBUG:root:use_vpn : True
DEBUG:root:syslog : None
DEBUG:root:ec2_url : http://127.0.0.1:8773/services/Cloud
DEBUG:root:rundir : .
DEBUG:root:storage_dev : /dev/sdb
DEBUG:root:s3_port : 3333
DEBUG:root:fake_libvirt : False
DEBUG:root:fake_storage : False
DEBUG:root:redis_db : 0
DEBUG:root:version : None
DEBUG:root:credential_key_file : pk.pem
DEBUG:root:? : None
DEBUG:root:rabbit_userid : guest
DEBUG:root:simple_network_gateway : 192.168.1.1
DEBUG:root:buckets_path : /home/armandomi/openstack/nova/trunk182/../data/buckets
DEBUG:root:instances_path : /home/armandomi/openstack/nova/trunk182/nova/../instances
DEBUG:root:debug : False
DEBUG:root:storage_name : armandomi-cieg
DEBUG:root:node_name : armandomi-cieg
DEBUG:root:networks_path : /home/armandomi/openstack/nova/trunk182/nova/../networks
DEBUG:root:s3_host : 127.0.0.1
DEBUG:root:aoe_export_dir : /var/lib/vblade-persist/vblades
DEBUG:root:images_path : /home/armandomi/openstack/nova/trunk182/../data/images
DEBUG:root:cloud_topic : cloud
DEBUG:root:verbose : True
DEBUG:root:default_kernel : aki-11111
DEBUG:root:vpn_key_suffix : -key
DEBUG:root:libvirt_xml_template : /home/armandomi/openstack/nova/trunk182/nova/compute/libvirt.xml.template
DEBUG:root:vpn_client_template : /home/armandomi/openstack/nova/trunk182/nova/cloudpipe/client.ovpn.template
DEBUG:root:process_pool_size : 4
DEBUG:root:gid : None
DEBUG:root:slots_per_shelf : 16
DEBUG:root:volume_group : nova-volumes
DEBUG:root:simple_network_broadcast : 192.168.1.255
DEBUG:root:credential_cert_file : cert.pem
DEBUG:root:simple_network_netmask : 255.255.255.0
DEBUG:root:simple_network_dns : 8.8.4.4
DEBUG:root:vpn_end_port : 2000
DEBUG:root:report_interval : 10
DEBUG:root:helpshort : None
DEBUG:root:ca_file : cacert.pem
DEBUG:root:use_intermediate_ca : False
DEBUG:root:vpn_image_id : ami-CLOUDPIPE
DEBUG:root:public_vlan : 1
DEBUG:root:use_s3 : True
DEBUG:root:b : None
DEBUG:root:private_range : 10.0.0.0/8
DEBUG:root:credentials_template : /home/armandomi/openstack/nova/trunk182/nova/auth/novarc.template
DEBUG:root:profiler : hotshot
DEBUG:root:vpn_ip : 127.0.0.1
DEBUG:root:r : None
DEBUG:root:default_image : ami-11111
DEBUG:root:savestats : None
DEBUG:root:credential_rc_file : novarc
DEBUG:root:fake_network : False
DEBUG:root:first_shelf_id : 270
DEBUG:root:dhcpbridge_flagfile : /etc/nova/nova-dhcpbridge.conf
DEBUG:root:uid : None
DEBUG:root:vlan_end : 4093
DEBUG:root:credential_cert_subject : /C=US/ST=California/L=MountainView/O=AnsoLabs/OU=NovaDev/CN=%s-%s
DEBUG:root:auth_driver : nova.auth.ldapdriver.FakeLdapDriver
DEBUG:root:cc_port : 8773
DEBUG:root:file : twistd.tap
DEBUG:root:bridge_dev : eth1
DEBUG:root:fake_rabbit : False
DEBUG:root:chroot : None
DEBUG:root:help_reactors : None
DEBUG:root:rabbit_host : localhost
DEBUG:root:simple_network_network : 192.168.1.0
DEBUG:root:superuser_roles : ['cloudadmin']
DEBUG:root:control_exchange : nova
DEBUG:root:profile : None
DEBUG:root:simple_network_template : /home/armandomi/openstack/nova/trunk182/nova/compute/interfaces.template
DEBUG:root:default_ramdisk : ari-11111
DEBUG:root:node_availability_zone : nova
DEBUG:root:vlan_start : 100
DEBUG:root:global_roles : ['cloudadmin', 'itsec']
DEBUG:root:no_save : True
DEBUG:root:rabbit_virtual_host : /
DEBUG:root:ca_path :
DEBUG:root:cnt_vpn_clients : 5
DEBUG:root:redis_port : 6379
DEBUG:root:network_topic : network
DEBUG:root:Started HTTP server on 3333
<twisted.python.components.Componentized instance at 0x8f4164c>
************** AFTER UPLOADING EUCA-UPLOAD-BUNDLE *******************
INFO:root:Looking up user: '3f7e2c7f-d5af-4150-a404-0315b39c0256'
INFO:root:user: User('novauser', 'novauser', '3f7e2c7f-d5af-4150-a404-0315b39c0256', '71095e77-16aa-4e39-8416-a84259bfc84d', True)
DEBUG:root:user.secret: 71095e77-16aa-4e39-8416-a84259bfc84d
DEBUG:root:expected_signature: CdFhlv2zhV5RxYskTriaJpg15DI=
DEBUG:root:signature: CdFhlv2zhV5RxYskTriaJpg15DI=
DEBUG:root:List keys for bucket newbucket
INFO:root:Looking up user: '3f7e2c7f-d5af-4150-a404-0315b39c0256'
INFO:root:user: User('novauser', 'novauser', '3f7e2c7f-d5af-4150-a404-0315b39c0256', '71095e77-16aa-4e39-8416-a84259bfc84d', True)
DEBUG:root:user.secret: 71095e77-16aa-4e39-8416-a84259bfc84d
DEBUG:root:expected_signature: MAuOG/p0+vbeQXSmmJZRno9mKoo=
DEBUG:root:signature: MAuOG/p0+vbeQXSmmJZRno9mKoo=
DEBUG:root:Creating bucket newbucket
DEBUG:root:calling bucket.Bucket.create('newbucket', <nova.endpoint.api.APIRequestContext object at 0x8f418ec>)
INFO:root:Looking up user: '3f7e2c7f-d5af-4150-a404-0315b39c0256'
INFO:root:user: User('novauser', 'novauser', '3f7e2c7f-d5af-4150-a404-0315b39c0256', '71095e77-16aa-4e39-8416-a84259bfc84d', True)
DEBUG:root:user.secret: 71095e77-16aa-4e39-8416-a84259bfc84d
DEBUG:root:expected_signature: Ie3hG+k3BlpJQZAF1hlaCvgx+UU=
DEBUG:root:signature: Ie3hG+k3BlpJQZAF1hlaCvgx+UU=
DEBUG:root:Putting object: newbucket / vmlinuz-2.6.32-23-server.manifest.xml
INFO:root:Looking up user: '3f7e2c7f-d5af-4150-a404-0315b39c0256'
INFO:root:user: User('novauser', 'novauser', '3f7e2c7f-d5af-4150-a404-0315b39c0256', '71095e77-16aa-4e39-8416-a84259bfc84d', True)
DEBUG:root:user.secret: 71095e77-16aa-4e39-8416-a84259bfc84d
DEBUG:root:expected_signature: uZI22wkeDVYxl1dSIaVlvQrBuiE=
DEBUG:root:signature: uZI22wkeDVYxl1dSIaVlvQrBuiE=
DEBUG:root:Putting object: newbucket / vmlinuz-2.6.32-23-server.part.0
INFO:root:Looking up user: '3f7e2c7f-d5af-4150-a404-0315b39c0256'
INFO:root:user: User('novauser', 'novauser', '3f7e2c7f-d5af-4150-a404-0315b39c0256', '71095e77-16aa-4e39-8416-a84259bfc84d', True)
DEBUG:root:user.secret: 71095e77-16aa-4e39-8416-a84259bfc84d
DEBUG:root:expected_signature: CdFhlv2zhV5RxYskTriaJpg15DI=
DEBUG:root:signature: CdFhlv2zhV5RxYskTriaJpg15DI=
DEBUG:root:List keys for bucket newbucket
INFO:root:Looking up user: '3f7e2c7f-d5af-4150-a404-0315b39c0256'
INFO:root:user: User('novauser', 'novauser', '3f7e2c7f-d5af-4150-a404-0315b39c0256', '71095e77-16aa-4e39-8416-a84259bfc84d', True)
DEBUG:root:user.secret: 71095e77-16aa-4e39-8416-a84259bfc84d
DEBUG:root:expected_signature: 2ZraO4swLt/OQXg6J5WyHk1CsQU=
DEBUG:root:signature: 2ZraO4swLt/OQXg6J5WyHk1CsQU=
DEBUG:root:Putting object: newbucket / initrd.img-2.6.32-23-server.manifest.xml
INFO:root:Looking up user: '3f7e2c7f-d5af-4150-a404-0315b39c0256'
INFO:root:user: User('novauser', 'novauser', '3f7e2c7f-d5af-4150-a404-0315b39c0256', '71095e77-16aa-4e39-8416-a84259bfc84d', True)
DEBUG:root:user.secret: 71095e77-16aa-4e39-8416-a84259bfc84d
DEBUG:root:expected_signature: 28ZaszzN2RKQqwUIkI71RKK1SAU=
DEBUG:root:signature: 28ZaszzN2RKQqwUIkI71RKK1SAU=
DEBUG:root:Putting object: newbucket / initrd.img-2.6.32-23-server.part.0
INFO:root:Looking up user: '3f7e2c7f-d5af-4150-a404-0315b39c0256'
INFO:root:user: User('novauser', 'novauser', '3f7e2c7f-d5af-4150-a404-0315b39c0256', '71095e77-16aa-4e39-8416-a84259bfc84d', True)
DEBUG:root:user.secret: 71095e77-16aa-4e39-8416-a84259bfc84d
DEBUG:root:expected_signature: mSGq9CO+5ve6ywkeLPf77Ytold8=
DEBUG:root:signature: mSGq9CO+5ve6ywkeLPf77Ytold8=
DEBUG:root:List keys for bucket newbucket
INFO:root:Looking up user: '3f7e2c7f-d5af-4150-a404-0315b39c0256'
INFO:root:user: User('novauser', 'novauser', '3f7e2c7f-d5af-4150-a404-0315b39c0256', '71095e77-16aa-4e39-8416-a84259bfc84d', True)
DEBUG:root:user.secret: 71095e77-16aa-4e39-8416-a84259bfc84d
DEBUG:root:expected_signature: zDqzpLrqVHdTgzrfCH2tpLXwVpA=
DEBUG:root:signature: zDqzpLrqVHdTgzrfCH2tpLXwVpA=
DEBUG:root:Putting object: newbucket / ubuntu-lucid.img.manifest.xml
INFO:root:Looking up user: '3f7e2c7f-d5af-4150-a404-0315b39c0256'
INFO:root:user: User('novauser', 'novauser', '3f7e2c7f-d5af-4150-a404-0315b39c0256', '71095e77-16aa-4e39-8416-a84259bfc84d', True)
DEBUG:root:user.secret: 71095e77-16aa-4e39-8416-a84259bfc84d
DEBUG:root:expected_signature: TextXrKFuACj6psJGmqeDgmcqaI=
DEBUG:root:signature: TextXrKFuACj6psJGmqeDgmcqaI=
DEBUG:root:Putting object: newbucket / ubuntu-lucid.img.part.0
INFO:root:Looking up user: '3f7e2c7f-d5af-4150-a404-0315b39c0256'
INFO:root:user: User('novauser', 'novauser', '3f7e2c7f-d5af-4150-a404-0315b39c0256', '71095e77-16aa-4e39-8416-a84259bfc84d', True)
DEBUG:root:user.secret: 71095e77-16aa-4e39-8416-a84259bfc84d
DEBUG:root:expected_signature: bRynTXJ3DoOLnJahUgA/mBfCLGw=
DEBUG:root:signature: bRynTXJ3DoOLnJahUgA/mBfCLGw=
DEBUG:root:Putting object: newbucket / ubuntu-lucid.img.part.1
INFO:root:Looking up user: '3f7e2c7f-d5af-4150-a404-0315b39c0256'
INFO:root:user: User('novauser', 'novauser', '3f7e2c7f-d5af-4150-a404-0315b39c0256', '71095e77-16aa-4e39-8416-a84259bfc84d', True)
DEBUG:root:user.secret: 71095e77-16aa-4e39-8416-a84259bfc84d
DEBUG:root:expected_signature: Wc9O04LvXJTtMAUzZGcCER1tZk8=
DEBUG:root:signature: Wc9O04LvXJTtMAUzZGcCER1tZk8=
DEBUG:root:Putting object: newbucket / ubuntu-lucid.img.part.2
INFO:root:Looking up user: '3f7e2c7f-d5af-4150-a404-0315b39c0256'
INFO:root:user: User('novauser', 'novauser', '3f7e2c7f-d5af-4150-a404-0315b39c0256', '71095e77-16aa-4e39-8416-a84259bfc84d', True)
DEBUG:root:user.secret: 71095e77-16aa-4e39-8416-a84259bfc84d
DEBUG:root:expected_signature: 58EKQ/p3nUsvZM7+QlyrxFE3Xz8=
DEBUG:root:signature: 58EKQ/p3nUsvZM7+QlyrxFE3Xz8=
DEBUG:root:Putting object: newbucket / ubuntu-lucid.img.part.3
INFO:root:Looking up user: '3f7e2c7f-d5af-4150-a404-0315b39c0256'
INFO:root:user: User('novauser', 'novauser', '3f7e2c7f-d5af-4150-a404-0315b39c0256', '71095e77-16aa-4e39-8416-a84259bfc84d', True)
DEBUG:root:user.secret: 71095e77-16aa-4e39-8416-a84259bfc84d
DEBUG:root:expected_signature: s35dRVuqGcLsobpB/h2cT75j8Dw=
DEBUG:root:signature: s35dRVuqGcLsobpB/h2cT75j8Dw=
DEBUG:root:Putting object: newbucket / ubuntu-lucid.img.part.4
INFO:root:Looking up user: '3f7e2c7f-d5af-4150-a404-0315b39c0256'
INFO:root:user: User('novauser', 'novauser', '3f7e2c7f-d5af-4150-a404-0315b39c0256', '71095e77-16aa-4e39-8416-a84259bfc84d', True)
DEBUG:root:user.secret: 71095e77-16aa-4e39-8416-a84259bfc84d
DEBUG:root:expected_signature: 5AsPAiU8MXTwx5bPrxm/srs5Z2Q=
DEBUG:root:signature: 5AsPAiU8MXTwx5bPrxm/srs5Z2Q=
DEBUG:root:Putting object: newbucket / ubuntu-lucid.img.part.5
INFO:root:Looking up user: '3f7e2c7f-d5af-4150-a404-0315b39c0256'
INFO:root:user: User('novauser', 'novauser', '3f7e2c7f-d5af-4150-a404-0315b39c0256', '71095e77-16aa-4e39-8416-a84259bfc84d', True)
DEBUG:root:user.secret: 71095e77-16aa-4e39-8416-a84259bfc84d
DEBUG:root:expected_signature: pkOfQOPvXLlBVHgbscpOGwdwjK0=
DEBUG:root:signature: pkOfQOPvXLlBVHgbscpOGwdwjK0=
DEBUG:root:Putting object: newbucket / ubuntu-lucid.img.part.6
INFO:root:Looking up user: '3f7e2c7f-d5af-4150-a404-0315b39c0256'
INFO:root:user: User('novauser', 'novauser', '3f7e2c7f-d5af-4150-a404-0315b39c0256', '71095e77-16aa-4e39-8416-a84259bfc84d', True)
DEBUG:root:user.secret: 71095e77-16aa-4e39-8416-a84259bfc84d
DEBUG:root:expected_signature: I4Bd+Rt+VRdiTrgArsuO1k0W77E=
DEBUG:root:signature: I4Bd+Rt+VRdiTrgArsuO1k0W77E=
DEBUG:root:Putting object: newbucket / ubuntu-lucid.img.part.7
INFO:root:Looking up user: '3f7e2c7f-d5af-4150-a404-0315b39c0256'
INFO:root:user: User('novauser', 'novauser', '3f7e2c7f-d5af-4150-a404-0315b39c0256', '71095e77-16aa-4e39-8416-a84259bfc84d', True)
DEBUG:root:user.secret: 71095e77-16aa-4e39-8416-a84259bfc84d
DEBUG:root:expected_signature: mYcIO/wJH+U52keY36ZJFAbiVIA=
DEBUG:root:signature: mYcIO/wJH+U52keY36ZJFAbiVIA=
DEBUG:root:Putting object: newbucket / ubuntu-lucid.img.part.8
INFO:root:Looking up user: '3f7e2c7f-d5af-4150-a404-0315b39c0256'
INFO:root:user: User('novauser', 'novauser', '3f7e2c7f-d5af-4150-a404-0315b39c0256', '71095e77-16aa-4e39-8416-a84259bfc84d', True)
DEBUG:root:user.secret: 71095e77-16aa-4e39-8416-a84259bfc84d
DEBUG:root:expected_signature: rB4dTar/+f7JV71kV3aMNSRUQjI=
DEBUG:root:signature: rB4dTar/+f7JV71kV3aMNSRUQjI=
DEBUG:root:Putting object: newbucket / ubuntu-lucid.img.part.9
INFO:root:Looking up user: '3f7e2c7f-d5af-4150-a404-0315b39c0256'
INFO:root:user: User('novauser', 'novauser', '3f7e2c7f-d5af-4150-a404-0315b39c0256', '71095e77-16aa-4e39-8416-a84259bfc84d', True)
DEBUG:root:user.secret: 71095e77-16aa-4e39-8416-a84259bfc84d
DEBUG:root:expected_signature: GOLBQIsBuitpSZ+Jlv2SzTwbVBI=
DEBUG:root:signature: GOLBQIsBuitpSZ+Jlv2SzTwbVBI=
DEBUG:root:Putting object: newbucket / ubuntu-lucid.img.part.10
INFO:root:Looking up user: '3f7e2c7f-d5af-4150-a404-0315b39c0256'
INFO:root:user: User('novauser', 'novauser', '3f7e2c7f-d5af-4150-a404-0315b39c0256', '71095e77-16aa-4e39-8416-a84259bfc84d', True)
DEBUG:root:user.secret: 71095e77-16aa-4e39-8416-a84259bfc84d
DEBUG:root:expected_signature: 0hr49hraQKsNTW5e7PHFXlfVeos=
DEBUG:root:signature: 0hr49hraQKsNTW5e7PHFXlfVeos=
DEBUG:root:Putting object: newbucket / ubuntu-lucid.img.part.11
INFO:root:Looking up user: '3f7e2c7f-d5af-4150-a404-0315b39c0256'
INFO:root:user: User('novauser', 'novauser', '3f7e2c7f-d5af-4150-a404-0315b39c0256', '71095e77-16aa-4e39-8416-a84259bfc84d', True)
DEBUG:root:user.secret: 71095e77-16aa-4e39-8416-a84259bfc84d
DEBUG:root:expected_signature: 2EzmkQKjPQgKnK0QP4gwon2g2zo=
DEBUG:root:signature: 2EzmkQKjPQgKnK0QP4gwon2g2zo=
DEBUG:root:Putting object: newbucket / ubuntu-lucid.img.part.12
INFO:root:Looking up user: '3f7e2c7f-d5af-4150-a404-0315b39c0256'
INFO:root:user: User('novauser', 'novauser', '3f7e2c7f-d5af-4150-a404-0315b39c0256', '71095e77-16aa-4e39-8416-a84259bfc84d', True)
DEBUG:root:user.secret: 71095e77-16aa-4e39-8416-a84259bfc84d
DEBUG:root:expected_signature: n/rut/cqNUeovFaJ/P5g95dPhZ8=
DEBUG:root:signature: n/rut/cqNUeovFaJ/P5g95dPhZ8=
DEBUG:root:Putting object: newbucket / ubuntu-lucid.img.part.13
************** EUCA-UPLOAD-BUNDLE EXECUTION ENDS *******************
************** EUCA-REGISTER KERNEL STARTS *************************
INFO:root:Looking up user: '3f7e2c7f-d5af-4150-a404-0315b39c0256'
INFO:root:user: User('novauser', 'novauser', '3f7e2c7f-d5af-4150-a404-0315b39c0256', '71095e77-16aa-4e39-8416-a84259bfc84d', True)
DEBUG:root:user.secret: 71095e77-16aa-4e39-8416-a84259bfc84d
DEBUG:root:expected_signature: s3sWf5IBVWLiI8hLxYbDBVQnJIQ=
DEBUG:root:signature: s3sWf5IBVWLiI8hLxYbDBVQnJIQ=
DEBUG:root:Result was 1
************** EUCA-REGISTER RAMDISK STARTS ************************
INFO:root:Looking up user: '3f7e2c7f-d5af-4150-a404-0315b39c0256'
INFO:root:user: User('novauser', 'novauser', '3f7e2c7f-d5af-4150-a404-0315b39c0256', '71095e77-16aa-4e39-8416-a84259bfc84d', True)
DEBUG:root:user.secret: 71095e77-16aa-4e39-8416-a84259bfc84d
DEBUG:root:expected_signature: r5xYu0GGTuC8NIw6BXvajdUe83Y=
DEBUG:root:signature: r5xYu0GGTuC8NIw6BXvajdUe83Y=
DEBUG:root:Result was 1
************** EUCA-REGISTER MACHINE STARTS ************************
INFO:root:Looking up user: '3f7e2c7f-d5af-4150-a404-0315b39c0256'
INFO:root:user: User('novauser', 'novauser', '3f7e2c7f-d5af-4150-a404-0315b39c0256', '71095e77-16aa-4e39-8416-a84259bfc84d', True)
DEBUG:root:user.secret: 71095e77-16aa-4e39-8416-a84259bfc84d
DEBUG:root:expected_signature: bq9Aj3x0dUAwft/JUH0LCEnnIVc=
DEBUG:root:signature: bq9Aj3x0dUAwft/JUH0LCEnnIVc=
DEBUG:root:Result was 1
*****************************************************************************************************
***************************************NOVA-COMPUTE ***************************************
root@armandomi-cieg:~/openstack/nova# nova-compute ${NOVA_COMPUTE_ARGS} --verbose $1 --nodaemon
DEBUG:root:Serving /home/armandomi/openstack/nova/trunk182/bin/nova-compute
DEBUG:root:Full set of FLAGS:
DEBUG:root:storage_availability_zone : nova
DEBUG:root:help : None
DEBUG:root:cloudpipe_start_port : 12000
DEBUG:root:encrypted : None
DEBUG:root:compute_topic : compute
DEBUG:root:keys_path : /home/armandomi/openstack/nova/trunk182/../data/keys
DEBUG:root:report_profile : None
DEBUG:root:rabbit_password : guest
DEBUG:root:prefix : nova-compute
DEBUG:root:originalname : None
DEBUG:root:simple_network_bridge : br0
DEBUG:root:public_interface : vlan1
DEBUG:root:reactor : None
DEBUG:root:network_size : 256
DEBUG:root:vpn_start_port : 1000
DEBUG:root:spew : None
DEBUG:root:umask : None
DEBUG:root:source : None
DEBUG:root:nothotshot : None
DEBUG:root:simple_network_ips : ['192.168.1.220', '192.168.1.221', '192.168.1.222']
DEBUG:root:helpxml : None
DEBUG:root:simple_network : True
DEBUG:root:python : /home/armandomi/openstack/nova/trunk182/bin/nova-compute
DEBUG:root:aoe_eth_dev : eth0
DEBUG:root:rabbit_port : 5672
DEBUG:root:euid : None
DEBUG:root:logfile : nova-compute.log
DEBUG:root:nodaemon : True
DEBUG:root:last_shelf_id : 279
DEBUG:root:redis_host : 127.0.0.1
DEBUG:root:pidfile : /home/armandomi/openstack/nova/nova-compute.pid
DEBUG:root:default_instance_type : m1.small
DEBUG:root:volume_topic : volume
DEBUG:root:public_range : 4.4.4.0/24
DEBUG:root:use_vpn : True
DEBUG:root:syslog : None
DEBUG:root:ec2_url : http://127.0.0.1:8773/services/Cloud
DEBUG:root:rundir : .
DEBUG:root:storage_dev : /dev/sdb
DEBUG:root:s3_port : 3333
DEBUG:root:fake_libvirt : False
DEBUG:root:fake_storage : False
DEBUG:root:redis_db : 0
DEBUG:root:version : None
DEBUG:root:credential_key_file : pk.pem
DEBUG:root:? : None
DEBUG:root:rabbit_userid : guest
DEBUG:root:simple_network_gateway : 192.168.1.1
DEBUG:root:buckets_path : /home/armandomi/openstack/nova/trunk182/nova/../buckets
DEBUG:root:instances_path : /home/armandomi/openstack/nova/trunk182/../data/instances
DEBUG:root:debug : False
DEBUG:root:storage_name : armandomi-cieg
DEBUG:root:node_name : armandomi-cieg
DEBUG:root:networks_path : /home/armandomi/openstack/nova/trunk182/../data/networks
DEBUG:root:s3_host : 127.0.0.1
DEBUG:root:aoe_export_dir : /var/lib/vblade-persist/vblades
DEBUG:root:images_path : /home/armandomi/openstack/nova/trunk182/nova/../images
DEBUG:root:verbose : True
DEBUG:root:default_kernel : aki-11111
DEBUG:root:vpn_key_suffix : -key
DEBUG:root:libvirt_xml_template : nova/compute/libvirt.xml.template
DEBUG:root:vpn_client_template : nova/cloudpipe/client.ovpn.template
DEBUG:root:process_pool_size : 4
DEBUG:root:gid : None
DEBUG:root:slots_per_shelf : 16
DEBUG:root:volume_group : nova-volumes
DEBUG:root:simple_network_broadcast : 192.168.1.255
DEBUG:root:credential_cert_file : cert.pem
DEBUG:root:simple_network_netmask : 255.255.255.0
DEBUG:root:simple_network_dns : 8.8.4.4
DEBUG:root:vpn_end_port : 2000
DEBUG:root:report_interval : 10
DEBUG:root:helpshort : None
DEBUG:root:ca_file : cacert.pem
DEBUG:root:use_intermediate_ca : False
DEBUG:root:vpn_image_id : ami-CLOUDPIPE
DEBUG:root:public_vlan : 1
DEBUG:root:use_s3 : True
DEBUG:root:b : None
DEBUG:root:private_range : 10.0.0.0/8
DEBUG:root:credentials_template : /nova/auth/novarc.template
DEBUG:root:profiler : hotshot
DEBUG:root:vpn_ip : 127.0.0.1
DEBUG:root:r : None
DEBUG:root:default_image : ami-11111
DEBUG:root:savestats : None
DEBUG:root:credential_rc_file : novarc
DEBUG:root:fake_network : False
DEBUG:root:first_shelf_id : 270
DEBUG:root:dhcpbridge_flagfile : /etc/nova/nova-dhcpbridge.conf
DEBUG:root:uid : None
DEBUG:root:vlan_end : 4093
DEBUG:root:credential_cert_subject : /C=US/ST=California/L=MountainView/O=AnsoLabs/OU=NovaDev/CN=%s-%s
DEBUG:root:auth_driver : nova.auth.ldapdriver.FakeLdapDriver
DEBUG:root:file : twistd.tap
DEBUG:root:bridge_dev : eth1
DEBUG:root:fake_rabbit : False
DEBUG:root:chroot : None
DEBUG:root:help_reactors : None
DEBUG:root:rabbit_host : localhost
DEBUG:root:simple_network_network : 192.168.1.0
DEBUG:root:superuser_roles : ['cloudadmin']
DEBUG:root:control_exchange : nova
DEBUG:root:profile : None
DEBUG:root:simple_network_template : /home/armandomi/openstack/nova/trunk182/nova/compute/interfaces.template
DEBUG:root:default_ramdisk : ari-11111
DEBUG:root:node_availability_zone : nova
DEBUG:root:vlan_start : 100
DEBUG:root:global_roles : ['cloudadmin', 'itsec']
DEBUG:root:no_save : True
DEBUG:root:rabbit_virtual_host : /
DEBUG:root:ca_path :
DEBUG:root:cnt_vpn_clients : 5
DEBUG:root:redis_port : 6379
DEBUG:root:network_topic : network
WARNING:root:Starting compute node
WARNING:root:no model class for Host when building association from <Daemon:armandomi-cieg:nova-compute>
************** EUCA-RUN-INSTANCES STARTS ************************
DEBUG:root:Starting instance i-ze0bnh1q...
DEBUG:root:Finished init of Instance with id of i-ze0bnh1q
INFO:root:Instances current state is 0
DEBUG:root:Instance state is: 0
DEBUG:root:Starting spawn in Instance
DEBUG:root:Starting the toXML method

Question information

Language:: English Edit question

Status:: Solved

For:: OpenStack Compute (nova) Edit question

Assignee:: No assignee Edit question

Solved by:: Armando Migliaccio

Solved:: 2010-08-18

Last query:: 2010-08-18

Last reply:: 2010-07-29

Link existing bug

Revision history for this message

Vish Ishaya (vishvananda) said on 2010-07-29:

i would first check your instances directory for i-ze0bnh1q dir
you should have the following files:
  libvirt.xml
  disk
  disk-raw
  kernel
  ramdisk

once the instance actually starts you should see a console.log
check the file sizes to see if they are reasonable. If any are missing/zero/very small then compute has messed up the download of the images from objectstore
Also check nova-compute.log for exceptions. Sometimes they don't show up in the console output

next, check the /var/log/libvirt/qemu/i-ze0bnh1q.log file to see if it exists and has any useful error message

finally try virsh create libvirt.xml from thie instances/i-ze0bnh1q directory and see if you get an error there.

Hopefully one of these things will lead to some more information and we can see what is going wrong.

Revision history for this message

Armando Migliaccio (armando-migliaccio) said on 2010-07-29:

Many thanks for the reply, that did point me in the right direction!! The instance directory shows all files but 'disk'; /var/log/libvirt/qemu/ is empty. When I look at the nova-compute.log there's the following error message:

DEBUG:root:Failed to mount filesystem: mount: you must specify the filesystem type

The nova-volume.log does not show any errors and when I try virsh, I clearly get the following error:

error: Failed to create domain from libvirt.xml
error: cannot open file '.../nova/data/instances/<instance-id>/disk': No such file or directory

Any clues? Thanks a lot!

Revision history for this message

Vish Ishaya (vishvananda) said on 2010-07-31:

so your disk-raw is not a valid image. Either the image is not correct before it is copied or it is not being downloaded from objectstore correctly. You may need this patch if your user and project name are different:
lp:~vishvananda/nova/fix-curl-project

Nova-volume shouldn't be doing anything at this point, but you might check nova-objectstore for errors. If the above doesn't help, your original disk image may be corrupted. Here is a method for grabbing a working image that we used on our first hack-day:

wget -q -O - http://173.203.107.207/ubuntu-lucid.tar | tar xSv
euca-bundle-image -i vmlinuz-2.6.32-23-server --kernel true
euca-bundle-image -i initrd.img-2.6.32-23-server --ramdisk true
euca-bundle-image -i ubuntu-lucid.img
euca-upload-bundle -b mybucket -m /tmp/vmlinuz-2.6.32-23-server.manifest.xml
euca-upload-bundle -b mybucket -m /tmp/initrd.img-2.6.32-23-server.manifest.xml
euca-upload-bundle -b mybucket -m /tmp/ubuntu-lucid.img.manifest.xml

kernel=$(euca-register mybucket/vmlinuz-2.6.32-23-server.manifest.xml | awk '{print $2}')
ramdisk=$(euca-register mybucket/initrd.img-2.6.32-23-server.manifest.xml | awk '{print $2}')
machine=$(euca-register mybucket/ubuntu-lucid.img.manifest.xml | awk '{print $2}')
euca-run-instances $machine --kernel $kernel --ramdisk $ramdisk
# wait a while
ssh ubuntu@10.0.0.3 # password is ubuntu

Revision history for this message

Armando Migliaccio (armando-migliaccio) said on 2010-08-04:

Project name and user are the same. disk-raw is an ASCII file showning this:

<html>
  <head><title>404 - No Such Resource</title></head>
  <body>
    <h1>No Such Resource</h1>
    <p>File not found.</p>
  </body>
</html>

Whereas the objectstore's log shows this:

2010-07-30 16:48:52+0100 [HTTPChannel,9,127.0.0.1] 127.0.0.1 - - [30/Jul/2010:15:48:51 +0000] "GET /_images/ami-3ig6lc4q/image HTTP/1.1" 404 145 "-" "curl/7.19.7 (i486-pc-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8k zlib/1.2.3.3 libidn/1.15"

I think this might be related to this line 'vi +111 trunk/nova/compute/disk.py', which does not raise the exception.

Revision history for this message

Soren Hansen (soren) said on 2010-08-16:

I'm rather sure this problem has been solved in never revision of Nova. Can you recheck please? If it's not, we should turn this into a bug report.

Revision history for this message

Armando Migliaccio (armando-migliaccio) said on 2010-08-18:

I was no longer able to replicate this, so I'd imagine that it has been fixed!

Thanks!

To post a message you must log in.

Ask a question

Edit question

OpenStack Compute (nova)

euca-run-instancess produce vm in indefinite spawning state

Question information

Related bugs

Related FAQ:

Subscribers