Nova completely stuck while giving some commands

This is the log of nova-api after the call

fabrizio@uranus:~/nova-log$ sudo nova delete ef61492b-14b6-4b43-9f49-51e6f152e26d

18:30:07 INFO nova.api.openstack.wsgi - GET http://localhost:8774/v1.1/<project_id>/servers/detail
18:30:07 DEBUG nova.api.openstack.wsgi - No Content-Type provided in request
18:30:07 DEBUG nova.compute.api - Searching by: {'deleted': False, 'project_id': u'<project_id>'}
18:30:08 INFO nova.api.openstack.wsgi - http://localhost:8774/v1.1/<project_id>/servers/detail returned with HTTP 200
18:30:08 INFO nova.osapi_compute.wsgi.server - 127.0.0.1 - - - "GET /v1.1/<project_id>/servers/detail HTTP/1.1" 200 1631 1.194018

18:30:08 INFO nova.api.openstack.wsgi - GET http://localhost:8774/v1.1/<project_id>/servers/detail
18:30:08 DEBUG nova.api.openstack.wsgi - No Content-Type provided in request
18:30:08 DEBUG nova.compute.api - Searching by: {'deleted': False, 'project_id': u'<project_id>'}
18:30:09 INFO nova.api.openstack.wsgi - http://localhost:8774/v1.1/<project_id>/servers/detail returned with HTTP 200
18:30:09 INFO nova.osapi_compute.wsgi.server - 127.0.0.1 - - - "GET /v1.1/<project_id>/servers/detail HTTP/1.1" 200 1631 1.179668

18:30:10 INFO nova.api.openstack.wsgi - DELETE http://localhost:8774/v1.1/<project_id>/servers/<instance-id>
18:30:10 DEBUG nova.api.openstack.wsgi - No Content-Type provided in request
18:30:10 DEBUG nova.compute.api - - Going to try to terminate instance
18:30:11 DEBUG nova.openstack.common.rpc.amqp - Making asynchronous cast on compute.hemera...
18:30:11 DEBUG nova.openstack.common.rpc.amqp - Pool creating new connection
18:30:11 INFO nova.openstack.common.rpc.common - Connected to AMQP server on 192.168.3.1:5672

Is their an more of the log you posted? If so can you post here.
What sort of a install have you?

It may be a good idea to just post the log if that is possible, use a paste bin like pastie.org

All my machines are old, 32bit pentium 4. The cluster controller is running with Ubuntu 12.04, no virtualization, with keystone, glance and all the nova services except nova-compute. nova-compute is running on the other 2 machines over XCP 1.6, but the problem appear in the cluster controller.
In this file there are all the logs and che configuration files for nova: http://wikisend.com/download/966144/nova-all.tar.gz

I uninstalled all the nova components and dependencies and then I installed again them. Everything seemed fine but when I nova boot the problem appeared again. After that command also nova network-list got stuck
Here there is all the configuration and logs of the cloud controller: http://wikisend.com/download/610934/node-controller.tar.gz and here everything aboud the compute node: http://wikisend.com/download/120592/node-compute.tar.gz

Hold tight I am looking at files thanks.

Sorry for delay in getting back to you, in you log their is a a error/exception that occurs when you start nova-compute. This error occurs because the database is missing a table called 'instances'. If this table is missing from the database, something has gone wrong with your install of the database. I would log in in to MySql and verify if this table is missing and then figure why you did not add it, or did something go wrong with your install in some way. Here is a snip of the error. Feel free to ask any other questions and I will try to help. If you get thing working please close this bug.

Keith

2013-01-28 17:19:29 AUDIT nova.service [-] Starting compute node (version 2012.2-LOCALBRANCH:LOCALREVISION)
2013-01-28 17:19:30 DEBUG nova.utils [req-5a439208-87d2-4545-a35f-08607fdb3f49 None None] backend <module 'nova.db.sqlalchemy.api' from '/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.pyc'> from (pid=945) __get_backend /usr/lib/python2.7/dist-packages/nova/utils.py:494
2013-01-28 17:19:31 CRITICAL nova [-] (ProgrammingError) (1146, "Table 'nova.instances' doesn't exist") 'SELECT instances.created_at AS instances_created_at, instances.updated_at AS instances_updated_at, instances.deleted_at AS instances_deleted_at, instances.deleted AS instances_deleted, instances.id AS instances_id, instances.user_id AS instances_user_id, instances.project_id AS instances_project_id, instances.image_ref AS instances_image_ref, instances.kernel_id AS instances_kernel_id, instances.ramdisk_id AS instances_ramdisk_id, instances.server_name AS instances_server_name, instances.launch_index AS instances_launch_index, instances.key_name AS instances_key_name, instances.key_data AS instances_key_data, instances.power_state AS instances_power_state, instances.vm_state AS instances_vm_state, instances.task_state AS instances_task_state, instances.memory_mb AS instances_memory_mb, instances.vcpus AS instances_vcpus, instances.root_gb AS instances_root_gb, instances.ephemeral_gb AS instances_ephemeral_gb, instances.hostname AS instances_hostname, instances.host AS instances_host, instances.instance_type_id AS instances_instance_type_id, instances.user_data AS instances_user_data, instances.reservation_id AS instances_reservation_id, instances.scheduled_at AS instances_scheduled_at, instances.launched_at AS instances_launched_at, instances.terminated_at AS instances_terminated_at, instances.availability_zone AS instances_availability_zone, instances.display_name AS instances_display_name, instances.display_description AS instances_display_description, instances.launched_on AS instances_launched_on, instances.locked AS instances_locked, instances.os_type AS instances_os_type, instances.architecture AS instances_architecture, instances.vm_mode AS instances_vm_mode, instances.uuid AS instances_uuid, instances.root_device_name AS instances_root_device_name, instances.default_ephemeral_device AS instances_default_ephemeral_device, instances.default_swap_device AS instances_default_swap_device, instances.config_drive AS instances_config_drive, instances.access_ip_v4 AS instances_access_ip_v4, instances.access_ip_v6 AS instances_access_ip_v6, instances.auto_disk_config AS instances_auto_disk_config, instances.progress AS instances_progress, instances.shutdown_terminate AS instances_shutdown_terminate, instances.disable_terminate AS instances_disable_terminate, instance_metadata_1.created_at AS instance_metadata_1_created_at, instance_metadata_1.updated_at AS instance_metadata_1_updated_at, instance_metadata_1.deleted_at AS instance_metadata_1_deleted_at, instance_metadata_1.deleted AS instance_metadata_1_deleted, instance_metadata_1.id AS instance_metadata_1_id, instance_metadata_1.`key` AS instance_metadata_1_key, instance_metadata_1.value AS instance_metadata_1_value, instance_metadata_1.instance_uuid AS instance_metadata_1_instance_uuid, security_groups_1.created_at AS security_groups_1_created_at, security_groups_1.updated_at AS security_groups_1_updated_at, security_groups_1.deleted_at AS security_groups_1_deleted_at, security_groups_1.deleted AS security_groups_1_deleted, security_groups_1.id AS security_groups_1_id, security_groups_1.name AS security_groups_1_name, security_groups_1.description AS security_groups_1_description, security_groups_1.user_id AS security_groups_1_user_id, security_groups_1.project_id AS security_groups_1_project_id, instance_types_1.created_at AS instance_types_1_created_at, instance_types_1.updated_at AS instance_types_1_updated_at, instance_types_1.deleted_at AS instance_types_1_deleted_at, instance_types_1.deleted AS instance_types_1_deleted, instance_types_1.id AS instance_types_1_id, instance_types_1.name AS instance_types_1_name, instance_types_1.memory_mb AS instance_types_1_memory_mb, instance_types_1.vcpus AS instance_types_1_vcpus, instance_types_1.root_gb AS instance_types_1_root_gb, instance_types_1.ephemeral_gb AS instance_types_1_ephemeral_gb, instance_types_1.flavorid AS instance_types_1_flavorid, instance_types_1.swap AS instance_types_1_swap, instance_types_1.rxtx_factor AS instance_types_1_rxtx_factor, instance_types_1.vcpu_weight AS instance_types_1_vcpu_weight, instance_types_1.disabled AS instance_types_1_disabled, instance_types_1.is_public AS instance_types_1_is_public, instance_info_caches_1.created_at AS instance_info_caches_1_created_at, instance_info_caches_1.updated_at AS instance_info_caches_1_updated_at, instance_info_caches_1.deleted_at AS instance_info_caches_1_deleted_at, instance_info_caches_1.deleted AS instance_info_caches_1_deleted, instance_info_caches_1.id AS instance_info_caches_1_id, instance_info_caches_1.network_info AS instance_info_caches_1_network_info, instance_info_caches_1.instance_uuid AS instance_info_caches_1_instance_uuid \nFROM instances LEFT OUTER JOIN instance_metadata AS instance_metadata_1 ON instance_metadata_1.instance_uuid = instances.uuid AND instance_metadata_1.deleted = %s LEFT OUTER JOIN security_group_instance_association AS security_group_instance_association_1 ON security_group_instance_association_1.instance_uuid = instances.uuid AND instances.deleted = %s LEFT OUTER JOIN security_groups AS security_groups_1 ON security_groups_1.id = security_group_instance_association_1.security_group_id AND security_group_instance_association_1.deleted = %s AND security_groups_1.deleted = %s LEFT OUTER JOIN instance_types AS instance_types_1 ON instances.instance_type_id = instance_types_1.id LEFT OUTER JOIN instance_info_caches AS instance_info_caches_1 ON instance_info_caches_1.instance_uuid = instances.uuid \nWHERE instances.deleted = %s AND instances.host = %s' (0, 0, 0, 0, 0, 'hemera')
2013-01-28 17:19:31 TRACE nova Traceback (most recent call last):
2013-01-28 17:19:31 TRACE nova File "/usr/bin/nova-compute", line 48, in <module>
2013-01-28 17:19:31 TRACE nova service.wait()
2013-01-28 17:19:31 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 659, in wait
2013-01-28 17:19:31 TRACE nova _launcher.wait()
2013-01-28 17:19:31 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 192, in wait
2013-01-28 17:19:31 TRACE nova super(ServiceLauncher, self).wait()
2013-01-28 17:19:31 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 162, in wait
2013-01-28 17:19:31 TRACE nova service.wait()
2013-01-28 17:19:31 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 166, in wait
2013-01-28 17:19:31 TRACE nova return self._exit_event.wait()
2013-01-28 17:19:31 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
2013-01-28 17:19:31 TRACE nova return hubs.get_hub().switch()
2013-01-28 17:19:31 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 177, in switch
2013-01-28 17:19:31 TRACE nova return self.greenlet.switch()
2013-01-28 17:19:31 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 192, in main
2013-01-28 17:19:31 TRACE nova result = function(*args, **kwargs)
2013-01-28 17:19:31 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 132, in run_server
2013-01-28 17:19:31 TRACE nova server.start()
2013-01-28 17:19:31 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 398, in start
2013-01-28 17:19:31 TRACE nova self.manager.init_host()
2013-01-28 17:19:31 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 274, in init_host
2013-01-28 17:19:31 TRACE nova instances = self.db.instance_get_all_by_host(context, self.host)
2013-01-28 17:19:31 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 626, in instance_get_all_by_host
2013-01-28 17:19:31 TRACE nova return IMPL.instance_get_all_by_host(context, host)
2013-01-28 17:19:31 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 111, in wrapper
2013-01-28 17:19:31 TRACE nova return f(*args, **kwargs)
2013-01-28 17:19:31 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 1694, in instance_get_all_by_host
2013-01-28 17:19:31 TRACE nova return _instance_get_all_query(context).filter_by(host=host).all()
2013-01-28 17:19:31 TRACE nova File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2115, in all
2013-01-28 17:19:31 TRACE nova return list(self)
2013-01-28 17:19:31 TRACE nova File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2227, in __iter__
2013-01-28 17:19:31 TRACE nova return self._execute_and_instances(context)
2013-01-28 17:19:31 TRACE nova File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2242, in _execute_and_instances
2013-01-28 17:19:31 TRACE nova result = conn.execute(querycontext.statement, self._params)
2013-01-28 17:19:31 TRACE nova File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1449, in execute
2013-01-28 17:19:31 TRACE nova params)
2013-01-28 17:19:31 TRACE nova File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1584, in _execute_clauseelement
2013-01-28 17:19:31 TRACE nova compiled_sql, distilled_params
2013-01-28 17:19:31 TRACE nova File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1698, in _execute_context
2013-01-28 17:19:31 TRACE nova context)
2013-01-28 17:19:31 TRACE nova File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1691, in _execute_context
2013-01-28 17:19:31 TRACE nova context)
2013-01-28 17:19:31 TRACE nova File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 331, in do_execute
2013-01-28 17:19:31 TRACE nova cursor.execute(statement, parameters)
2013-01-28 17:19:31 TRACE nova File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
2013-01-28 17:19:31 TRACE nova self.errorhandler(self, exc, value)
2013-01-28 17:19:31 TRACE nova File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
2013-01-28 17:19:31 TRACE nova raise errorclass, errorvalue
2013-01-28 17:19:31 TRACE nova ProgrammingError: (ProgrammingError) (1146, "Table 'nova.instances' doesn't exist") 'SELECT instances.created_at AS instances_created_at, instances.updated_at AS instances_updated_at, instances.deleted_at AS instances_deleted_at, instances.deleted AS instances_deleted, instances.id AS instances_id, instances.user_id AS instances_user_id, instances.project_id AS instances_project_id, instances.image_ref AS instances_image_ref, instances.kernel_id AS instances_kernel_id, instances.ramdisk_id AS instances_ramdisk_id, instances.server_name AS instances_server_name, instances.launch_index AS instances_launch_index, instances.key_name AS instances_key_name, instances.key_data AS instances_key_data, instances.power_state AS instances_power_state, instances.vm_state AS instances_vm_state, instances.task_state AS instances_task_state, instances.memory_mb AS instances_memory_mb, instances.vcpus AS instances_vcpus, instances.root_gb AS instances_root_gb, instances.ephemeral_gb AS instances_ephemeral_gb, instances.hostname AS instances_hostname, instances.host AS instances_host, instances.instance_type_id AS instances_instance_type_id, instances.user_data AS instances_user_data, instances.reservation_id AS instances_reservation_id, instances.scheduled_at AS instances_scheduled_at, instances.launched_at AS instances_launched_at, instances.terminated_at AS instances_terminated_at, instances.availability_zone AS instances_availability_zone, instances.display_name AS instances_display_name, instances.display_description AS instances_display_description, instances.launched_on AS instances_launched_on, instances.locked AS instances_locked, instances.os_type AS instances_os_type, instances.architecture AS instances_architecture, instances.vm_mode AS instances_vm_mode, instances.uuid AS instances_uuid, instances.root_device_name AS instances_root_device_name, instances.default_ephemeral_device AS instances_default_ephemeral_device, instances.default_swap_device AS instances_default_swap_device, instances.config_drive AS instances_config_drive, instances.access_ip_v4 AS instances_access_ip_v4, instances.access_ip_v6 AS instances_access_ip_v6, instances.auto_disk_config AS instances_auto_disk_config, instances.progress AS instances_progress, instances.shutdown_terminate AS instances_shutdown_terminate, instances.disable_terminate AS instances_disable_terminate, instance_metadata_1.created_at AS instance_metadata_1_created_at, instance_metadata_1.updated_at AS instance_metadata_1_updated_at, instance_metadata_1.deleted_at AS instance_metadata_1_deleted_at, instance_metadata_1.deleted AS instance_metadata_1_deleted, instance_metadata_1.id AS instance_metadata_1_id, instance_metadata_1.`key` AS instance_metadata_1_key, instance_metadata_1.value AS instance_metadata_1_value, instance_metadata_1.instance_uuid AS instance_metadata_1_instance_uuid, security_groups_1.created_at AS security_groups_1_created_at, security_groups_1.updated_at AS security_groups_1_updated_at, security_groups_1.deleted_at AS security_groups_1_deleted_at, security_groups_1.deleted AS security_groups_1_deleted, security_groups_1.id AS security_groups_1_id, security_groups_1.name AS security_groups_1_name, security_groups_1.description AS security_groups_1_description, security_groups_1.user_id AS security_groups_1_user_id, security_groups_1.project_id AS security_groups_1_project_id, instance_types_1.created_at AS instance_types_1_created_at, instance_types_1.updated_at AS instance_types_1_updated_at, instance_types_1.deleted_at AS instance_types_1_deleted_at, instance_types_1.deleted AS instance_types_1_deleted, instance_types_1.id AS instance_types_1_id, instance_types_1.name AS instance_types_1_name, instance_types_1.memory_mb AS instance_types_1_memory_mb, instance_types_1.vcpus AS instance_types_1_vcpus, instance_types_1.root_gb AS instance_types_1_root_gb, instance_types_1.ephemeral_gb AS instance_types_1_ephemeral_gb, instance_types_1.flavorid AS instance_types_1_flavorid, instance_types_1.swap AS instance_types_1_swap, instance_types_1.rxtx_factor AS instance_types_1_rxtx_factor, instance_types_1.vcpu_weight AS instance_types_1_vcpu_weight, instance_types_1.disabled AS instance_types_1_disabled, instance_types_1.is_public AS instance_types_1_is_public, instance_info_caches_1.created_at AS instance_info_caches_1_created_at, instance_info_caches_1.updated_at AS instance_info_caches_1_updated_at, instance_info_caches_1.deleted_at AS instance_info_caches_1_deleted_at, instance_info_caches_1.deleted AS instance_info_caches_1_deleted, instance_info_caches_1.id AS instance_info_caches_1_id, instance_info_caches_1.network_info AS instance_info_caches_1_network_info, instance_info_caches_1.instance_uuid AS instance_info_caches_1_instance_uuid \nFROM instances LEFT OUTER JOIN instance_metadata AS instance_metadata_1 ON instance_metadata_1.instance_uuid = instances.uuid AND instance_metadata_1.deleted = %s LEFT OUTER JOIN security_group_instance_association AS security_group_instance_association_1 ON security_group_instance_association_1.instance_uuid = instances.uuid AND instances.deleted = %s LEFT OUTER JOIN security_groups AS security_groups_1 ON security_groups_1.id = security_group_instance_association_1.security_group_id AND security_group_instance_association_1.deleted = %s AND security_groups_1.deleted = %s LEFT OUTER JOIN instance_types AS instance_types_1 ON instances.instance_type_id = instance_types_1.id LEFT OUTER JOIN instance_info_caches AS instance_info_caches_1 ON instance_info_caches_1.instance_uuid = instances.uuid \nWHERE instances.deleted = %s AND instances.host = %s' (0, 0, 0, 0, 0, 'hemera')
2013-01-28 17:19:31 TRACE nova

Adding to my above/last post, you have a second error in you config file, I would need to have a further look at what may cause it but, I would try fix the database problem I pointed out in last post and then restart all the services and see if you have problems still, if so post the new logs and I will help.

2013-01-28 17:15:48 CRITICAL nova [-] [Errno 13] Permission denied
2013-01-28 17:15:48 TRACE nova Traceback (most recent call last):
2013-01-28 17:15:48 TRACE nova File "/usr/bin/nova-compute", line 48, in <module>
2013-01-28 17:15:48 TRACE nova service.wait()
2013-01-28 17:15:48 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 659, in wait
2013-01-28 17:15:48 TRACE nova _launcher.wait()
2013-01-28 17:15:48 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 192, in wait
2013-01-28 17:15:48 TRACE nova super(ServiceLauncher, self).wait()
2013-01-28 17:15:48 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 162, in wait
2013-01-28 17:15:48 TRACE nova service.wait()
2013-01-28 17:15:48 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 166, in wait
2013-01-28 17:15:48 TRACE nova return self._exit_event.wait()
2013-01-28 17:15:48 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
2013-01-28 17:15:48 TRACE nova return hubs.get_hub().switch()
2013-01-28 17:15:48 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 177, in switch
2013-01-28 17:15:48 TRACE nova return self.greenlet.switch()
2013-01-28 17:15:48 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 192, in main
2013-01-28 17:15:48 TRACE nova result = function(*args, **kwargs)
2013-01-28 17:15:48 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 132, in run_server
2013-01-28 17:15:48 TRACE nova server.start()
2013-01-28 17:15:48 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 398, in start
2013-01-28 17:15:48 TRACE nova self.manager.init_host()
2013-01-28 17:15:48 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 272, in init_host
2013-01-28 17:15:48 TRACE nova self.driver.init_host(host=self.host)
2013-01-28 17:15:48 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/driver.py", line 164, in init_host
2013-01-28 17:15:48 TRACE nova vm_utils.ensure_correct_host(self._session)
2013-01-28 17:15:48 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vm_utils.py", line 2155, in ensure_correct_host
2013-01-28 17:15:48 TRACE nova this_vm_uuid = get_this_vm_uuid()
2013-01-28 17:15:48 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vm_utils.py", line 1803, in get_this_vm_uuid
2013-01-28 17:15:48 TRACE nova return f.readline().strip()
2013-01-28 17:15:48 TRACE nova IOError: [Errno 13] Permission denied
2013-01-28 17:15:48 TRACE nova

The table 'instances' is present in the database and the hypervisor is fully accessible. Those where installation issues that I already solved before testing. For example after uninstalling and then installing again the command "nova network-list" worked without problem. Only after I used "nova boot" the first time then "nova network-list" stopped to work. Something locked after nova boot.

While waiting for your kind response I set up a clone of the cluster controller where I managed to get everything working. This means that the compute node is ok as the config files on the controller.
I think I will erase the controller to get things done faster with the hope of never see this problem again.

Thanks for your help Keith ;)