OpenStack Compute (nova)

nova-compute reporting XXX

Asked by Jacob Godin

I have a few nodes setup in a development env. to test an OpenStack deployment. Currently, I'm trying to get the cloud controller to communicate with the compute node. The setup is as follows:

controller1 -
OpenStack services:
openstack-glance-api
openstack-glance-registry
openstack-keystone
openstack-nova-api
openstack-nova-cert
openstack-nova-console
openstack-nova-consoleauth
openstack-nova-metadata-api
openstack-nova-network
openstack-nova-objectstore
openstack-nova-scheduler
openstack-nova-volume
openstack-nova-xvpvncproxy

compute1 -
OpenStack services:
openstack-nova-compute

compute1 is running qemu via libvirt and appears to be able to connect properly:
2012-11-27 14:01:03 DEBUG nova.utils [req-e73569ba-9a7e-48e0-b955-bee3defcbb7f None None] backend <module 'nova.db.sqlalchemy.api' from '/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.pyc'> from (pid=4344) __get_backend /usr/lib/python2.6/site-packages/nova/utils.py:494
2012-11-27 14:01:04 DEBUG nova.utils [-] Got semaphore "compute_resources" for method "update_available_resource"... from (pid=4344) inner /usr/lib/python2.6/site-packages/nova/utils.py:713
2012-11-27 14:01:06 DEBUG nova.compute.resource_tracker [-] Hypervisor: free ram (MB): 3652 from (pid=4344) _report_hypervisor_resource_view /usr/lib/python2.6/site-packages/nova/compute/resource_tracker.py:447
2012-11-27 14:01:06 DEBUG nova.compute.resource_tracker [-] Hypervisor: free disk (GB): 5 from (pid=4344) _report_hypervisor_resource_view /usr/lib/python2.6/site-packages/nova/compute/resource_tracker.py:448
2012-11-27 14:01:06 DEBUG nova.compute.resource_tracker [-] Hypervisor: free VCPUs: 2 from (pid=4344) _report_hypervisor_resource_view /usr/lib/python2.6/site-packages/nova/compute/resource_tracker.py:453
2012-11-27 14:01:06 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 3321
2012-11-27 14:01:06 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 7
2012-11-27 14:01:06 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 2
2012-11-27 14:01:06 INFO nova.compute.resource_tracker [-] Compute_service record updated for compute1

compute1 is also able to connect to the DB on controller1:
[root@compute1 nova]# cat nova.conf | grep sql
sql_connection = mysql://nova:nova@10.0.0.10/nova
[root@compute1 nova]# mysql -u nova -pnova -h 10.0.0.10 -e "SHOW tables" nova
+-------------------------------------+
| Tables_in_nova |
+-------------------------------------+
| agent_builds |
| aggregate_hosts |
| aggregate_metadata |
| aggregates |
| block_device_mapping |
| bw_usage_cache |
| cells |
| certificates |
| compute_node_stats |
| compute_nodes |
| console_pools |
| consoles |
| dns_domains |
....

controller1 is not receiving status updates from compute1:
[root@controller1 nova]# nova-manage service list
2012-11-27 16:08:42 DEBUG nova.utils [req-16c46196-f79b-4d5a-a9f1-0b3c2f1f253c None None] backend <module 'nova.db.sqlalchemy.api' from '/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.pyc'> from (pid=30052) __get_backend /usr/lib/python2.6/site-packages/nova/utils.py:494
Binary Host Zone Status State Updated_At
nova-compute compute1 nova enabled XXX 2012-11-27 17:41:33
nova-network controller1 nova enabled :-) 2012-11-27 20:08:41
nova-scheduler controller1 nova enabled :-) 2012-11-27 20:08:38
nova-volume controller1 nova enabled :-) 2012-11-27 20:08:37
nova-console controller1 nova enabled :-) 2012-11-27 20:08:38
[root@controller1 nova]# mysql -u nova -pnova -e "select * from services" nova
+---------------------+---------------------+------------+---------+----+----------------------------------------+----------------+-----------+--------------+----------+-------------------+
| created_at | updated_at | deleted_at | deleted | id | host | binary | topic | report_count | disabled | availability_zone |
+---------------------+---------------------+------------+---------+----+----------------------------------------+----------------+-----------+--------------+----------+-------------------+
| 2012-11-22 12:35:57 | 2012-11-27 17:41:33 | NULL | 0 | 1 | compute1 | nova-compute | compute | 0 | 0 | nova |
| 2012-11-22 12:38:30 | 2012-11-27 20:09:32 | NULL | 0 | 2 | controller1 | nova-network | network | 35723 | 0 | nova |
| 2012-11-22 12:38:44 | 2012-11-27 20:09:28 | NULL | 0 | 3 | controller1 | nova-scheduler | scheduler | 35722 | 0 | nova |
| 2012-11-23 16:44:26 | 2012-11-27 20:09:28 | NULL | 0 | 4 | controller1 | nova-volume | volume | 35618 | 0 | nova |
| 2012-11-27 17:05:48 | 2012-11-27 20:09:28 | NULL | 0 | 5 | controller1 | nova-console | console | 1097 | 0 | nova

Dates are in sync via NTP.

Any ideas?

Question information

Language:
English Edit question
Status:
Solved
For:
OpenStack Compute (nova) Edit question
Assignee:
No assignee Edit question
Solved by:
Jacob Godin
Solved:
Last query:
Last reply:
Revision history for this message
Jacob Godin (jacobgodin) said : 		#1

Just a quick note that I'm running FOLSOM RPMs from EPEL

Revision history for this message
Sandy Walsh (sandy-walsh) said : 		#2

if you tail the compute log file, you should see periodic tasks firing every few minutes (for a variety of purposes).

Are you seeing this? Or, once launched, does the node go silent?

Revision history for this message
Jacob Godin (jacobgodin) said : 		#3

[root@compute1 nova]# ls -las
total 120
 4 drwxr-xr-x. 2 nova root 4096 Nov 28 03:24 .
 4 drwxr-xr-x. 12 root root 4096 Nov 27 12:48 ..
 0 -rw-r--r-- 1 nova nova 0 Nov 28 03:24 compute.log
64 -rw-r--r-- 1 nova nova 64539 Nov 28 03:24 compute.log-20121128.gz
 4 -rw-r--r--. 1 root root 3217 Nov 27 13:37 nova-manage.log
[root@compute1 nova]# date
Wed Nov 28 14:48:40 AST 2012
[root@compute1 nova]# cat compute.log
[root@compute1 nova]#

Revision history for this message
Jacob Godin (jacobgodin) said : 		#4

Also seeing the following new line if I run the compute service directly with nova-compute:

2012-11-29 09:22:20 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 7
2012-11-29 09:22:20 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 2
2012-11-29 09:22:20 INFO nova.compute.resource_tracker [-] Compute_service record updated for compute1.osc.hfx.dynamichosting.biz
No handlers could be found for logger "qpid.messaging" <--------

Looks like maybe the compute node either doesn't see the scheduler on the controller or can't communicate with it?

Revision history for this message
Jacob Godin (jacobgodin) said : 		#5

This appears to have been an issue with using Apache QPID. I removed QPID from my scheduler and replaced it with RabbitMQ. Everything seems happy now :-)

[root@controller1 rabbitmq]# nova-manage service list
2012-11-29 12:16:24 DEBUG nova.utils [req-8423af36-02ab-480f-a658-a1363a6e45ae None None] backend <module 'nova.db.sqlalchemy.api' from '/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.pyc'> from (pid=5789) __get_backend /usr/lib/python2.6/site-packages/nova/utils.py:494
Binary Host Zone Status State Updated_At
nova-compute compute1 nova enabled :-) 2012-11-29 16:16:19
nova-network controller1 nova enabled :-) 2012-11-29 16:16:21
nova-scheduler controller1 nova enabled :-) 2012-11-29 16:16:25
nova-volume controller1 nova enabled :-) 2012-11-29 16:16:20
nova-console controller1 nova enabled :-) 2012-11-29 16:16:24