ClientConnectionError: Unable to connect to server.

Asked by R. Krishna Kumar

Hello,

We are trying to configure multiple compute nodes. The hypervisor we are using is VMware.
There are two machines running:
1. Cloud controller (all the nova services are running. This server also has "glance" for image) : IP - 10.207.230.122
2. Compute node (nova-compute is installed - using nova-install -t compute) : IP - 10.207.230.121

We are able to launch multiple instances through single node. However when we try to launch using multi-node configuration, we are encountering an error. The instances are launched successfully on the blade addressed in /etc/nova/nova.conf file of Cloud controller but not on the blade mentioned in compute node. Following are the /etc/nova/nova.conf files

Cloud controller:
--dhcpbridge_flagfile=/etc/nova/nova.conf
--dhcpbridge=/usr/bin/nova-dhcpbridge
--logdir=/var/log/nova
--state_path=/var/lib/nova
--lock_path=/var/lock/nova
--verbose
--s3_host=127.0.0.1
--rabbit_host=127.0.0.1
--cc_host=127.0.0.1
--ec2_url=http://127.0.0.1:8773/services/Cloud
--fixed_range=192.168.1.0/24
--network_size=255
--FAKE_subdomain=ec2
--routing_source_ip=127.0.0.1
--verbose
--sql_connection=mysql://root:wipro@123@127.0.0.1/nova
--network_manager=nova.network.manager.FlatManager
--connection_type=vmwareapi
--vmwareapi_host_ip=10.207.228.71
--vmwareapi_host_username=arun
--vmwareapi_host_password=wipro@123
--vmwareapi_wsdl_loc=http://127.0.0.1:8080/wsdl/vim25/vimService.wsdl
--glance_api_servers=127.0.0.1:9292
--image_service=nova.image.glance.GlanceImageService

compute node:
--dhcpbridge_flagfile=/etc/nova/nova.conf
--dhcpbridge=/usr/bin/nova-dhcpbridge
--logdir=/var/log/nova
--state_path=/var/lib/nova
--lock_path=/var/lock/nova
--verbose
--s3_host=10.207.230.122
--rabbit_host=10.207.230.122
--cc_host=10.207.230.122
--ec2_url=http://10.207.230.122:8773/services/Cloud
--sql_connection=mysql://root:wipro@123@10.207.230.122/nova
--network_manager=nova.network.manager.FlatManager
--connection_type=vmwareapi
--vmwareapi_host_ip=10.207.228.70
--vmwareapi_host_username=root
--vmwareapi_host_password=CISCOCOE@123
--vmwareapi_wsdl_loc=http://10.207.230.122:8080/wsdl/vim25/vimService.wsdl
--glance_api_servers=10.207.230.122:9292
--image_service=nova.image.glance.GlanceImageService

following is the output from /var/log/nova/nova-compute.log from "compute node":
2011-09-20 12:09:58,353 DEBUG suds.metrics [-] method 'RetrieveProperties' invoked: 551 (ms) from (pid=2427) invoke /usr/lib/pymodules/python2.7/suds/client.py:607
2011-09-20 12:09:58,354 DEBUG nova.virt.vmwareapi.vmware_images [-] Getting image size for the image 1 from (pid=2427) get_vmdk_size_and_properties /usr/lib/pymodules/python2.7/nova/virt/vmwareapi/vmware_images.py:189
2011-09-20 12:09:58,354 ERROR nova.compute.manager [FYF93F85H-DP8GUSOH9K arun openstack] Instance '65' failed to spawn. Is virtualization enabled in the BIOS?
(nova.compute.manager): TRACE: Traceback (most recent call last):
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.7/nova/compute/manager.py", line 234, in run_instance
(nova.compute.manager): TRACE: self.driver.spawn(instance_ref)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.7/nova/virt/vmwareapi_conn.py", line 129, in spawn
(nova.compute.manager): TRACE: self._vmops.spawn(instance)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.7/nova/virt/vmwareapi/vmops.py", line 162, in spawn
(nova.compute.manager): TRACE: vmdk_file_size_in_kb, os_type, adapter_type = _get_image_properties()
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.7/nova/virt/vmwareapi/vmops.py", line 155, in _get_image_properties
(nova.compute.manager): TRACE: instance.image_id, instance)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.7/nova/virt/vmwareapi/vmware_images.py", line 193, in get_vmdk_size_and_properties
(nova.compute.manager): TRACE: meta_data = glance_client.get_image_meta(image)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.7/glance/client.py", line 235, in get_image_meta
(nova.compute.manager): TRACE: res = self.do_request("HEAD", "/images/%s" % image_id)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.7/glance/client.py", line 167, in do_request
(nova.compute.manager): TRACE: "server. Got error: %s" % e)
(nova.compute.manager): TRACE: ClientConnectionError: Unable to connect to server. Got error: [Errno 111] ECONNREFUSED
(nova.compute.manager): TRACE:
2011-09-20 12:09:58,444 DEBUG suds.resolver [-] searching schema for (ns0:SelectionSpec) from (pid=2427) root /usr/lib/pymodules/python2.7/suds/resolver.py:128
2011-09-20 12:09:58,444 DEBUG suds.xsd.query [-] ('SelectionSpec', urn:vim25), found as: <Complex:0xc49ea4c name="SelectionSpec" /> from (pid=2427) result /usr/lib/pymodules/python2.7/suds/xsd/query.py:85
2011-09-20 12:09:58,444 DEBUG suds.resolver [-] found (ns0:SelectionSpec) as (<Complex:0xc49ea4c name="SelectionSpec" />) from (pid=2427) root /usr/lib/pymodules/python2.7/suds/resolver.py:136
2011-09-20 12:09:58,445 DEBUG suds.metrics [-] ns0:SelectionSpec created: 0 (ms) from (pid=2427) create /usr/lib/pymodules/python2.7/suds/client.py:246
2011-09-20 12:09:58,445 DEBUG suds.resolver [-] searching schema for (ns0:TraversalSpec) from (pid=2427) root /usr/lib/pymodules/python2.7/suds/resolver.py:128
2011-09-20 12:09:58,445 DEBUG suds.xsd.query [-] ('TraversalSpec', urn:vim25), found as: <Complex:0xc49eccc name="TraversalSpec" /> from (pid=2427) result /usr/lib/pymodules/python2.7/suds/xsd/query.py:85
2011-09-20 12:09:58,445 DEBUG suds.resolver [-] found (ns0:TraversalSpec) as (<Complex:0xc49eccc name="TraversalSpec" />) from (pid=2427) root /usr/lib/pymodules/python2.7/suds/resolver.py:136
2011-09-20 12:09:58,446 DEBUG suds.metrics [-] ns0:TraversalSpec created: 0 (ms) from (pid=2427) create /usr/lib/pymodules/python2.7/suds/client.py:246
2011-09-20 12:09:58,446 DEBUG suds.resolver [-] searching schema for (ns0:TraversalSpec) from (pid=2427) root /usr/lib/pymodules/python2.7/suds/resolver.py:128
2011-09-20 12:09:58,446 DEBUG suds.xsd.query [-] ('TraversalSpec', urn:vim25), found as: <Complex:0xc49eccc name="TraversalSpec" /> from (pid=2427) result /usr/lib/pymodules/python2.7/suds/xsd/query.py:85
2011-09-20 12:09:58,446 DEBUG suds.resolver [-] found (ns0:TraversalSpec) as (<Complex:0xc49eccc name="TraversalSpec" />) from (pid=2427) root /usr/lib/pymodules/python2.7/suds/resolver.py:136
2011-09-20 12:09:58,447 DEBUG suds.metrics [-] ns0:TraversalSpec created: 0 (ms) from (pid=2427) create /usr/lib/pymodules/python2.7/suds/client.py:246
2011-09-20 12:09:58,447 DEBUG suds.resolver [-] searching schema for (ns0:TraversalSpec) from (pid=2427) root /usr/lib/pymodules/python2.7/suds/resolver.py:128
2011-09-20 12:09:58,447 DEBUG suds.xsd.query [-] ('TraversalSpec', urn:vim25), found as: <Complex:0xc49eccc name="TraversalSpec" /> from (pid=2427) result /usr/lib/pymodules/python2.7/suds/xsd/query.py:85
2011-09-20 12:09:58,447 DEBUG suds.resolver [-] found (ns0:TraversalSpec) as (<Complex:0xc49eccc name="TraversalSpec" />) from (pid=2427) root /usr/lib/pymodules/python2.7/suds/resolver.py:136
2011-09-20 12:09:58,448 DEBUG suds.metrics [-] ns0:TraversalSpec created: 0 (ms) from (pid=2427) create /usr/lib/pymodules/python2.7/suds/client.py:246

There are no errors in nova-compute.log present in the Cloud controller. On searching the net, we found that this was because the compute node was not able to communicate with the glance server. So we tried to telnet the glance ip from compute node and we were able to do it without any problems.

Any help would be highly appreciated.

Thanks

Question information

Language:
English Edit question
Status:
Solved
For:
OpenStack Compute (nova) Edit question
Assignee:
No assignee Edit question
Solved by:
R. Krishna Kumar
Solved:
Last query:
Last reply:
Revision history for this message
Sateesh (sateesh-chodapuneedi) said :
#1

It looks like your glance server is accessible from compute node.
Can you try telnet to glance ports (9292,9191)?

Hope your glance server is bound to address 0.0.0.0, this can be verified in glance configuration files
Also, in case if you modify the configuration file nova.conf, make sure nova-compute service is restarted.

Revision history for this message
R. Krishna Kumar (krishna-kumar2) said :
#2

Thanks for the reply. We were able to telnet to the glance ports (9191 and 9292) as below:
root@krishna-VMachine:~/python# telnet 10.207.230.126 9292
Trying 10.207.230.126...
Connected to 10.207.230.126.
Escape character is '^]'.

root@krishna-VMachine:~/python# telnet 10.207.230.126 9191
Trying 10.207.230.126...
Connected to 10.207.230.126.
Escape character is '^]'.

Following are the entries in /etc/glance/glance.conf for ports:
bind_host = 0.0.0.0
bind_port = 9292
registry_host = 0.0.0.0
registry_port = 9191

We restarted the nova services whenever we modified the nova.conf file.

Despite all this, the problem still persists. Kindly let us know if we are missing anything.

Thanks again.

Revision history for this message
Sateesh (sateesh-chodapuneedi) said :
#3

I see that your glance services are running on -glance_api_servers=10.207.230.122:9292 as per the flags specified in the above description, but you are trying telnet to 10.207.230.126:9292.
If 10.207.230.126 is the server with glance running, please upate the flags accordingly.

Hope that helps.

Revision history for this message
R. Krishna Kumar (krishna-kumar2) said :
#4

Very sorry. That was a copying mistake. Telnet to 10.207.230.122 yields the same result. Thanks Sateesh for all the responses. We figured out the error and resolved it.

Revision history for this message
Lucky T (tejeshmk) said :
#5

hi krishna, can you share me how you resolved this issue. i'm getting the same error.

Revision history for this message
R. Krishna Kumar (krishna-kumar2) said :
#6

Hi Lucky,

Try adding the following line in your compute node's /etc/nova/nova.conf file:

--glance_host=10.207.230.122

The IP mentioned above is our cloud controller where glance is also running. This line is needed because when we checked the code, openstack was looking for this tag to get the IP where glance is running.

Add this line and restart nova services in both Cloud controller and compute node.
Let me know if this works.

Thanks