OpenStack Compute (nova)

nova get-vnc-console does not return VNC url. Fails with HTTP 500 error

Asked by Elvinas Piliponis

Deployed a multinode OpenStack system from Ubuntu 12.04 supplied packages.

Everything seems to be working well except for VNC console in Dashboard. I have traced it to not being able to receive VNC url from nova. stack201 is main controller node with management services, stack208 compute node.

When running nova get-vnc-console on main controller node I do get:
---------------------------------------------------
root@stack201:/etc/keystone# nova --debug get-vnc-console 16034854-60ec-42d1-aeda-a9173213a447 novnc
connect: (stack201, 5000)
send: 'POST /v2.0/tokens HTTP/1.1\r\nHost: stack201:5000\r\nContent-Length: 112\r\ncontent-type: application/json\r\naccept-encoding: gzip, deflate\r\naccept: application/json\r\nuser-agent: python-novaclient\r\n\r\n{"auth": {"tenantName": "projecta", "passwordCredentials": {"username": "adminUser", "password": "secretword"}}}'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: application/json
header: Vary: X-Auth-Token
header: Date: Wed, 21 Mar 2012 09:06:36 GMT
header: Transfer-Encoding: chunked
connect: (stack201, 8774)
send: u'GET /v1.1/9ca6fca4d5c0409aa064cc461ac74d24/servers/16034854-60ec-42d1-aeda-a9173213a447 HTTP/1.1\r\nHost: stack201:8774\r\nx-auth-project-id: projecta\r\nx-auth-token: 44372f0179054d1db8d8ebe6ea198273\r\naccept-encoding: gzip, deflate\r\naccept: application/json\r\nuser-agent: python-novaclient\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: X-Compute-Request-Id: req-9a3022d1-0c5b-4f6c-9724-e0be591027fc
header: Content-Type: application/json
header: Content-Length: 1377
header: Date: Wed, 21 Mar 2012 09:06:36 GMT
send: u'GET /v1.1/9ca6fca4d5c0409aa064cc461ac74d24/servers/16034854-60ec-42d1-aeda-a9173213a447 HTTP/1.1\r\nHost: stack201:8774\r\nx-auth-project-id: projecta\r\nx-auth-token: 44372f0179054d1db8d8ebe6ea198273\r\naccept-encoding: gzip, deflate\r\naccept: application/json\r\nuser-agent: python-novaclient\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: X-Compute-Request-Id: req-c4ff6e00-78d5-48ad-9924-ce37f5b0c4f5
header: Content-Type: application/json
header: Content-Length: 1377
header: Date: Wed, 21 Mar 2012 09:06:36 GMT
send: u'POST /v1.1/9ca6fca4d5c0409aa064cc461ac74d24/servers/16034854-60ec-42d1-aeda-a9173213a447/action HTTP/1.1\r\nHost: stack201:8774\r\nContent-Length: 39\r\nx-auth-project-id: projecta\r\naccept-encoding: gzip, deflate\r\naccept: application/json\r\nx-auth-token: 44372f0179054d1db8d8ebe6ea198273\r\nuser-agent: python-novaclient\r\ncontent-type: application/json\r\n\r\n{"os-getVNCConsole": {"type": "novnc"}}'
reply: 'HTTP/1.1 500 Internal Server Error\r\n'
header: Content-Length: 128
header: Content-Type: application/json; charset=UTF-8
header: X-Compute-Request-Id: req-c8f1025d-65bf-4e96-822c-2eb54e23e97b
header: Date: Wed, 21 Mar 2012 09:07:37 GMT
DEBUG (shell:394) The server has either erred or is incapable of performing the requested operation. (HTTP 500)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 391, in main
    OpenStackComputeShell().main(sys.argv[1:])
  File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 342, in main
    args.func(self.cs, args)
  File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/shell.py", line 996, in do_get_vnc_console
    data = server.get_vnc_console(args.console_type)
  File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/servers.py", line 64, in get_vnc_console
    return self.manager.get_vnc_console(self, console_type)
  File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/servers.py", line 316, in get_vnc_console
    {'type': console_type})[1]
  File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/servers.py", line 589, in _action
    return self.api.client.post(url, body=body)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 137, in post
    return self._cs_request(url, 'POST', **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 122, in _cs_request
    **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 105, in request
    raise exceptions.from_response(resp, body)
ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500)
ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500)
-------------------------------------------------
On corresponding nova-compute server in the logs I do see:

2012-03-21 09:06:36 DEBUG nova.rpc.amqp [-] received {u'_context_roles': [u'cloudadmin', u'sysadmin', u'Admin', u'KeystoneServiceAdmin', u'netadmin', u
'Member', u'KeystoneAdmin'], u'_msg_id': u'1b08f0f52f714a9cbed6e8591f624bb0', u'_context_read_deleted': u'no', u'_context_request_id': u'req-c8f1025d-6
5bf-4e96-822c-2eb54e23e97b', u'args': {u'instance_uuid': u'16034854-60ec-42d1-aeda-a9173213a447', u'console_type': u'novnc'}, u'_context_auth_token': u
'44372f0179054d1db8d8ebe6ea198273', u'_context_is_admin': True, u'_context_project_id': u'9ca6fca4d5c0409aa064cc461ac74d24', u'_context_timestamp': u'2
012-03-21T09:06:36.771457', u'_context_user_id': u'f9cc33d443fd4a00a5eaba5b46d9b60e', u'method': u'get_vnc_console', u'_context_remote_address': u'2.0.
4.8'} from (pid=3618) _safe_log /usr/lib/python2.7/dist-packages/nova/rpc/common.py:144
2012-03-21 09:06:36 DEBUG nova.rpc.amqp [req-c8f1025d-65bf-4e96-822c-2eb54e23e97b f9cc33d443fd4a00a5eaba5b46d9b60e 9ca6fca4d5c0409aa064cc461ac74d24] un
packed context: {'request_id': u'req-c8f1025d-65bf-4e96-822c-2eb54e23e97b', 'user_id': u'f9cc33d443fd4a00a5eaba5b46d9b60e', 'roles': [u'cloudadmin', u'
sysadmin', u'Admin', u'KeystoneServiceAdmin', u'netadmin', u'Member', u'KeystoneAdmin', 'admin'], 'timestamp': '2012-03-21T09:06:36.771457', 'is_admin'
: True, 'auth_token': u'44372f0179054d1db8d8ebe6ea198273', 'project_id': u'9ca6fca4d5c0409aa064cc461ac74d24', 'remote_address': u'stack201-IP-ADDR', 'read_delet
ed': u'no'} from (pid=3618) unpack_context /usr/lib/python2.7/dist-packages/nova/rpc/amqp.py:188
2012-03-21 09:06:36 DEBUG nova.compute.manager [-] instance 16034854-60ec-42d1-aeda-a9173213a447: getting vnc console from (pid=3618) get_vnc_console /
usr/lib/python2.7/dist-packages/nova/compute/manager.py:1643

My nova conf:
--dhcpbridge_flagfile=/etc/nova/nova.conf
--dhcpbridge=/usr/bin/nova-dhcpbridge
--logdir=/var/log/nova
--state_path=/var/lib/nova
--lock_path=/var/lock/nova
--force_dhcp_release
--iscsi_helper=tgtadm
--libvirt_use_virtio_for_bridges
--connection_type=libvirt
--root_helper=sudo nova-rootwrap
--verbose
--debug

#Our stuffa
--s3_host=stack201
--rabbit_host=stack201
--cc_host=stack201
--nova_url=http://stack201:8774/v1.1/

--public_interface=eth0
--flat_interface=eth1
--network-host=stack201
--flat_network_bridge=br100

--network_manager = nova.network.manager.FlatDHCPManager

--fixed_range=10.0.0.0/16
--flat_network_dhcp_start=10.0.0.30
--network_size=32768
#--routing_source_ip=10.0.0.1
--routing_source_ip=stack201-IP-ADDR
--iscsi_ip_prefix=10.0.0
--use_ipv6=False

--image_service-nova.image.glance.GlanceImageService
--scheduler_driver=nova.scheduler.simple.SimpleScheduler
--keystone_ec2_url=http://stack201:5000/v2.0/ec2tokens
--auth_strategy=keystone

--sql_connection=mysql://novadbadmin:novasecret@stack201/nova

--glance_api_servers=stack202:9292

--vnc_console_proxy_url=http://stack201:6081
--xvpvncproxy_port=6081
--vnc_enabled=True
--novnc_enabled=True
--vncproxy_url=http://stack201:6081
--xvpvncproxy_base_url=http://stack201:6081/console
--novncproxy_base_url=http://stack201:6080/vnc_auto.html
#--vncserver_listen=0.0.0.0
#--vncserver_proxyclient_address="stack202"

--max_cores=48

On the computing node nova.conf does have:
--vncserver_listen=stack208
I have tried also 0.0.0.0 and KVM seems to be listening to this. Not sure about vncserver_proxyclient_address option. I tried to set it to controller node stack201, comment out, set only on the computing node. Description is a bit cryptic. when I did not set it, then 'remote_address': in compute node was u'127.0.0.1'. After I have tried to set it and restarted all services it changed to u'stack201-IP-ADDR' and left as so even after I did comment out the setting. Why? What is proper setting for it?

root@stack201:/etc/keystone# dpkg -l | grep keystone
ii keystone 2012.1~rc1~20120308.2103-0ubuntu1 OpenStack identity service - Daemons
ii keystone-doc 2012.1~rc1~20120308.2103-0ubuntu1 OpenStack identity service - Documentation
ii python-keystone 2012.1~rc1~20120308.2103-0ubuntu1 OpenStack identity service - Python library
ii python-keystoneclient 2012.1~rc1~20120307.0-0ubuntu1 Client libary for Openstack Keystone API
root@stack201:/etc/keystone# dpkg -l | grep nova
ii nova-api 2012.1~rc1~20120309.13261-0ubuntu1 OpenStack Compute - API frontend
ii nova-cert 2012.1~rc1~20120309.13261-0ubuntu1 OpenStack Compute - virtual machine scheduler
ii nova-common 2012.1~rc1~20120309.13261-0ubuntu1 OpenStack Compute - common files
ii nova-doc 2012.1~rc1~20120309.13261-0ubuntu1 OpenStack Compute - documetation
ii nova-network 2012.1~rc1~20120309.13261-0ubuntu1 OpenStack Compute - Network manager
ii nova-objectstore 2012.1~rc1~20120309.13261-0ubuntu1 OpenStack Compute - object store
ii nova-scheduler 2012.1~rc1~20120309.13261-0ubuntu1 OpenStack Compute - virtual machine scheduler
ii nova-vncproxy 2012.1~rc1~20120309.13261-0ubuntu1 OpenStack Compute - VNC proxy
ii python-nova 2012.1~rc1~20120309.13261-0ubuntu1 OpenStack Compute Python libraries
ii python-novaclient 2012.1~rc1~20120307.520-0ubuntu1 client library for OpenStack Compute API

Any suggestions? If you need anything more, please say so as I am stuck a bit.

Thanks

Question information

Language:
English Edit question
Status:
Solved
For:
OpenStack Compute (nova) Edit question
Assignee:
No assignee Edit question
Solved by:
Elvinas Piliponis
Solved:
Last query:
Last reply:
Revision history for this message
Elvinas Piliponis (elvinas-piliponis) said : 		#1

Issue partially solved. This was both configuration and Ubuntu packaging issue. Ubuntu did not have nova-consoleauth package (not nova-console which is deprecated XEN related package). Also ubuntu 12.04 packages mentioned below still DOES NOT have FULL nova-vncproxy functionality so while the update fixes "nova get-vnc-console" issue, VNC console still does not work in DashBoard! See below:

I have updated local mirror (as servers does not have external access) and installed nova-consoleauth on main controller and updated packages to the following versions:
--------------------------------------
root@stack201:~# dpkg -l | grep nova
ii nova-api 2012.1~rc1-0ubuntu2 OpenStack Compute - API frontend
ii nova-cert 2012.1~rc1-0ubuntu2 OpenStack Compute - certificate management
ii nova-common 2012.1~rc1-0ubuntu2 OpenStack Compute - common files
ii nova-consoleauth 2012.1~rc1-0ubuntu2 OpenStack Compute - Console Authenticator
ii nova-doc 2012.1~rc1~20120316.13416-0ubuntu1 OpenStack Compute - documentation
ii nova-network 2012.1~rc1-0ubuntu2 OpenStack Compute - Network manager
ii nova-objectstore 2012.1~rc1-0ubuntu2 OpenStack Compute - object store
ii nova-scheduler 2012.1~rc1-0ubuntu2 OpenStack Compute - virtual machine scheduler
ii nova-vncproxy 2012.1~rc1-0ubuntu2 OpenStack Compute - VNC proxy
ii python-nova 2012.1~rc1-0ubuntu2 OpenStack Compute Python libraries
ii python-novaclient 2012.1~rc1-0ubuntu1 client library for OpenStack Compute API
---------------------------

This update fixes URL retrieval but not Dashboard issue. To allow Dashboard VNC access NoVNC is required which is not included in "RC1-0" packages yet. Package nova-vncproxy does not contain nova-vnxproxy (it is actually removed in favour of not yet included novncproxy) but only nova-xvpvncproxy, which is different and does not provide Nova Dashboard functionality I had to download nova-vncproxy manually from https://github.com/cloudbuilders/noVNC as zip file, unpacked it to /home/localadmin/novnc and create the following upstart init script:
--------------------------------------------
root@stack201:~# cat /etc/init/nova-vncproxy.conf
description "Nova VNC proxy"
author "Vishvananda Ishaya <email address hidden>"

start on (filesystem and net-device-up IFACE!=lo)
stop on runlevel [016]

chdir /var/run

pre-start script
        mkdir -p /var/run/nova
        chown nova:root /var/run/nova/

        mkdir -p /var/lock/nova
        chown nova:root /var/lock/nova/
end script

exec su -s /bin/sh -c "exec nova-xvpvncproxy --flagfile=/etc/nova/nova.conf" nova
root@stack201:~# cat /etc/init/nova-git-vncproxy.conf
description "Nova VNC proxy GIT"
author "Vishvananda Ishaya <email address hidden>"

start on (filesystem and net-device-up IFACE!=lo)
stop on runlevel [016]

chdir /home/localadmin/novnc/

pre-start script
        mkdir -p /var/run/nova
        chown nova:root /var/run/nova/

        mkdir -p /var/lock/nova
        chown nova:root /var/lock/nova/
end script

exec su -s /bin/sh -c "exec /home/localadmin/novnc/utils/nova-novncproxy --flagfile=/etc/nova/nova.conf" nova
-----------------------------------

After ln -s /lib/init/upstart-job /etc/init.d/nova-git-vncproxy I am able to start NoVNC proxy service and DashBoard functionality.