Devstack stable/essex release issue

Asked by Neelakantam

Hi,

We are trying to test the openstack with quantum stable/essex release using devstack. We are seeing the following issue at our setup:

=========================================================

+ /opt/stack/nova/bin/nova-manage network create private 10.0.0.0/24 1 4096
2012-07-16 21:47:37 WARNING nova.network.quantum.manager [req-17beca79-321c-4645-9614-8aad8a1548c9 None None] Ignoring unnecessary parameter 'network_size'
2012-07-16 21:47:37 WARNING nova.network.quantum.manager [req-17beca79-321c-4645-9614-8aad8a1548c9 None None] QuantumManager does not use 'bridge' parameter.
2012-07-16 21:47:37 WARNING nova.network.quantum.manager [req-17beca79-321c-4645-9614-8aad8a1548c9 None None] QuantumManager does not use 'bridge_interface' parameter.
2012-07-16 21:47:37 DEBUG nova.network.quantum.quantum_connection [req-17beca79-321c-4645-9614-8aad8a1548c9 None None] Quantum Client Request: POST /v1.1/tenants/default/networks.json from (pid=15691) do_request /opt/stack/nova/nova/network/quantum/client.py:182
2012-07-16 21:47:37 DEBUG nova.network.quantum.quantum_connection [req-17beca79-321c-4645-9614-8aad8a1548c9 None None] {"network": {"name": "private", "nova_id": "nova"}} from (pid=15691) do_request /opt/stack/nova/nova/network/quantum/client.py:184
Command failed, please check log for more info
++ failed
++ local r=1
++ set +o xtrace
stack.sh failed: full log in /home/stack/logs/stack.log.2012-07-16-214633

=========================================================

Our localrc file is as follows:

==============================================================
# Hosts
HOST_IP=<ip of eth0>

# Services
ENABLED_SERVICES=g-api,g-reg,key,n-api,n-crt,n-obj,n-net,n-vol,n-sch,n-novnc,n-xvnc,n-cauth,horizon,mysql,rabbit,quantum,q-svc,q-agt,n-cpu

# Networks
FIXED_NETWORK_SIZE=4096
FIXED_RANGE=10.0.0.0/24
FLAT_INTERFACE=eth1
FLOATING_RANGE=172.16.142.128/25
Q_INTERFACE=eth1

# Misc

MULTI_HOST=0
SERVICE_TOKEN=xyzpdqlazydog
SYSLOG=True
Q_PLUGIN=openvswitch

SYSLOG=True
UPLOAD_LEGACY_TTY=1
DEFAULT_IMAGE_NAME=tty-quantum

GLANCE_BRANCH=stable/essex
HORIZON_BRANCH=stable/essex
KEYSTONE_BRANCH=stable/essex
NOVA_BRANCH=stable/essex
QUANTUM_BRANCH=stable/essex
SWIFT_BRANCH=1.4.8

===============================================================================

Quantum server logs :

cd /opt/stack/quantum && PYTHO ^MNPATH=.:/opt/stack/python-quantumclient: python /opt/stack/quantum/bin/quantum-s ^Merver /etc/quantum/quantum.conf
Traceback (most recent call last):
  File "/opt/stack/quantum/bin/quantum-server", line 22, in <module>
    from quantum.server import main as server
  File "/opt/stack/quantum/quantum/server/__init__.py", line 36, in <module>
    from quantum import service
  File "/opt/stack/quantum/quantum/service.py", line 19, in <module>
    from quantum.common import config
ImportError: No module named common

We are trying this test on ubuntu 12.04 64 bit version. Did we miss anything in our configuration ? Please help us solving this issue.

Question information

Language:
English Edit question
Status:
Solved
For:
neutron Edit question
Assignee:
No assignee Edit question
Solved by:
dan wendlandt
Solved:
Last query:
Last reply:
Revision history for this message
dan wendlandt (danwent) said :
#1

The quantum server logs point to the problem. Quantum is not running, as it cannot import quantum.common

Most likely, this is because you are running the stable/essex branch for the 'quantum' repo, but not for the 'python-quantumclient'. During essex, quantum-server imported quantum.common from the python-quantumclient, but in folsom this was removed. You likely have a folsom version of python-quantumclient, and hence, there is no quantum.common to import.

Revision history for this message
Neelakantam (neelugaddam) said :
#2

Hi Dan,

Thanks for your reply.

You are correct we are running stable/essex for quantum. But which branch of 'python-quantumclient' works for quantum stable/essex release. What is the working combination of branches for nova, python nova client and keystone, python keystone client. Please help me understanding the working combination of all openstack components. Is this information documented anywhere ?

Revision history for this message
Sumit Naiksatam (snaiksat) said :
#3

Can you try the following config in your stackrc:

# quantum client
QUANTUM_CLIENT_REPO=https://github.com/openstack/python-quantumclient
QUANTUM_CLIENT_BRANCH=2012.1

Revision history for this message
Neelakantam (neelugaddam) said :
#4

Hi,

I have successfully installed openstack with the following changes on controller localrc file.
I can launch and ping the VMs on this single node setup.

Changes in localrc file on controller :
--------------------------------------

GLANCE_BRANCH=stable/essex
HORIZON_BRANCH=stable/essex
KEYSTONE_BRANCH=stable/essex
NOVA_BRANCH=stable/essex
QUANTUM_BRANCH=stable/essex
SWIFT_BRANCH=1.4.8

KEYSTONECLIENT_BRANCH=milestone-proposed
NOVACLIENT_BRANCH=2012.1
QUANTUM_CLIENT_BRANCH=2012.1

Changes /etc/libvirt/qemu.conf of controller node.
--------------------------------------------------

Add the floowing to the file : /etc/libvirt/qemu.conf

cgroup_device_acl = [
   "/dev/null", "/dev/full", "/dev/zero",
   "/dev/random", "/dev/urandom",
   "/dev/ptmx", "/dev/kvm", "/dev/kqemu",
   "/dev/rtc", "/dev/hpet", "/dev/net/tun",
]

When I try add the compute node with following configuration, VMs not getting launched on compute node.
From the logs, there seems to be a database syncing issue. But I verified the versions of mysql and nova on controller and compute nodes and they are same.

localrc on compute node:
------------------------

ENABLED_SERVICES=n-cpu,n-api,quantum,q-agt
Q_PLUGIN=openvswitch
LIBVIRT_TYPE=qemu

HOST_IP=<etho IP>

Q_INTERFACE=eth1

GLANCE_HOSTPORT=<controller ip>:9292
KEYSTONE_HOST=<controller ip>
MYSQL_HOST=<controller ip>
Q_HOST=<controller ip>
RABBIT_HOST=<controller ip>
SYSLOG_HOST=<controller ip>

NOVA_BRANCH=stable/essex
NOVACLIENT_BRANCH=2012.1
QUANTUM_BRANCH=stable/essex
QUANTUM_CLIENT_BRANCH=2012.1
KEYSTONECLIENT_BRANCH=milestone-proposed

QUANTUM_CONF_DIR=/etc/quantum # This is needed because QUANTUM_CONF_DIR is not initialized.

nova compute logs on compute node:
-----------------------------------

2012-07-17 19:40:54 TRACE nova.manager Traceback (most recent call last):
2012-07-17 19:40:54 TRACE nova.manager File "/opt/stack/nova/nova/manager.py", line 167, in periodic_tasks
2012-07-17 19:40:54 TRACE nova.manager task(self, context)
2012-07-17 19:40:54 TRACE nova.manager File "/opt/stack/nova/nova/compute/manager.py", line 2293, in _heal_instance_info_cache
2012-07-17 19:40:54 TRACE nova.manager context, self.host)
2012-07-17 19:40:54 TRACE nova.manager File "/opt/stack/nova/nova/db/api.py", line 617, in instance_get_all_by_host
2012-07-17 19:40:54 TRACE nova.manager return IMPL.instance_get_all_by_host(context, host)
2012-07-17 19:40:54 TRACE nova.manager File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 114, in wrapper
2012-07-17 19:40:54 TRACE nova.manager return f(*args, **kwargs)
2012-07-17 19:40:54 TRACE nova.manager File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1619, in instance_get_all_by_host
2012-07-17 19:40:54 TRACE nova.manager return _instance_get_all_query(context).filter_by(host=host).all()
2012-07-17 19:40:54 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 1947, in all
2012-07-17 19:40:54 TRACE nova.manager return list(self)
2012-07-17 19:40:54 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2057, in __iter__
2012-07-17 19:40:54 TRACE nova.manager return self._execute_and_instances(context)
2012-07-17 19:40:54 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2072, in _execute_and_instances
2012-07-17 19:40:54 TRACE nova.manager result = conn.execute(querycontext.statement, self._params)
2012-07-17 19:40:54 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1405, in execute
2012-07-17 19:40:54 TRACE nova.manager params)
2012-07-17 19:40:54 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1538, in _execute_clauseelement
2012-07-17 19:40:54 TRACE nova.manager compiled_sql, distilled_params
2012-07-17 19:40:54 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1646, in _execute_context
2012-07-17 19:40:54 TRACE nova.manager context)
2012-07-17 19:40:54 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1639, in _execute_context
2012-07-17 19:40:54 TRACE nova.manager context)
2012-07-17 19:40:54 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 330, in do_execute
2012-07-17 19:40:54 TRACE nova.manager cursor.execute(statement, parameters)
2012-07-17 19:40:54 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
2012-07-17 19:40:54 TRACE nova.manager self.errorhandler(self, exc, value)
2012-07-17 19:40:54 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
2012-07-17 19:40:54 TRACE nova.manager raise errorclass, errorvalue
2012-07-17 19:40:54 TRACE nova.manager OperationalError: (OperationalError) (1054, "Unknown column 'instance_types_1.disabled' in 'field list'") 'SELECT instances.created_at AS instances_created_at, instances.updated_at AS instances_updated_at, instances.deleted_at AS instances_deleted_at, instances.deleted AS instances_deleted, instances.id AS instances_id, instances.user_id AS instances_user_id, instances.project_id AS instances_project_id, instances.image_ref AS instances_image_ref, instances.kernel_id AS instances_kernel_id, instances.ramdisk_id AS instances_ramdisk_id, instances.server_name AS instances_server_name, instances.launch_index AS instances_launch_index, instances.key_name AS instances_key_name, instances.key_data AS instances_key_data, instances.power_state AS instances_power_state, instances.vm_state AS instances_vm_state, instances.task_state AS instances_task_state, instances.memory_mb AS instances_memory_mb, instances.vcpus AS instances_vcpus, instances.root_gb AS instances_root_gb, instances.ephemeral_gb AS instances_ephemeral_gb, instances.hostname AS instances_hostname, instances.host AS instances_host, instances.instance_type_id AS instances_instance_type_id, instances.user_data AS instances_user_data, instances.reservation_id AS instances_reservation_id, instances.scheduled_at AS instances_scheduled_at, instances.launched_at AS instances_launched_at, instances.terminated_at AS instances_terminated_at, instances.availability_zone AS instances_availability_zone, instances.display_name AS instances_display_name, instances.display_description AS instances_display_description, instances.launched_on AS instances_launched_on, instances.locked AS instances_locked, instances.os_type AS instances_os_type, instances.architecture AS instances_architecture, instances.vm_mode AS instances_vm_mode, instances.uuid AS instances_uuid, instances.root_device_name AS instances_root_device_name, instances.default_ephemeral_device AS instances_default_ephemeral_device, instances.default_swap_device AS instances_default_swap_device, instances.config_drive AS instances_config_drive, instances.access_ip_v4 AS instances_access_ip_v4, instances.access_ip_v6 AS instances_access_ip_v6, instances.auto_disk_config AS instances_auto_disk_config, instances.progress AS instances_progress, instances.shutdown_terminate AS instances_shutdown_terminate, instances.disable_terminate AS instances_disable_terminate, instance_types_1.created_at AS instance_types_1_created_at, instance_types_1.updated_at AS instance_types_1_updated_at, instance_types_1.deleted_at AS instance_types_1_deleted_at, instance_types_1.deleted AS instance_types_1_deleted, instance_types_1.id AS instance_types_1_id, instance_types_1.name AS instance_types_1_name, instance_types_1.memory_mb AS instance_types_1_memory_mb, instance_types_1.vcpus AS instance_types_1_vcpus, instance_types_1.root_gb AS instance_types_1_root_gb, instance_types_1.ephemeral_gb AS instance_types_1_ephemeral_gb, instance_types_1.flavorid AS instance_types_1_flavorid, instance_types_1.swap AS instance_types_1_swap, instance_types_1.rxtx_factor AS instance_types_1_rxtx_factor, instance_types_1.vcpu_weight AS instance_types_1_vcpu_weight, instance_types_1.disabled AS instance_types_1_disabled, security_groups_1.created_at AS security_groups_1_created_at, security_groups_1.updated_at AS security_groups_1_updated_at, security_groups_1.deleted_at AS security_groups_1_deleted_at, security_groups_1.deleted AS security_groups_1_deleted, security_groups_1.id AS security_groups_1_id, security_groups_1.name AS security_groups_1_name, security_groups_1.description AS security_groups_1_description, security_groups_1.user_id AS security_groups_1_user_id, security_groups_1.project_id AS security_groups_1_project_id, instance_metadata_1.created_at AS instance_metadata_1_created_at, instance_metadata_1.updated_at AS instance_metadata_1_updated_at, instance_metadata_1.deleted_at AS instance_metadata_1_deleted_at, instance_metadata_1.deleted AS instance_metadata_1_deleted, instance_metadata_1.id AS instance_metadata_1_id, instance_metadata_1.`key` AS instance_metadata_1_key, instance_metadata_1.value AS instance_metadata_1_value, instance_metadata_1.instance_uuid AS instance_metadata_1_instance_uuid, instance_info_caches_1.created_at AS instance_info_caches_1_created_at, instance_info_caches_1.updated_at AS instance_info_caches_1_updated_at, instance_info_caches_1.deleted_at AS instance_info_caches_1_deleted_at, instance_info_caches_1.deleted AS instance_info_caches_1_deleted, instance_info_caches_1.id AS instance_info_caches_1_id, instance_info_caches_1.network_info AS instance_info_caches_1_network_info, instance_info_caches_1.instance_uuid AS instance_info_caches_1_instance_uuid \nFROM instances LEFT OUTER JOIN instance_types AS instance_types_1 ON instances.instance_type_id = instance_types_1.id LEFT OUTER JOIN security_group_instance_association AS security_group_instance_association_1 ON security_group_instance_association_1.instance_uuid = instances.uuid AND instances.deleted = %s LEFT OUTER JOIN security_groups AS security_groups_1 ON security_groups_1.id = security_group_instance_association_1.security_group_id AND security_group_instance_association_1.deleted = %s AND security_groups_1.deleted = %s LEFT OUTER JOIN instance_metadata AS instance_metadata_1 ON instance_metadata_1.instance_uuid = instances.uuid AND instance_metadata_1.deleted = %s LEFT OUTER JOIN instance_info_caches AS instance_info_caches_1 ON instance_info_caches_1.instance_uuid = instances.uuid \nWHERE instances.deleted = %s AND instances.host = %s' (0, 0, 0, 0, 0, 'nvp-desktop')
2012-07-17 19:40:54 TRACE nova.manager

I am using stable/essex release of devstack. Please help me to solve this error. What is the way to do multi-node stable/essex setup using devstack and quantum .

Thanks for your help.

Revision history for this message
dan wendlandt (danwent) said :
#5

this may be a database schema mismatch between what you have in mysql and what the code is expecting. Are you seeing this even if you unstack and restack? I've seen a related issue where stale pyc file in /opt/stack/nova caused problems as well, so I would suggest making sure you pull down a clean directory with the stable essex code, rather than re-using an /opt/stack/nova directory that had previously contained folsom code.

Revision history for this message
Neelakantam (neelugaddam) said :
#6

Hi Dan,

Yes, I am seeing the same issue when I did unstack and restack also. I did a fresh checkout of nova from stable/essex after deleting /opt/stack directory and observed the same issue. Also observed the same issue when I use the default branches of devstack of stable/essex (without changing the branch in localrc). Is there any work around to solve this isse ? Please help me .

Revision history for this message
Best dan wendlandt (danwent) said :
#7

at this point, this is not a quantum issue, its a nova issue, and since its beyond my shallow knowledge of nova, I'd suggest switching it from the quantum project to the nova project above and trying to get help there.

Revision history for this message
Neelakantam (neelugaddam) said :
#8

Thanks dan wendlandt, that solved my question.

Revision history for this message
Kurt (burnbrighter) said :
#9

What solved your question? How was this fixed? What solution/changes were made for stable/essex to work? Thanks

Revision history for this message
Neelakantam (neelugaddam) said :
#10

I removed /opt/stack/nova/ directory and ran the stack.sh again to reclone the nova code. With this, I did not see the issue again.

As Dan said, I too observed some stale .pyc files in the/opt/stack/nova/ in the code that was not working. Due to slow internet connection, nova is not cloning properly in my setup. So I did a manual clone of nova and copied to /opt/stack. This produced the issue for me.

Revision history for this message
Kurt (burnbrighter) said :
#11

Thanks for the reply. I have managed to get my multi-node stable/essex running, though I haven't tried to create VMs yet. That's next.