OpenStack Compute (Nova)

Trying to use zeroMQ, but it doesn't work.

Asked by Sippakorn Tansutthiwess on 2012-05-05

I try to use zeroMQ plug-in. However, when I created new instance, it throw errors. Could you help me to figure out what I'm done wrong? Or if you used zeroMQ before, which version and setup did you use? I can't find any full document about setup zeroMQ. I appreciate for every help.

This is my setup:
- Ubuntu 12.04 LTS
- Nova 2012.1-0ubuntu2 (the one that come with default apt-get repository)
- Glance 2012.1-0ubuntu2
- ZeroMQ 2.2.0 (installation steps from http://drumcoder.co.uk/blog/2010/dec/05/simple-zeromq-ubuntu/, but package from official website http://www.zeromq.org/intro:get-the-software)
- ZeroMQ plug-in from https://github.com/cloudscaling/nova-mq

This is my configuration files:

nova.conf

--multi_host=true
#nova.conf file for single node, FlatDHCP networking with keystone
--verbose
--rpc_zmq_bind_address=0.0.0.0
--rpc_zmq_broker_ip=192.168.245.64
--rpc_zmq_matchmaker=MatchMakerFanoutRing

#logging and other administrative
--dhcpbridge_flagfile=/etc/nova/nova.conf
--dhcpbridge=/usr/bin/nova-dhcpbridge
--logdir=/var/log/nova
--state_path=/var/lib/nova
--lock_path=/var/lock/nova

#network
--network_manager=nova.network.manager.FlatDHCPManager
--public_interface=eth0
--flat_interface=eth0
--flat_network_bridge=br100
--fixed_range=192.168.246.224/27
--flat_network_dhcp_start=192.168.246.226
--flat_network_dns=8.8.4.4

--network_size=255
--flat_injected=true

--auth_strategy=keystone

#scheduler
--scheduler_driver=nova.scheduler.simple.SimpleScheduler

#nova database
--sql_connection=mysql://nova:notnova@192.168.245.64/nova

--force_dhcp_release
--iscsi_helper=tgtadm
--libvirt_use_virtio_for_bridges
--connection_type=libvirt
--root_helper=sudo nova-rootwrap
--ec2_private_dns_show_ip
--rpc_backend=cloudscaling.nova.rpc.impl_zmq

matchmaker_ring.json
{
        "nova": [ "obelix64" ],
        "nested": [ "obelix64" ],
        "scheduler": [ "obelix64" ],
        "network": [ "obelix64" ],
        "cert": [ "obelix64" ],
        "vsa": [ "obelix64" ],
        "compute": [ "obelix64" ],
        "volume": [ "obelix64" ],
        "zmq_replies":[ "obelix64" ],
        "test":[ "obelix64" ],
        "console":[ "obelix64" ],
        "consoleauth":[ "obelix64" ]
}

Note that everything deploy on single machine ip 192.168.245.64 (obelix64).

I can create instance with rabbitMQ with no problem.
Then I changed rpc_backend to zmq, and it start without error. Here is log file of nova-compute.log: http://pastebin.com/PsipwkPH

However, when I create instance it throw "Instance failed network setup" by the following log: http://pastebin.com/3F4u6Y9v

Here is the log from nova-network.log: http://pastebin.com/A301xnJa

When I execute nova list, it seem that instance is assigned the ip address(and scheduled to this compute), but the status is ERROR

# nova list
+--------------------------------------+--------+--------+-------------------------+
| ID | Name | Status | Networks |
+--------------------------------------+--------+--------+-------------------------+
| c0f37c59-eae2-464e-9063-69794c8b949b | natty1 | ERROR | novanet=192.168.246.226 |
+--------------------------------------+--------+--------+-------------------------+

Question information

Language:
English Edit question
Status:
Expired
For:
OpenStack Compute (nova) Edit question
Assignee:
No assignee Edit question
Last query:
2012-05-07
Last reply:
2012-05-23
Russell Bryant (russellb) said : #1

I subscribed the author of the zeromq driver. Hopefully he can help you out.

Eric Windisch (ewindisch) said : #2

It would be useful to see your nova-network.log. Can you provide this?

Nova-network is either not running on 'obelix64' or has erred in sending a response. Message replies are sent to FLAGS.host (--host), which defaults to the system's 'hostname' in Essex.

Additionally, I will update the documentation, but late in development, I made it so that IP addresses will now work. This reduces the complexity of needing working name resolution. Starting with IPs may be easier than relying on host resolution.

--ip-based configuration example--

nova.conf:
# set to real, addressable IP for this machine
--host=192.168.1.2

# Note that you really should only *need* scheduler and network below (in Essex)
# network is optional, depending on the network manager.
matchmaker_ring.json:
{
        "nested": [ "192.168.1.2" ],
        "scheduler": [ "192.168.1.2" ],
        "network": [ "192.168.1.2" ],
        "cert": [ "192.168.1.2" ],
        "compute": [ "192.168.1.2" ],
        "volume": [ "192.168.1.2" ],
        "console":[ "192.168.1.2" ],
        "consoleauth":[ "192.168.1.2" ]
}

Thanks Russel for forwarding it to Eric.

Hi Eric, I have pasted the link of both nova-compute and nova-network logs.
For the sake of ease of reading, I am highlighting the exact portions of the logs:

1.) nova-compute log file: (http://pastebin.com/3F4u6Y9v) -- at line-28 (2012-05-04 19:40:09) log reports that nova-compute makes a _multi_send() call which eventually timesout at (2012-05-04 19:40:29) , and reports an error of "Instance failed network setup"

2.) nova-network log file ( http://pastebin.com/A301xnJa) -- during the timeout period nova-network log reports only two entries which are present on lines 42 & 43

we have updated nova.conf and added a FLAG
--host=obelix64
(We also tried with IP-addresses but each time we get the same error).

thanks
Kobe

Launchpad Janitor (janitor) said : #4

This question was expired because it remained in the 'Open' state without activity for the last 15 days.

Eric Windisch (ewindisch) said : #5

Are you running the bin/nova-zmq-rpc-receiver daemon? This is necessary to receive replies. Looking at the logs, messaging was happening, but the reply was lost.