AMQP ECONNREFUSED on Essex

Asked by Rashmi

Hi,

I am working on Essex on Ubuntu distr. When I try to run a VM instance it remains in "error" and "scheduling" state.
The nova-*.log indicate a [Errno 111] ECONNREFUSED.
I have verified the rabbitMQ info in the nova.conf file to be correct. RabbitMQ starts without errors as seen from /var/log/rabbitmq/ Logs. ANd I do not have any disk shortage as well

Below are the some logs:

************************************************************************************************************

rabbitmqctl status
Status of node rabbit@rashmi ...
[{pid,15212},
 {running_applications,[{rabbit,"RabbitMQ","2.7.1"},
                        {os_mon,"CPO CXC 138 46","2.2.7"},
                        {sasl,"SASL CXC 138 11","2.1.10"},
                        {mnesia,"MNESIA CXC 138 12","4.5"},
                        {stdlib,"ERTS CXC 138 10","1.17.5"},
                        {kernel,"ERTS CXC 138 10","2.14.5"}]},
 {os,{unix,linux}},
 {erlang_version,"Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:2:2] [rq:2] [async-threads:30] [kernel-poll:true]\n"},
 {memory,[{total,25319272},
          {processes,10409464},
          {processes_used,10403088},
          {system,14909808},
          {atom,1124441},
          {atom_used,1120218},
          {binary,77888},
          {code,11134417},
          {ets,796136}]},
 {vm_memory_high_watermark,0.39999999941084335},
 {vm_memory_limit,407361945}]
...done.

************************************************************************************************************

nova.network.log-->

2012-10-24 15:49:08 INFO nova.compute.manager [-] Updating host status
2012-10-24 15:49:30 ERROR nova.rpc.common [-] Failed to consume message from queue: Socket closed
2012-10-24 15:49:30 TRACE nova.rpc.common Traceback (most recent call last):
2012-10-24 15:49:30 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 490, in ensure
2012-10-24 15:49:30 TRACE nova.rpc.common return method(*args, **kwargs)
2012-10-24 15:49:30 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 567, in _consume
2012-10-24 15:49:30 TRACE nova.rpc.common return self.connection.drain_events(timeout=timeout)
2012-10-24 15:49:30 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 175, in drain_events
2012-10-24 15:49:30 TRACE nova.rpc.common return self.transport.drain_events(self.connection, **kwargs)
2012-10-24 15:49:30 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 238, in drain_events
2012-10-24 15:49:30 TRACE nova.rpc.common return connection.drain_events(**kwargs)
2012-10-24 15:49:30 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 57, in drain_events
2012-10-24 15:49:30 TRACE nova.rpc.common return self.wait_multi(self.channels.values(), timeout=timeout)
2012-10-24 15:49:30 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 63, in wait_multi
2012-10-24 15:49:30 TRACE nova.rpc.common chanmap.keys(), allowed_methods, timeout=timeout)
2012-10-24 15:49:30 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 120, in _wait_multiple
2012-10-24 15:49:30 TRACE nova.rpc.common channel, method_sig, args, content = read_timeout(timeout)
2012-10-24 15:49:30 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 88, in read_timeout
2012-10-24 15:49:30 TRACE nova.rpc.common return self.method_reader.read_method()
2012-10-24 15:49:30 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py", line 221, in read_method
2012-10-24 15:49:30 TRACE nova.rpc.common raise m
2012-10-24 15:49:30 TRACE nova.rpc.common IOError: Socket closed
2012-10-24 15:49:30 TRACE nova.rpc.common
2012-10-24 15:49:30 INFO nova.rpc.common [-] Reconnecting to AMQP server on 192.168.5.212:5672
2012-10-24 15:49:30 ERROR nova.rpc.common [-] AMQP server on 192.168.5.212:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
2012-10-24 15:49:30 TRACE nova.rpc.common Traceback (most recent call last):
2012-10-24 15:49:30 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 446, in reconnect
2012-10-24 15:49:30 TRACE nova.rpc.common self._connect()
2012-10-24 15:49:30 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 423, in _connect
2012-10-24 15:49:30 TRACE nova.rpc.common self.connection.connect()
2012-10-24 15:49:30 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 154, in connect
2012-10-24 15:49:30 TRACE nova.rpc.common return self.connection
2012-10-24 15:49:30 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 560, in connection
2012-10-24 15:49:30 TRACE nova.rpc.common self._connection = self._establish_connection()
2012-10-24 15:49:30 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 521, in _establish_connection
2012-10-24 15:49:30 TRACE nova.rpc.common conn = self.transport.establish_connection()
2012-10-24 15:49:30 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 255, in establish_connection
2012-10-24 15:49:30 TRACE nova.rpc.common connect_timeout=conninfo.connect_timeout)
2012-10-24 15:49:30 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 52, in __init__
2012-10-24 15:49:30 TRACE nova.rpc.common super(Connection, self).__init__(*args, **kwargs)
2012-10-24 15:49:30 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/connection.py", line 129, in __init__
2012-10-24 15:49:30 TRACE nova.rpc.common self.transport = create_transport(host, connect_timeout, ssl)
2012-10-24 15:49:30 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/transport.py", line 281, in create_transport
2012-10-24 15:49:30 TRACE nova.rpc.common return TCPTransport(host, connect_timeout)
2012-10-24 15:49:30 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/transport.py", line 85, in __init__
2012-10-24 15:49:30 TRACE nova.rpc.common raise socket.error, msg
2012-10-24 15:49:30 TRACE nova.rpc.common error: [Errno 111] ECONNREFUSED
2012-10-24 15:49:30 TRACE nova.rpc.common
2012-10-24 15:49:31 INFO nova.rpc.common [-] Reconnecting to AMQP server on 192.168.5.212:5672
2012-10-24 15:49:31 ERROR nova.rpc.common [-] AMQP server on 192.168.5.212:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 3 seconds.

Question information

Language:
English Edit question
Status:
Answered
For:
OpenStack Compute (nova) Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Yatin Kumbhare (yatinkumbhare-c) said :
#1

is port 5672 open? check with netstat, check firewall configuration or in case iptables service running, stop it.

Revision history for this message
Rashmi (rushmep3) said :
#2

Yes the port 5672 is open. I have disabled the Firewall already.. I still get the same error
Is there anything else that I could possibly be missing?
--------------------------------------------------------------------------------------------------------------------------------------
ufw status
Status: inactive
--------------------------------------------------------------------------------------------------------------------------------------
netstat -anp | grep 5672
tcp 0 0 192.168.5.212:46926 192.168.5.212:5672 ESTABLISHED 15415/python
tcp 0 0 192.168.5.212:46919 192.168.5.212:5672 ESTABLISHED 15512/python
tcp 0 0 192.168.5.212:46927 192.168.5.212:5672 ESTABLISHED 15415/python
tcp 0 0 192.168.5.212:46920 192.168.5.212:5672 ESTABLISHED 15505/python
tcp 0 0 192.168.5.212:46923 192.168.5.212:5672 ESTABLISHED 15493/python
tcp 0 0 192.168.5.212:46918 192.168.5.212:5672 ESTABLISHED 15480/python
tcp 0 0 192.168.5.212:46925 192.168.5.212:5672 ESTABLISHED 15403/python
tcp6 0 0 :::5672 :::* LISTEN 15212/beam.smp
tcp6 0 0 192.168.5.212:5672 192.168.5.212:46926 ESTABLISHED 15212/beam.smp
tcp6 0 0 192.168.5.212:5672 192.168.5.212:46927 ESTABLISHED 15212/beam.smp
tcp6 0 0 192.168.5.212:5672 192.168.5.212:46925 ESTABLISHED 15212/beam.smp
tcp6 0 0 192.168.5.212:5672 192.168.5.212:46920 ESTABLISHED 15212/beam.smp
tcp6 0 0 192.168.5.212:5672 192.168.5.212:46923 ESTABLISHED 15212/beam.smp
tcp6 0 0 192.168.5.212:5672 192.168.5.212:46918 ESTABLISHED 15212/beam.smp
tcp6 0 0 192.168.5.212:5672 192.168.5.212:46919 ESTABLISHED 15212/beam.smp

Can you help with this problem?

Provide an answer of your own, or ask Rashmi for more information if necessary.

To post a message you must log in.