Murano

Murano Agent log has IOError: Socket closed.

Asked by Ellen Batbouta

I suspect this is a configuration problem on my part but am hoping someone can help me figure out
what I need to do to fix this. I am running OpenStack Mikata. Thank you.

I have deployed my murano application but the deployment fails after my script has run to completion on the newly created instance. Here is what I see in the murano agent log:

2016-11-23 15:19:35.839 2224 WARNING muranoagent.app [-] Communication error
2016-11-23 15:19:35.839 2224 TRACE muranoagent.app Traceback (most recent call last):
2016-11-23 15:19:35.839 2224 TRACE muranoagent.app File "/usr/lib/python2.7/site-packages/muranoagent/app.py", line 137, in _wait_plan
2016-11-23 15:19:35.839 2224 TRACE muranoagent.app delay = 5
2016-11-23 15:19:35.839 2224 TRACE muranoagent.app File "/usr/lib/python2.7/site-packages/muranoagent/common/messaging/subscription.py", line 41, in __exit__
2016-11-23 15:19:35.839 2224 TRACE muranoagent.app self._consumer.cancel()
2016-11-23 15:19:35.839 2224 TRACE muranoagent.app File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 450, in cancel
2016-11-23 15:19:35.839 2224 TRACE muranoagent.app cancel(tag)
2016-11-23 15:19:35.839 2224 TRACE muranoagent.app File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 1632, in basic_cancel
2016-11-23 15:19:35.839 2224 TRACE muranoagent.app (60, 31), # Channel.basic_cancel_ok
2016-11-23 15:19:35.839 2224 TRACE muranoagent.app File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 67, in wait
2016-11-23 15:19:35.839 2224 TRACE muranoagent.app self.channel_id, allowed_methods)
2016-11-23 15:19:35.839 2224 TRACE muranoagent.app File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 237, in _wait_method
2016-11-23 15:19:35.839 2224 TRACE muranoagent.app self.method_reader.read_method()
2016-11-23 15:19:35.839 2224 TRACE muranoagent.app File "/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 189, in read_method
2016-11-23 15:19:35.839 2224 TRACE muranoagent.app raise m
2016-11-23 15:19:35.839 2224 TRACE muranoagent.app IOError: Socket closed

Here is the agent conf file on my instance:

[DEFAULT]
debug=True
verbose=True
log_file = /var/log/murano-agent.log

storage=/var/murano/plans

[rabbitmq]

# Input queue name
input_queue = ee0edaa268aed44b090c887f3f7e96290-h0bf13f54-f4b0-4031-aa02-da6e18c3c14f

# Output routing key (usually queue name)
result_routing_key = -execution-results-ekpjuivvd2gk21

# Connection parameters to RabbitMQ service

# Hostname or IP address where RabbitMQ is located.
host = 10.147.24.203

# RabbitMQ port (5672 is a default)
port = 5672

# Use SSL for RabbitMQ connections (True or False)
ssl = false

# Do not verify SSL certificates
insecure = false

# Path to SSL CA certificate or empty to allow self signed server certificate
ca_certs = '/etc/murano/certs/ca_certs'

# RabbitMQ credentials. Fresh RabbitMQ installation has "guest" account with "guest" password.
login = openstack
password = UGq8IcMJNnZrs9eKltePRjK9Jig59YgNNrOC0AyW

# RabbitMQ virtual host (vhost). Fresh RabbitMQ installation has "/" vhost preconfigured.
virtual_host = /

=======================================================

Also, in the murano_engine container, my murano.conf file has:

[oslo_messaging_notifications]
driver = messagingv2

Question information

Language:
English Edit question
Status:
Solved
For:
Murano Edit question
Assignee:
No assignee Edit question
Solved by:
Serg Melikyan
Solved:
Last query:
Last reply:
Revision history for this message
Best Serg Melikyan (smelikyan) said : 		#1

Any chance your script my interfere with connectivity to RabbitMQ? Looks like murano-agent getting scripts, executes and than looses connection to the RabbitMQ

Revision history for this message
Ellen Batbouta (ellen-batbouta) said : 		#2

Thank you for the quick reply. I don't believe the script is causing the problem. It mounts 2 volumes and installs
a database on one of the volumes. All local operations.

I'll trouble shoot this more after the Thanksgiving holiday. I thought I would post my question here in case the
solution was obvious to someone.

Revision history for this message
Stan Lagun (slagun) said : 		#3

IOError: Socket closed error in murano agent indicates that there were issues with RabbitMQ connectivity since RabbitMQ is the only thing that agent talks to through network sockets. Sometimes even local operation can cause it. For example you may have installed database which, as part of its installation, made changes to local firewall settings to expose its network port and restarted the firewall, which caused existing network connection to break.

However the fact that your deployment failed says that the problem was not fatal. Otherwise how would murano know that something went wrong on the agent if the later had no connectivity to RabbitMQ to signal it. So, unless you got a timeout error after 1 hour of deployment socket error is not what caused your deployment to fail.

There are 2 options that I can think of here:
1) Something went wrong on the MuranoPL side. In this case the explanation might be in the murano engine log file and/or the deployment logs that can be seen in murano dashboard
2) The script returned non-zero exit code even though it successfully installed the software. By default non-zero exit code indicates an error and causes deployment to fail. If the script cannot be fixed then there is an option to suppress this logic. Exit code of the script can be found in the agent's log file. On the engine side it causes AgentException to be thrown and it can be seen in the engine log file

Revision history for this message
Ellen Batbouta (ellen-batbouta) said : 		#4

Thanks Serg Melikyan, that solved my question.