libvirrt error: Connection reset by peer

Asked by Max Schilling

I'm using a two node setup with Ubuntu 12.04, one cloud controller and one compute node. I'm using kvm and whenever I try to start an instance the compute node throws a libvirrt error:

2012-05-24 08:47:30 TRACE nova.compute.manager [instance: b0d33c8a-9bf6-4e39-b26c-5f9b382f2766] Traceback (most recent call last):
2012-05-24 08:47:30 TRACE nova.compute.manager [instance: b0d33c8a-9bf6-4e39-b26c-5f9b382f2766] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 592, in _spawn
2012-05-24 08:47:30 TRACE nova.compute.manager [instance: b0d33c8a-9bf6-4e39-b26c-5f9b382f2766] self._legacy_nw_info(network_info), block_device_info)
2012-05-24 08:47:30 TRACE nova.compute.manager [instance: b0d33c8a-9bf6-4e39-b26c-5f9b382f2766] File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 114, in wrapped
2012-05-24 08:47:30 TRACE nova.compute.manager [instance: b0d33c8a-9bf6-4e39-b26c-5f9b382f2766] return f(*args, **kw)
2012-05-24 08:47:30 TRACE nova.compute.manager [instance: b0d33c8a-9bf6-4e39-b26c-5f9b382f2766] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py", line 922, in spawn
2012-05-24 08:47:30 TRACE nova.compute.manager [instance: b0d33c8a-9bf6-4e39-b26c-5f9b382f2766] self._create_new_domain(xml)
2012-05-24 08:47:30 TRACE nova.compute.manager [instance: b0d33c8a-9bf6-4e39-b26c-5f9b382f2766] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py", line 1575, in _create_new_domain
2012-05-24 08:47:30 TRACE nova.compute.manager [instance: b0d33c8a-9bf6-4e39-b26c-5f9b382f2766] domain.createWithFlags(launch_flags)
2012-05-24 08:47:30 TRACE nova.compute.manager [instance: b0d33c8a-9bf6-4e39-b26c-5f9b382f2766] File "/usr/lib/python2.7/dist-packages/libvirt.py", line 581, in createWithFlags
2012-05-24 08:47:30 TRACE nova.compute.manager [instance: b0d33c8a-9bf6-4e39-b26c-5f9b382f2766] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2012-05-24 08:47:30 TRACE nova.compute.manager [instance: b0d33c8a-9bf6-4e39-b26c-5f9b382f2766] libvirtError: Unable to read from monitor: Connection reset by peer

I checked the bios, I checked "lsmod | grep kvm" and "lsmod | grep nbd". KVM is enabled in the bios and the modules are loaded.

When I try using "virsh start domain-name", I see the same error: "error: Unable to read from monitor: Connection reset by peer" . Any hints on what's wrong here?

Question information

Language:
English Edit question
Status:
Solved
For:
OpenStack Compute (nova) Edit question
Assignee:
No assignee Edit question
Solved by:
Igor Laskovy
Solved:
Last query:
Last reply:
Revision history for this message
Adrian Moya (adrianmoya) said :
#1

I'm hitting this error too. Same case, ubuntu 12.04, followed CSSCorp guide with some changes. The error is the same.

Revision history for this message
Igor Laskovy (igor-laskovy) said :
#2

Yep! Have the same trouble, but in VM under ESXi5 like nested hypervisor.

Revision history for this message
Best Igor Laskovy (igor-laskovy) said :
#3

Well, found workaround.
Since I don't need to use horizont I have del following from nova.conf:
--novncproxy_base_url=http://<ip_of_your_controller>:6080/vnc_auto.html
--vncserver_proxyclient_address=<ip_of_your_controller>
--vncserver_listen=<ip_of_your_controller>

Revision history for this message
Max Schilling (mx-chilly) said :
#4

I tried that workaround and I updated the ubuntu on the host for the 100st time (sure feels like it) and now the error is gone. For me this doesn't make sense. Why should vnc settings have an influence on libvirt? And these were settings I copied from a working setup...

I also tried to reproduce the error by adding those lines again and restarting nova-compute. Well, it now works even with vns settings. But I would still love to hear an explanation of what the error actually was...

Revision history for this message
Igor Laskovy (igor-laskovy) said :
#5

Well, I have reread documentation and found in the http://docs.openstack.org/trunk/openstack-compute/install/content/ch_install-dashboard.html the following:

Q: My vnc proxy worked fine during my All-In-One test, but now it doesn't work on multi host. Why?
A: The default options work for an All-In-One install, but changes must be made on your compute hosts once you start to build a cluster. As an example, suppose you have two servers:
PROXYSERVER (public_ip=172.24.1.1, management_ip=192.168.1.1)
COMPUTESERVER (management_ip=192.168.1.2)
# These flags help construct a connection data structure
vncserver_proxyclient_address=192.168.1.2
novncproxy_base_url=http://172.24.1.1:6080/vnc_auto.html
xvpvncproxy_base_url=http://172.24.1.1:6081/console
# This is the address where the underlying vncserver (not the proxy)
# will listen for connections.
vncserver_listen=192.168.1.2

So, looks like we need to change vncserver_proxyclient_address and vncserver_listen to local address on each compute nodes accordingly.

This is work for me too!

Revision history for this message
Adrian Moya (adrianmoya) said :
#6

Igor, thanks, that worked perfectly!

Revision history for this message
Igor Laskovy (igor-laskovy) said :
#7

You are welcome))

Revision history for this message
Max Schilling (mx-chilly) said :
#8

Thanks Igor Laskovy, that solved my question.

Revision history for this message
Igor Laskovy (igor-laskovy) said :
#9

You are welcome, Max))

Revision history for this message
Dirk Grunwald (grunwald) said :
#10

I'm having a similar problem to this -- I've gone through the process of checking the parameters as indicated above, and they all match the intended configuration.

In general, I'm having a hard time figuring out how to debug then nova setup -- I have debug output enabled, but the exception handler appears to be fairly generic, and I don't know how to get figure out what's causing

2012-07-23 16:21:46 TRACE nova File "/usr/lib/python2.7/dist-packages/libvirt.py", line 551, in create
2012-07-23 16:21:46 TRACE nova if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
2012-07-23 16:21:46 TRACE nova libvirtError: Unable to read from monitor: Connection reset by peer

How did you figure it out in this instance?

Revision history for this message
Max Schilling (mx-chilly) said :
#11

I'm also no expert in debugging nova and had quite some troubles with it. I knew that libvirt was used, so I located the instances on the compute node and tried to start them manually using virsh (https://help.ubuntu.com/community/KVM/Virsh). Didn't work, but so I could be sure it's a problem with libvirt and the xml file that defines the instance.

Well it turned out that there was an entry in the xml file for the vnc server which was not correct. After it was removed, I could start the machines.