libvirtError: internal error process exited while connecting to monitor: char device redirected to /dev/pts/13

Asked by qirui

2011-11-25 15:23:15,509 DEBUG nova.virt.libvirt_conn [-] ---------domain is <libvirt.virDomain instance at 0x2db1098> from (pid=18750) _create_new_domain /opt/nova_d/nova/virt/libvirt/connection.py:1213
2011-11-25 15:23:19,136 ERROR nova.exception [-] Uncaught exception
(nova.exception): TRACE: Traceback (most recent call last):
(nova.exception): TRACE: File "/opt/nova_d/nova/exception.py", line 98, in wrapped
(nova.exception): TRACE: return f(*args, **kw)
(nova.exception): TRACE: File "/opt/nova_d/nova/virt/libvirt/connection.py", line 603, in spawn
(nova.exception): TRACE: domain = self._create_new_domain(xml)
(nova.exception): TRACE: File "/opt/nova_d/nova/virt/libvirt/connection.py", line 1214, in _create_new_domain
(nova.exception): TRACE: domain.createWithFlags(launch_flags)
(nova.exception): TRACE: File "/usr/lib64/python2.6/site-packages/libvirt.py", line 372, in createWithFlags
(nova.exception): TRACE: if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
(nova.exception): TRACE: libvirtError: internal error process exited while connecting to monitor: char device redirected to /dev/pts/13
(nova.exception): TRACE: qemu-kvm: -netdev tap,ifname=tapdff80acf-61,script=,id=hostnet0: could not open /dev/net/tun: Permission denied
(nova.exception): TRACE: qemu-kvm: -netdev tap,ifname=tapdff80acf-61,script=,id=hostnet0: Device 'tap' could not be initialized
(nova.exception): TRACE:
(nova.exception): TRACE:
2011-11-25 15:23:19,137 ERROR nova.compute.manager [-] Instance '47' failed to spawn. Is virtualization enabled in the BIOS? Details: internal error process exited while connecting to monitor: char device redirected to /dev/pts/13
qemu-kvm: -netdev tap,ifname=tapdff80acf-61,script=,id=hostnet0: could not open /dev/net/tun: Permission denied
qemu-kvm: -netdev tap,ifname=tapdff80acf-61,script=,id=hostnet0: Device 'tap' could not be initialized
(nova.compute.manager): TRACE: Traceback (most recent call last):
(nova.compute.manager): TRACE: File "/opt/nova_d/nova/compute/manager.py", line 424, in _run_instance
(nova.compute.manager): TRACE: network_info, block_device_info)
(nova.compute.manager): TRACE: File "/opt/nova_d/nova/exception.py", line 129, in wrapped
(nova.compute.manager): TRACE: raise Error(str(e))
(nova.compute.manager): TRACE: Error: internal error process exited while connecting to monitor: char device redirected to /dev/pts/13
(nova.compute.manager): TRACE: qemu-kvm: -netdev tap,ifname=tapdff80acf-61,script=,id=hostnet0: could not open /dev/net/tun: Permission denied
(nova.compute.manager): TRACE: qemu-kvm: -netdev tap,ifname=tapdff80acf-61,script=,id=hostnet0: Device 'tap' could not be initialized
(nova.compute.manager): TRACE:

Question information

Language:
English Edit question
Status:
Solved
For:
neutron Edit question
Assignee:
No assignee Edit question
Solved by:
qirui
Solved:
Last query:
Last reply:

This question was reopened

Revision history for this message
dan wendlandt (danwent) said :
#1

hi.

can you post more information about your platform and how you are running nova (particularly the flags file?)

I assume you're using using the LibvirtOpenVswitchDriver?

Revision history for this message
qirui (amy-qi) said :
#2

hi.

    You are right. I deploy a single node with diablo and adopt quantum to achieve virtual network service.And quantum need openvswitch support,so I install openvswitch. When I create instance,the nova-compute.log show the information that mentioned above.
   Please help me,thank you!

Revision history for this message
dan wendlandt (danwent) said :
#3

are you running ubuntu, centos/RHEL, something else?

Revision history for this message
qirui (amy-qi) said :
#4

Sorry,I forget telling you that I running SUSE 11 SP2

Revision history for this message
dan wendlandt (danwent) said :
#5

I've never run openstack on suse, but my guess is that are are running into an issue that is similar to something I have seen on RHEL.

The current open vswitch vif-plugging mechanism creates a tap device for each VM NIC, then has libvirt use that tap using an <interface type="ethernet" device="tapX">

This works fine on Ubuntu, but some distros have things locked down a bit more, which seems to prevent libvirt from using these tap devices.

I've seen some success working around this issue on RHEL by doing some combination of the following changes to "/etc/libvirt/qemu.conf" and then restarting libvirt:

Uncomment the line:

cgroup_controllers = [ "cpu", "devices", "memory" ]

Uncomment the following lines and add the reference to "/dev/net/tun":
cgroup_device_acl = [
    "/dev/null", "/dev/full", "/dev/zero",
    "/dev/random", "/dev/urandom",
    "/dev/ptmx", "/dev/kvm", "/dev/kqemu",
    "/dev/rtc", "/dev/hpet", "/dev/net/tun"
]

uncomment and set
clear_emulator_capabilities=0

Also change the user to run as root
user = "root"
group = "root"

That said, if you're thinking about using this in production, you will have to put some time into exploring whether these changes are something you are comfortable with, as I believe the implication is that a malicious user that finds a way to break out of the KVM isolation would have root on your box, rather than just the permissions of the libvirt user.

It may also be the case that some of these problems go away if we instead set the permissions on the tap device to correspond to the libvirt user after creating it... I'm not really sure.

If you have any luck exploring this or have suggestions on how we can change the vif-plugging to work better on SUSE, let me know.

Revision history for this message
Ryu Ishimoto (ryu-midokura) said :
#6

hi,

I have no experience in libvirt + SUSE so I may not be much of help here, but I just wanted to add that one possibility might be an issue with apparmor's libvirt profile setting. If apparmor is enabled, it might be worth a look. When I tweaked around with libvirt on Ubuntu, and this type of permission error came up, it was often solved by configuring its apparmor settings(although in my case, they were not networking related). Sorry that I can't be much of assistance here. I'm also very curious to see how vif-plugging runs on non-Ubuntu systems.

Revision history for this message
qirui (amy-qi) said :
#7

The problem still exist.I modify the qemu.conf,please help me

Revision history for this message
dan wendlandt (danwent) said :
#8

Hi Qirui,

The two platforms that OpenStack in general and Quantum in particular are tested on are Ubuntu and RHEL. I know SUSE recently announced that they are joining OpenStack, so perhaps contacting them directly about this problem could help. We'd be happy to work with them on this.

Its worth noting that I suspect this issue can be reproduced just with libvirt itself, without openstack, so you might also try reproducing the problem with libvirt alone when asking the SUSE folks for help. Particularly, you need to be able to:

Create a tap device using (must be root):

ip tuntap add tap-001 mode tap

This will create a device tap-001.

Then start a VM with a type=ethernet interface that uses that tap device:

        <interface type='ethernet'>
            <target dev='tap-001' />
            <mac address='ca:fe:de:ad:be:ef' />
            <script path='' />
        </interface>

Revision history for this message
qirui (amy-qi) said :
#9

Hi
    I can run instance.I install OVS bridge compatibility mode by following INSTALL.bridge and create a fake bridge for each VLAN you want available.I use these command:

ovs-brcompatd --pidfile --detach
     ovs-vsctl add-br br101 br0 101

Revision history for this message
qirui (amy-qi) said :
#10

   Thank you for your help. I think my problem is that tap can not communicate with openvswitch. But your proposal is also very useful.
   I have other problem,please help me. I use command "euca-describe-availability-zones verbose"
the result is:

   AVAILABILITYZONE nova available
AVAILABILITYZONE |- linux-cma03
AVAILABILITYZONE | |- nova-volume enabled XXX 2011-11-29 02:28:30
AVAILABILITYZONE | |- nova-network enabled XXX 2011-11-29 02:28:30
AVAILABILITYZONE | |- nova-scheduler enabled XXX 2011-11-29 02:28:30
AVAILABILITYZONE | |- nova-compute enabled XXX 2011-11-29 02:28:29
AVAILABILITYZONE |- linux_cma03
AVAILABILITYZONE | |- nova-volume enabled :-) 2011-11-29 07:59:59
AVAILABILITYZONE | |- nova-network enabled :-) 2011-11-29 07:59:59
AVAILABILITYZONE | |- nova-scheduler enabled :-) 2011-11-29 07:59:59
AVAILABILITYZONE | |- nova-compute enabled :-) 2011-11-29 07:59:59

Revision history for this message
qirui (amy-qi) said :
#11

I delete the table "services" of nova database,this problem solved.

Revision history for this message
qirui (amy-qi) said :
#12

I delete information about 2011-11-29 02:28:29 of "services",and implement
command "euca-describe-availability-zones verbose":
AVAILABILITYZONE nova available
AVAILABILITYZONE |- linux_cma03
AVAILABILITYZONE | |- nova-volume enabled :-) 2011-11-30 08:42:56
AVAILABILITYZONE | |- nova-network enabled :-) 2011-11-30 08:42:56
AVAILABILITYZONE | |- nova-scheduler enabled :-) 2011-11-30 08:42:56
AVAILABILITYZONE | |- nova-compute enabled :-) 2011-11-30 08:42:56