I have been following the guide to install openstack havana and I have encountered a problem where the profile loaded into apparmor prevents a vm from attaching a cinder volume. I have been able to mount the cinder iscsi target manualy from the controller with no issues. The issue seems to happen when libvirt is updating the profile for the vm. It adds "/dev/sdb" rw, to the bottom of the apparmor profile specific to the vm then tells apparmor to reload it. However when nova tries to attach the volume to the vm it is denied by apparmor.
My current openstack setup involves one controller node and one compute node, running ubuntu server 12.04 amd64 and ubuntu server 13.10 amd64 respectively.
---- Here is the syslog for the transaction:
[ 940.868707] scsi5 : iSCSI Initiator over TCP/IP
[ 941.375404] scsi 5:0:0:0: RAID IET Controller 0001 PQ: 0 ANSI: 5
[ 941.377049] scsi 5:0:0:0: Attached scsi generic sg3 type 12
[ 941.378866] scsi 5:0:0:1: Direct-Access IET VIRTUAL-DISK 0001 PQ: 0 ANSI: 5
[ 941.379830] sd 5:0:0:1: Attached scsi generic sg4 type 0
[ 941.380250] sd 5:0:0:1: [sdb] 2097152 512-byte logical blocks: (1.07 GB/1.00 GiB)
[ 941.380863] sd 5:0:0:1: [sdb] Write Protect is off
[ 941.380868] sd 5:0:0:1: [sdb] Mode Sense: 49 00 00 08
[ 941.381228] sd 5:0:0:1: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 941.384558] sdb: unknown partition table
[ 941.386602] sd 5:0:0:1: [sdb] Attached SCSI disk
Connection3:0 to [target: iqn.2010-10.org.openstack:volume-1275fe44-3014-4cee-a712-5a5acec4a310, portal: 10.0.1.1,3260] through [iface: default] is operational now
[ 941.864524] type=1400 audit(1388357102.732:49): apparmor="STATUS" operation="profile_replace" parent=3651 profile="unconfined" name="libvirt-e8041077-5488-4d8d-850e-e041463367c8" pid=3652 comm="apparmor_parser"
[ 941.868422] type=1400 audit(1388357102.736:50): apparmor="DENIED" operation="open" parent=1 profile="libvirt-e8041077-5488-4d8d-850e-e041463367c8" name="/dev/sdb" pid=3258 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=109 ouid=109
[ 941.868432] type=1400 audit(1388357102.736:51): apparmor="DENIED" operation="open" parent=1 profile="libvirt-e8041077-5488-4d8d-850e-e041463367c8" name="/dev/sdb" pid=3258 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=109 ouid=109
[ 941.868460] type=1400 audit(1388357102.736:52): apparmor="DENIED" operation="open" parent=1 profile="libvirt-e8041077-5488-4d8d-850e-e041463367c8" name="/dev/sdb" pid=3258 comm="qemu-system-x86" requested_mask="rw" denied_mask="rw" fsuid=109 ouid=109
[ 941.868761] type=1400 audit(1388357102.736:53): apparmor="DENIED" operation="capable" parent=1 profile="/usr/sbin/libvirtd" pid=1595 comm="libvirtd" pid=1595 comm="libvirtd" capability=29 capname="audit_write"
[ 942.292018] type=1400 audit(1388357103.160:54): apparmor="STATUS" operation="profile_replace" parent=3653 profile="unconfined" name="libvirt-e8041077-5488-4d8d-850e-e041463367c8" pid=3654 comm="apparmor_parser"
[ 942.395211] sd 5:0:0:1: [sdb] Synchronizing SCSI cache
[ 942.646286] connection3:0: detected conn error (1020)
iscsid: Connection3:0 to [target: iqn.2010-10.org.openstack:volume-1275fe44-3014-4cee-a712-5a5acec4a310, portal: 10.0.1.1,3260] through [iface: default] is shutdown.
---- And here is the nova-compute log:
nova.compute.manager [req-d04d51f7-6df1-4b16-92ba-97010a35fe08 b4b10173f6de4dbbae0ef8a1d2e42425 744cfaad88c849ee8fae24c38f896327] [instance: e8041077-5488-4d8d-850e-e041463367c8] Attaching volume 1275fe44-3014-4cee-a712-5a5acec4a310 to /dev/vdc
nova.virt.libvirt.utils [req-d04d51f7-6df1-4b16-92ba-97010a35fe08 b4b10173f6de4dbbae0ef8a1d2e42425 744cfaad88c849ee8fae24c38f896327] systool is not installed
nova.virt.libvirt.utils [req-d04d51f7-6df1-4b16-92ba-97010a35fe08 b4b10173f6de4dbbae0ef8a1d2e42425 744cfaad88c849ee8fae24c38f896327] systool is not installed
nova.compute.manager [req-d04d51f7-6df1-4b16-92ba-97010a35fe08 b4b10173f6de4dbbae0ef8a1d2e42425 744cfaad88c849ee8fae24c38f896327] [instance: e8041077-5488-4d8d-850e-e041463367c8] Failed to attach volume 1275fe44-3014-4cee-a712-5a5acec4a310 at /dev/vdc
nova.compute.manager [instance: e8041077-5488-4d8d-850e-e041463367c8] Traceback (most recent call last):
nova.compute.manager [instance: e8041077-5488-4d8d-850e-e041463367c8] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3669, in _attach_volume
nova.compute.manager [instance: e8041077-5488-4d8d-850e-e041463367c8] encryption=encryption)
nova.compute.manager [instance: e8041077-5488-4d8d-850e-e041463367c8] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1100, in attach_volume
nova.compute.manager [instance: e8041077-5488-4d8d-850e-e041463367c8] raise exception.DeviceIsBusy(device=disk_dev)
nova.compute.manager [instance: e8041077-5488-4d8d-850e-e041463367c8] DeviceIsBusy: The supplied device (vdc) is busy.
nova.compute.manager [instance: e8041077-5488-4d8d-850e-e041463367c8]
nova.openstack.common.rpc.amqp [req-d04d51f7-6df1-4b16-92ba-97010a35fe08 b4b10173f6de4dbbae0ef8a1d2e42425 744cfaad88c849ee8fae24c38f896327] Exception during message handling
nova.openstack.common.rpc.amqp Traceback (most recent call last):
nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
nova.openstack.common.rpc.amqp **args)
nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
nova.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 90, in wrapped
nova.openstack.common.rpc.amqp payload)
nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 73, in wrapped
nova.openstack.common.rpc.amqp return f(self, context, *args, **kw)
nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 243, in decorated_function
nova.openstack.common.rpc.amqp pass
nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 229, in decorated_function
nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 271, in decorated_function
nova.openstack.common.rpc.amqp e, sys.exc_info())
nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 258, in decorated_function
nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3638, in attach_volume
nova.openstack.common.rpc.amqp context, instance, mountpoint)
nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3633, in attach_volume
nova.openstack.common.rpc.amqp mountpoint, instance)
nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3679, in _attach_volume
nova.openstack.common.rpc.amqp connector)
nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3669, in _attach_volume
nova.openstack.common.rpc.amqp encryption=encryption)
nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1100, in attach_volume
nova.openstack.common.rpc.amqp raise exception.DeviceIsBusy(device=disk_dev)
nova.openstack.common.rpc.amqp DeviceIsBusy: The supplied device (vdc) is busy.
---- And the libvirt-<vm-id>.files profile:
# DO NOT EDIT THIS FILE DIRECTLY. IT IS MANAGED BY LIBVIRT.
"/var/log/libvirt/**/instance-00000002.log" w,
"/var/lib/libvirt/**/instance-00000002.monitor" rw,
"/var/run/libvirt/**/instance-00000002.pid" rwk,
"/run/libvirt/**/instance-00000002.pid" rwk,
"/var/run/libvirt/**/*.tunnelmigrate.dest.instance-00000002" rw,
"/run/libvirt/**/*.tunnelmigrate.dest.instance-00000002" rw,
"/var/lib/nova/instances/e8041077-5488-4d8d-850e-e041463367c8/disk" rw,
"/var/lib/nova/instances/_base/c95d21495cba412aeb58b646c419e7b4ca1b910b" r,
# don't audit writes to readonly files
deny "/var/lib/nova/instances/_base/c95d21495cba412aeb58b646c419e7b4ca1b910b" w,
"/var/lib/nova/instances/e8041077-5488-4d8d-850e-e041463367c8/console.log" rw,
"/var/lib/nova/instances/e8041077-5488-4d8d-850e-e041463367c8/console.log" rw,
"/dev/sdb" rw,
I know it is possible to solve the problem by disabling apparmor for the libvirt but I was hoping to avoid that since that leaves the host un-protected from the guests. Why is apparmor deyning access to /dev/sdb for the vm when it's profile includes that device as rw. Any help would be greatly appreciated!
