Comment 66 for bug 1787405

Used the Kernel from Proposed:
apt-cache policy linux-image-4.15.0-42-generic
linux-image-4.15.0-42-generic:
  Installed: 4.15.0-42.45
  Candidate: 4.15.0-42.45

Libvirt/Qemu from PPA [1]

Having one device assigned to my LPAR atm:
$ ll /sys/bus/ap/devices/
total 0
drwxr-xr-x 2 root root 0 Nov 23 03:29 ./
drwxr-xr-x 4 root root 0 Nov 23 03:29 ../
lrwxrwxrwx 1 root root 0 Nov 23 03:29 00.0016 -> ../../../devices/ap/card00/00.0016/
lrwxrwxrwx 1 root root 0 Nov 23 03:29 card00 -> ../../../devices/ap/card00/

# mask out the adapters/queues of your choice that you want to virtualize
# In my case i have card 0 queue 16 (hex 16 dec 22 to match HMC config)
$ lszcrypt
CARD.DOMAIN TYPE MODE STATUS REQUEST_CNT
-------------------------------------------------
00 CEX5C CCA-Coproc online 5
00.0016 CEX5C CCA-Coproc online 5
# so lets assign that to vfio-ap instead of zcrypt use

# Adapter
$ cat /sys/bus/ap/apmask
0xffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
$ echo -0x0 | sudo tee /sys/bus/ap/apmask
$ cat /sys/bus/ap/apmask
0x7fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
# Domain
$ cat /sys/bus/ap/aqmask
0xffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
$ echo -0x16 | sudo tee /sys/bus/ap/aqmask
$ cat /sys/bus/ap/aqmask
0xfffffdffffffffffffffffffffffffffffffffffffffffffffffffffffffffff

$ sudo modprobe vfio_ap
$ dmesg | tail
...
[272006.492864] vfio_ap matrix: MDEV: Registered

# create a new MDEV
$ uuid=$(uuidgen)
$ echo ${uuid} | sudo tee /sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough/create

in Dmesg:
[272197.818811] iommu: Adding device 24f952b3-03d1-4df2-9967-0d5f7d63d5f2 to group 0
[272197.818815] vfio_mdev 24f952b3-03d1-4df2-9967-0d5f7d63d5f2: MDEV: group_id = 0

# Assign adapter 0 to vfio-ap
echo +0x0 > /sys/devices/vfio_ap/matrix/${uuid}/assign_adapter
# Assign domain 16 (22) to vfio-ap
$ echo +0x16 | sudo tee /sys/devices/vfio_ap/matrix/${uuid}/assign_domain
$ echo +0x16 | sudo tee /sys/devices/vfio_ap/matrix/${uuid}/assign_control_domain

Check the matrix you have set up
$ cat /sys/devices/vfio_ap/matrix/${uuid}/matrix
00.0016

Get something to bootable to then start it with the MDEV assigned:
$ uvt-kvm create --memory=1024 --password=ubuntu bionic-vfio-ap arch=s390x label=daily release=bionic
# wait until initialized and shut it down
$ virsh shutdown bionic-vfio-ap

# Modify to also use the MDEV
$ virsh edit bionic-vfio-ap
# add a snippet matching your UUID like:
<hostdev mode='subsystem' type='mdev' model='vfio-ap'>
  <source>
    <address uuid='24f952b3-03d1-4df2-9967-0d5f7d63d5f2'/>
  </source>
</hostdev>

When restarting the guest this correctly adds the commandline argument:
  -device vfio-ap,id=hostdev0,sysfsdev=/sys/bus/mdev/devices/24f952b3-03d1-4df2-9967-0d5f7d63d5f2
We also see virt-aa helper generting vfio rules
$ grep '/dev/vfio' /etc/apparmor.d/libvirt/$(virsh dominfo bionic-vfio-ap | awk '/^Security label/ {print $3}').files
  "/dev/vfio/vfio" rw,
  "/dev/vfio/[0-9]*" rw,

And most importantly in the guest the adapter is present:
$ lszcrypt
CARD.DOMAIN TYPE MODE STATUS REQUEST_CNT
-------------------------------------------------
00 CEX5C CCA-Coproc online 1
00.0016 CEX5C CCA-Coproc online 1

Thanks Halil that I was watching [2] in Edinburgh :-)
And thanks cborntra for the WIP script to set those up.

[1]: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3520
[2]: https://events.linuxfoundation.org/wp-content/uploads/2017/12/vfio-ap-The-Perils-of-the-Weird-Halil-Pasic-IBM.pdf