[Impact]
An attempt to run cryptsetup open on a newly created LUKS partition on Ubuntu Core 20 causes a kernel crash. This happens in 100% of the attempts on the snapd Core 20 installation test, but on an image created to reproduce this bug it happens only when certain parameters are passed to cryptsetup. Both images are built similarly so the reason for this discrepancy is unknown. The kernel was installed from pc-kernel_374.snap.
[Test Case]
$ dir=$(mktemp -d /tmp/lp1860231.XXXXX)
$ dmsetup create lp1860231 --notable
$ mount -t ext4 \
"/dev/dm-$(dmsetup info -c -o minor --noheadings lp1860231)" "$dir"
Now check the logs for a backtrace.
[Regression Potential]
The currently proposed fix introduces no chance of stability regressions. There is a chance of a very small performance regression since an additional pointer comparison is performed on each block layer request but this is unlikely to be noticeable.
[Original Report]
Linux version 5.4.0-11-generic (buildd@lgw01-amd64-021) (gcc version 9.2.1 20200104 (Ubuntu 9.2.1-22ubuntu2)) #14-Ubuntu SMP Thu Jan 9 16:14:26 UTC 2020
Version signature: Ubuntu 5.4.0-11.14-generic 5.4.8
How to reproduce the crash in 3 "easy" steps:
1. Build a Core 20 image using the attached model file:
1.1. Install the ubuntu-image from latest/edge
$ sudo snap install --channel latest/edge ubuntu-image
1.2. Build the image
$ sudo ubuntu-image --image-size=4G ubuntu-core-20-amd64.model
2. Boot the image in kvm
2.1. Install ovmf version 0~20190606.20d2e5a1-2ubuntu1 or newer (the
stock ovmf from bionic may not work)
2.2. Boot the image
$ sudo kvm -snapshot -m 2048 -smp 4 \
-netdev user,id=mynet0,hostfwd=tcp::8022-:22,hostfwd=tcp::8090-:80 \
-device virtio-net-pci,netdev=mynet0 \
-drive file=pc.img,if=virtio \
-bios /usr/share/OVMF/OVMF_CODE.ms.fd
2.3. In the grub menu, edit the default option to include parameter
"systemd.debug-shell=1" in the kernel command line
2.4. Boot the kernel
3. Crash the kernel
3.1. When the system boots to the "Press enter to configure"
message, press ALT-F9 to enter the debug shell.
3.2. The system should have two partitions in /dev/vda. Create a
third one with fdisk.
3.3. Create a LUKS encrypted partition:
# echo 123|cryptsetup luksFormat -q --type luks2 --key-file - --pbkdf argon2i --iter-time 1 /dev/vda3
(the system will complain about a missing locking directory,
just ignore it.)
3.4. Open the encrypted device:
# echo 123|cryptsetup open --key-file - /dev/vda name
The Core 20 images contain the following udev rule which causes
the new block device to be mounted automatically. This mount is
what triggers the BUG:
ACTION=="add", SUBSYSTEM=="block", KERNEL!="loop*", KERNEL!="ram*" \
RUN+="/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/%k"
3.5. Read the crash message
The attached screenshots show these steps being executed.
A few notes:
- The backtrace seems very similar to the one reported in bug #1835279, however that problem was possibly caused by a race between partition creation and LUKS formatting. This time it doesn't seem to be the case, delays between commands don't help us here.
- In the test case above using large values of KDF iter-time may prevent the crash. I successfully opened the device in kernel 5.4.0-9 with --iter-time larger than 100, but 5.4.0-11 seems to require values closer to 1000. Regardless of the --iter-time value used, the crash always happen when running the test in a spread-driven automated environment (same kernel with image built in the same way, some other variable seems to be disturbing the system).
- All necessary modules are loaded before the LUKS partition creation (i.e. it doesn't seem to be caused by a race between dm-crypt loading and cryptsetup luksFormat for example).
Snap versions used to build the image:
pc-kernel_374.snap
pc_83.snap
snapd_6113.snap
core20_322.snap