udisks2 fails to install in a container with systemd 243

Hi, I'm the santa_ above, so, to explain the full picture I'm going to write 3 separate messages below trying to answer these 3 questions:

1. How to reproduce the issue?
2. Why this is happening?
3. How we could fix it?

1. How to reproduce the issue?

Very easy, you just have to create an eoan LXD container, add the ppa with systemd 243:
https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3801/+packages

and then just do "apt dist-upgrade" to upgrate to systemd 243 and "apt install udisks2", the udisks2 installation will fail like this (obviously the devices in question could be different in your case, but the error will be similar):

Failed to write 'change' to '/sys/devices/pci0000:00/0000:00:1f.2/ata1/host0/target0:0:0/0:0:0:0/block/sda/uevent': Permission denied
Failed to write 'change' to '/sys/devices/pci0000:00/0000:00:1f.2/ata1/host0/target0:0:0/0:0:0:0/block/sda/sda1/uevent': Permission denied
Failed to write 'change' to '/sys/devices/pci0000:00/0000:00:1f.2/ata2/host1/target1:0:0/1:0:0:0/block/sdb/uevent': Permission denied
Failed to write 'change' to '/sys/devices/pci0000:00/0000:00:1f.2/ata2/host1/target1:0:0/1:0:0:0/block/sdb/sdb1/uevent': Permission denied
Failed to write 'change' to '/sys/devices/pci0000:00/0000:00:1f.2/ata2/host1/target1:0:0/1:0:0:0/block/sdb/sdb2/uevent': Permission denied
Failed to write 'change' to '/sys/devices/pci0000:00/0000:00:1f.2/ata2/host1/target1:0:0/1:0:0:0/block/sdb/sdb3/uevent': Permission denied
Failed to write 'change' to '/sys/devices/pci0000:00/0000:00:1f.2/ata2/host1/target1:0:0/1:0:0:0/block/sdb/sdb4/uevent': Permission denied
Failed to write 'change' to '/sys/devices/pci0000:00/0000:00:1f.2/ata2/host1/target1:0:0/1:0:0:0/block/sdb/sdb5/uevent': Permission denied
Failed to write 'change' to '/sys/devices/virtual/block/loop0/uevent': Permission denied
Failed to write 'change' to '/sys/devices/virtual/block/loop1/uevent': Permission denied
Failed to write 'change' to '/sys/devices/virtual/block/loop2/uevent': Permission denied
Failed to write 'change' to '/sys/devices/virtual/block/loop3/uevent': Permission denied
Failed to write 'change' to '/sys/devices/virtual/block/loop4/uevent': Permission denied
Failed to write 'change' to '/sys/devices/virtual/block/loop5/uevent': Permission denied
Failed to write 'change' to '/sys/devices/virtual/block/loop6/uevent': Permission denied
Failed to write 'change' to '/sys/devices/virtual/block/loop7/uevent': Permission denied
Failed to write 'change' to '/sys/devices/virtual/block/dm-0/uevent': Permission denied
dpkg: error processing package udisks2 (--configure):
 installed udisks2 package post-installation script subprocess returned error exit status 1
Setting up libblockdev-crypto2:amd64 (2.22-1) ...
Processing triggers for man-db (2.8.7-3) ...
Processing triggers for dbus (1.12.14-1ubuntu2) ...
Processing triggers for libc-bin (2.30-0ubuntu2) ...
Errors were encountered while processing:
 udisks2
E: Sub-process /usr/bin/dpkg returned an error code (1)

2. Why this is happening?

First of all, the udisks2 installation fails because the postinst script fails. Inspecting the postinst script of udisks2, we can see these lines:

if [ "$1" = "configure" ]; then
    # we ship udev rules, so trigger an update
    udevadm trigger --subsystem-match=block --action=change
fi

That udevadm command fails, i.e. ends with non-zero status with systemd >= 243. This new different behaviour of udevadm makes the postinst script fail, and that makes the package installation fail.

While I didn't git bisect the thing, reading the systemd git history makes me believe that the specific commit which produced this new behaviour could be this one:
https://github.com/systemd/systemd/commit/0584b17a8c7d17649aef9f06a8aee304dc80eb7e

3. How we could fix it?

Well, given the data I posted above, my *guess* is that we could safely ignore the udevadm non-zero exit status in the udisks2 postinst script with something like this:

if [ "$1" = "configure" ]; then
    # we ship udev rules, so trigger an update
    udevadm trigger --subsystem-match=block --action=change || true
fi

Note the final "|| true". In case the conjecture I just made above is correct, I prepared a modified udisks2 package in a PPA which doesn't fail to install:
https://launchpad.net/~panfaust/+archive/ubuntu/udisks2

If the conjecture I made and/or proposed fix are wrong, I hope at least the data I provided leads to the correct solution.

Regards.

Status changed to 'Confirmed' because the bug affects multiple users.

Hey BTW, I suspect this 'error' is due to upstream commit 97afc0351a96e0daa83964df33937967c75c644f which changed udevadm's behavior; previously it silently ignored errors writing to /sys/devices/*/uevent files, but now it logs error and returns error code. So this is likely not *actually* a new failure, it's just now being reported as failure by udevadm.

There is a similar 'failure' in bug 1845319 where (only on ppc64el) it gets ENODEV when writing to /sys/devices/vio/uevent, but I checked older releases and the vio device has been returning ENODEV for a long time; udevadm was just ignoring that before now.

And yeah, ignoring the udevadm return value with ||: could be one way to avoid the error.

Oh, and a better fix would be not calling udevadm when 'systemd-detect-virt --private-users' is true. It's a bit annoying that every udevadm caller would need to know that, however.

On thinking about this, it does seem the behavior change of udevadm will be highly disruptive to existing callers of udevadm trigger, so I opened upstream bug to try to address it without needing all callers to ignore its return code.

@panfaust I believe this should be fixed now upstream by https://github.com/systemd/systemd/pull/13656

can you verify that is correct? And we shouldn't need anything in Eoan since it's going to be based on 242, which doesn't have commit 97afc0351a96e0daa83964df33937967c75c644f that introduces this problem.

This bug was fixed in the package systemd - 243-2ubuntu1

---------------
systemd (243-2ubuntu1) focal; urgency=medium

  * Merge to Ubuntu from experimental
  * Refresh patches:
    - Dropped changes:
      * Cherrypick ask-password: prevent buffer overrow when reading from keyring.
        File: debian/patches/ask-password-prevent-buffer-overrow-when-reading-fro.patch
        https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=6d6e9cbd4fc6e018031a4762e88f2c3aa19e24e8
      * random-util: eat up bad RDRAND values seen on AMD CPUs.
        File: debian/patches/+rdrand-workaround-on-amd.patch
        https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?h=ubuntu-eoan&id=6ab88231efca4b04b26de6cfb5d671be154aabe0
    - Remaining changes:
      * Recommend networkd-dispatcher
        File: debian/control
        https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=d1e3b2c7e4757119da0d550b0b3c0a6626a176dc
      * Enable EFI/bootctl on armhf.
        File: debian/control
        https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=043122f7d8a1487bfd357e815a6ece1ceea6e7d1
      * debian/control: strengthen dependencies.
        File: debian/control
        https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=d1ecf0c372f5212129c85ae60fddf26b2271a1fe
      * Add conflicts with upstart and systemd-shim
        File: debian/control
        https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=83ed7496afc7c27be026014d109855f7d0ad1176
      * Specify Ubuntu's Vcs-Git
        File: debian/control
        https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=fd832930ef280c9a4a9dda2440d5a46a6fdb6232
      * Ubuntu/extra: ship dhclient-enter hook.
        Files:
        - debian/extra/dhclient-enter-resolved-hook
        - debian/rules
        https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=f3398a213f80b02bf3db0c1ce9e22d69f6d56764
        https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=258893bae8cbb12670e4807636fe8f7e9fb5407a
        https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=0725c1169ddde4f41cacba7af3e546704e2206be
      * udev-udeb: ship modprobe.d snippet to force scsi_mod.scan=sync in d-i.
        Files:
        - debian/extra/modprobe.d-udeb/scsi-mod-scan-sync.conf
        - debian/udev-udeb.install
        https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=eb6d8a2b9504917abb7aa2c4035fdbb7b98227f7
      * debian/extra/start-udev: Set scsi_mod scan=sync even if it's builtin to the kernel (we previously only set it in modprobe.d)
        Files:
        - debian/extra/start-udev
        https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=6b72628f8de991e2c67ac4289fc74daf3abe7d14
      * debian/extra/units/systemd-resolved.service.d/resolvconf.conf:
        drop resolvconf.conf drop-in, resolved integration moved to resolvconf package.
      * debian/extra/wrap_cl.py: add changelog formatter
        Files:
        - debian/extra/wrap_cl.py
        - debian/gbp...