40-vm-hotadd.rules attempts to set non-existent sysfs parameters

Note that same udevd rule exists as far back as 16.04 LTS:

ubuntu@xenial-test:~$ md5sum /lib/udev/rules.d/40-vm-hotadd.rules
d8569b53b5bd1874a86b743b82a2a5b4 /lib/udev/rules.d/40-vm-hotadd.rules

ubuntu@focal-test:~$ md5sum /usr/lib/udev/rules.d/40-vm-hotadd.rules
d8569b53b5bd1874a86b743b82a2a5b4 /usr/lib/udev/rules.d/40-vm-hotadd.rules

There is a previous bug, https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1518457 which addressed a different symptom but may be related.

> Apr 29 22:36:46 focal01 systemd-udevd[266]: memory7: /usr/lib/udev/rules.d/40-vm-hotadd.rules:9 Failed to write ATTR{/sys/devices/system/memory/memory7/state}, ignoring: Invalid argument

This isn't because the file doesn't exist, because the udev rule checks for its existence before trying to write to it. The error is because the kernel is rejecting the write to its state file for some reason. In a really quick test, it seems like this is because it's already online; the kernel default has been CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y since at least 4.15.

I think the error message is harmless, since the memory is already online, but the udev rule should be fixed to check if it's offline before actually trying to online it.

Can you test with the systemd from this ppa to see if the error message is gone?
https://launchpad.net/~ddstreet/+archive/ubuntu/lp1876018

Also please make sure all your memory and cpus are online after booting/hotadding.

Note that I might just remove this rule entirely, since it really shouldn't be needed anymore, now that the kernel default is to online hotplugged cpus/memory, but I want to verify that first.

@ddstreet, thanks, the fix works and I'll include it in the next upload to Groovy, in the next days.

This bug was fixed in the package systemd - 245.5-3ubuntu1

---------------
systemd (245.5-3ubuntu1) groovy; urgency=medium

  [ Dan Streetman ]
  * Hotadd only offline memory and CPUs (LP: #1876018)
    File: debian/extra/rules-ubuntu/40-vm-hotadd.rules
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=dc68569fb46b04791202a40326003799becae859

  [ Balint Reczey ]
  * Merge to Ubuntu from Debian unstable
  - Dropped changes:
    * Cherry-pick fix from upstream master to adjust UseGateway= default
  * journald: stream pid change newline fix (LP: #1875708)
    Files:
    - debian/patches/debian/fsckd-daemon-for-inter-fsckd-communication.patch
    - debian/patches/lp1875708/journald-Increase-stdout-buffer-size-sooner-when-almost-f.patch
    - debian/patches/lp1875708/journald-rework-end-of-line-marker-handling-to-use-a-fiel.patch
    - debian/patches/lp1875708/journald-rework-pid-change-handling.patch
    - debian/patches/lp1875708/journald-use-log_warning_errno-where-appropriate.patch
    - debian/patches/lp1875708/journald-use-the-fact-that-client_context_release-returns.patch
    - debian/patches/lp1875708/man-document-the-new-_LINE_BREAK-type.patch
    - debian/patches/lp1875708/socket-util-introduce-type-safe-dereferencing-wrapper-CMS.patch
    - debian/patches/lp1875708/test-Add-a-test-case-for-15654.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=95302fbe53202bdac33f86031aa56026444a92d3

  [ Dimitri John Ledkov ]
  * meson: initialize time-epoch to reproducible builds compatible value
    (LP: #1878969)
    File: debian/patches/meson-initialize-time-epoch-to-reproducible-builds-compat.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=6785dfd98f81dfe3e3d5273511ec33b196ddb0c3

systemd (245.5-3) unstable; urgency=medium

  [ Michael Biebl ]
  * Bump priority of libnss-systemd to standard
  * logind: avoid shadow lookups when doing userdb client side
  * Disable DNSSEC support by default in resolved.
    The upstream default, DNSSEC=allow-downgrade can lead to compatibility
    issues with certain network access points. Previously, DNSSEC support
    was only turned off when built for a stable Debian release, but it is
    safer and more consistent to just generally change the default to
    DNSSEC=no. (Closes: #959996)
  * Bump debhelper compatibility level to 13.
    Use the debhelper-compat package to declare the compatibility level and
    drop debian/compat.
  * Convert to dh_installsystemd and disable dh_installsystemduser
  * Drop custom initramfs update code.
    Now handled by dh_installinitramfs which is enabled by default in compat
    level 12 and above.

  [ Dan Streetman ]
  * Cherry-pick fix from upstream master to adjust UseGateway= default
    - network: change UseGateway= default to UseRoutes= setting
    - network: honor SetDNSRoutes= even if UseGateway=False
    (LP: #1867375)

  [ Topi Miettinen ]
  * Delete empty lines at end of file.
    Upstream commit hooks don't allow empty lines and of course they serve no
    purpose.

 -- Balint Reczey <email address hidden> Tue, 19 May 2020 19:43:50 +0200

@ddstreet Could you please include this fix in the next upload to Focal?

will do, thnx

Hello Jose, or anyone else affected,

Accepted systemd into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/245.4-4ubuntu3.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted systemd (245.4-4ubuntu3.2) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

apt/unknown (armhf)
indicator-session/17.3.20+19.10.20190921-0ubuntu1 (arm64)
dovecot/1:2.3.7.2-1ubuntu3.1 (armhf)
postgresql-12/unknown (armhf)
mir/unknown (armhf)
systemd/245.4-4ubuntu3.2 (amd64)
umockdev/unknown (armhf)
policykit-1/unknown (armhf)
asterisk/unknown (armhf)
anbox/unknown (armhf)
php7.4/unknown (armhf)
ksystemlog/unknown (armhf)
polkit-qt-1/unknown (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

focal:

with the reproducer hack from the description (comment out the 'goto' line in the udev rule), and the system rebooted:

ubuntu@lp1876018-f:~$ dpkg -l systemd|grep systemd
ii systemd 245.4-4ubuntu3.1 amd64 system and service manager
ubuntu@lp1876018-f:~$ journalctl -g vm-hotadd
-- Logs begin at Wed 2020-07-22 18:11:53 UTC, end at Wed 2020-07-22 18:46:23 UTC. --
Jul 22 18:45:15 lp1876018-f sudo[12380]: ubuntu : TTY=ttyS0 ; PWD=/home/ubuntu ; USER=root ; COMMAND=/usr/bin/vi /lib/udev/rules.d/40-vm-hotadd.rules
-- Reboot --
Jul 22 18:45:48 lp1876018-f systemd-udevd[457]: memory0: /usr/lib/udev/rules.d/40-vm-hotadd.rules:9 Failed to write ATTR{/sys/devices/system/memory/memory0/state}, ignoring: Invalid argument
Jul 22 18:45:48 lp1876018-f systemd-udevd[457]: memory1: /usr/lib/udev/rules.d/40-vm-hotadd.rules:9 Failed to write ATTR{/sys/devices/system/memory/memory1/state}, ignoring: Invalid argument
Jul 22 18:45:48 lp1876018-f systemd-udevd[457]: memory10: /usr/lib/udev/rules.d/40-vm-hotadd.rules:9 Failed to write ATTR{/sys/devices/system/memory/memory10/state}, ignoring: Invalid argument
...(many more)...

and with the udev rule again edited after the systemd upgrade (since it replaces the edited file), and the system rebooted:

ubuntu@lp1876018-f:~$ dpkg -l systemd|grep systemd
ii systemd 245.4-4ubuntu3.2 amd64 system and service manager
ubuntu@lp1876018-f:~$ journalctl -b -g vm-hotadd
-- Logs begin at Wed 2020-07-22 18:11:53 UTC, end at Wed 2020-07-22 18:50:00 UT>
-- No entries --

This bug was fixed in the package systemd - 245.4-4ubuntu3.2

---------------
systemd (245.4-4ubuntu3.2) focal; urgency=medium

   [ Dan Streetman ]
   * Hotadd only offline memory and CPUs (LP: #1876018)
     File: debian/extra/rules-ubuntu/40-vm-hotadd.rules
     https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=72d815471596056b7727be5b10f87513ff1d5757
   * Lock swap blockdevice while calling mkswap (LP: #1838329)
     Files:
     - d/p/lp1838329/0001-blockdev-propagate-one-more-unexpected-error.patch
     - d/p/lp1838329/0002-makefs-log-about-OOM-condition.patch
     - d/p/lp1838329/0003-dissect-use-log_debug_errno-where-appropriate.patch
     - d/p/lp1838329/0004-blockdev-add-helper-for-locking-whole-block-device.patch
     - d/p/lp1838329/0005-makefs-lock-device-while-we-operate.patch
     - d/p/lp1838329/0006-makefs-normalize-logging-a-bit.patch
     - d/p/lp1838329/0007-cryptsetup-generator-use-systemd-makefs-for-implemen.patch
     https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=c81b75c4297cbb04554488b070b6f79996b8cceb

   [ Balint Reczey ]
   * debian/udev.postinst: Allow kvm to be an already present non-system group
     (LP: #1880541)
     File: debian/udev.postinst
     https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=8b5c31828d4323ddb719326b1316c179b7cdbdef
   * d/p/hwdb-Mask-rfkill-event-from-intel-hid-on-HP-platforms.patch:
     hwdb: Mask rfkill event from intel-hid on HP platforms
     (LP: #1883846)
     https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=164c016b466210c7d6d05963fd753eccf4679844
   * journald: stream pid change newline fix (LP: #1875708)
     Files:
     - debian/patches/lp1875708/journald-Increase-stdout-buffer-size-sooner-when-almost-f.patch
     - debian/patches/lp1875708/journald-rework-end-of-line-marker-handling-to-use-a-fiel.patch
     - debian/patches/lp1875708/journald-rework-pid-change-handling.patch
     - debian/patches/lp1875708/journald-use-log_warning_errno-where-appropriate.patch
     - debian/patches/lp1875708/journald-use-the-fact-that-client_context_release-returns.patch
     - debian/patches/lp1875708/man-document-the-new-_LINE_BREAK-type.patch
     - debian/patches/lp1875708/socket-util-introduce-type-safe-dereferencing-wrapper-CMS.patch
     - debian/patches/lp1875708/test-Add-a-test-case-for-15654.patch
     https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=2dc19f7ae4aad7277e9d89849182453ff1d046dc

 -- Dan Streetman <email address hidden> Mon, 06 Jul 2020 17:38:31 -0400

The verification of the Stable Release Update for systemd has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Hello Jose, or anyone else affected,

Accepted systemd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/237-3ubuntu10.42 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

bionic verified, with udev rule changed per description testcase:

ii systemd 237-3ubuntu10.42 amd64 system and service manager
ubuntu@lp1876018-b:~$ journalctl -b |grep vm-hot
Jul 30 16:25:23 lp1876018-b systemd-udevd[509]: Reading rules file: /lib/udev/rules.d/40-vm-hotadd.rules

All autopkgtests for the newly accepted systemd (237-3ubuntu10.42) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

linux-hwe-5.0/5.0.0-58.62~18.04.1 (armhf)
linux-aws-edge/5.0.0-1019.21~18.04.1 (amd64)
linux-hwe-5.4/5.4.0-42.46~18.04.1 (arm64, armhf)
lxc/3.0.3-0ubuntu1~18.04.1 (arm64)
dovecot/1:2.2.33.2-1ubuntu4.5 (armhf)
casync/2+61.20180112-1 (amd64)
linux-raspi2-5.3/5.3.0-1030.32~18.04.2 (armhf)
umockdev/0.11.1-1 (armhf)
netplan.io/0.99-0ubuntu3~18.04.3 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

This bug was fixed in the package systemd - 237-3ubuntu10.42

---------------
systemd (237-3ubuntu10.42) bionic; urgency=medium

  [ Dan Streetman ]
  * d/p/lp1860926/0001-networkd-Allow-to-retain-configs-even-if-carrier-is-.patch,
    d/p/lp1860926/0002-network-Change-IgnoreCarrierLoss-default-to-value-of.patch,
    d/p/lp1860926/0003-network-always-drop-configs-when-corresponding-netwo.patch:
    - Add IgnoreCarrierLoss and default to value of ConfigureWithoutCarrier
      (LP: #1860926)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=9a12a31a62f1a50cd3a67a164ee34c546809815e
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=3cc3870fde47982a4dda53f820e18065e5488e7e
  * d/e/rules-ubuntu/40-vm-hotadd.rules:
    - Hotadd only offline memory and CPUs
      (LP: #1876018)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=ba305d7ad00e80bc1a03f93e6986eef7cbbb18fc
  * d/p/lp1881972-network-strdup-iif-and-oif-when-creating-RoutingPoli.patch:
    - Avoid double-free by strdup'ing iif/oif strings for new policy rules
      (LP: #1881972)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=874056f0d429aaa2cc872c3b35ec33cd3b740483
  * d/p/lp1886197-seccomp-more-comprehensive-protection-against-libsec.patch
    - Fix FTBFS on arm64 due to libseccomp changes (LP: #1886197)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=c284a72ca2e3d87bfe1c20afb2fcfb379cda544f
  * d/p/lp1832754/0001-umount-Try-unmounting-even-if-remounting-read-only-f.patch,
    d/p/lp1832754/0002-umount-Don-t-bother-remounting-api-and-ro-filesystem.patch:
    - Try unmounting even if ro-remount fails, and don't bother remounting api/ro fs
      (LP: #1832754)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=a518baa673aeaaf42000a3a01b7e03347652b216

  [ Alex Murray, Jamie Strandboge ]
  * d/p/lp1886115-pid1-fix-free-of-uninitialized-pointer-in-unit_fail_.patch:
    - Fix free of uninitialized pointer (LP: #1886115)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=491c76fd0f2fba0007a9b54d63a50f21add643c8

 -- Dan Streetman <email address hidden> Wed, 08 Jul 2020 14:59:14 -0400

Hello Jose, or anyone else affected,

Accepted systemd into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/229-4ubuntu21.29 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted systemd (229-4ubuntu21.29) for xenial have finished running.
The following regressions have been reported in tests triggered by the package:

systemd/229-4ubuntu21.29 (i386, amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/xenial/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

xenial verified, with test case repro modification to rules file:

ubuntu@lp1876018-x:~$ dpkg -l systemd|grep systemd
ii systemd 229-4ubuntu21.29 amd64 system and service manager
ubuntu@lp1876018-x:~$ journalctl -b |grep vm-hot
ubuntu@lp1876018-x:~$

This bug was fixed in the package systemd - 229-4ubuntu21.29

---------------
systemd (229-4ubuntu21.29) xenial; urgency=medium

  [ Dan Streetman ]
  * d/p/lp1881312/0001-core-don-t-dispatch-load-queue-when-setting-Slice-fo.patch,
    d/p/lp1881312/0002-run-make-slice-work-in-conjunction-with-scope.patch:
    - run new scope unit in specified slice (LP: #1881312)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=b5a778c3cd93e3d684602146f281f315f30778a8
  * d/p/lp1877176-sd-dhcp-client-validate-hostnames-stricter-7308.patch:
    - strictly validate hostname sent to dhcp server (LP: #1877176)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=7a1f1c41dcd860fcba6be42a2153cfe4315ebf17
  * d/e/rules-ubuntu/40-vm-hotadd.rules:
    - Hotadd only offline memory and CPUs
      (LP: #1876018)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=43357bd260a1c3e7b4951d33cbacaebda1d086b3
  * d/p/lp1698388-journal-remote-Ensure-reallocation-of-source-buf-doe.patch:
    - Fix assertion failure when journald source->filled < source->size
      (LP: #1698388)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=d1cc15b88113510227e5a76654e103ef54a8b8f3

  [ Heitor Alves de Siqueira ]
  * d/p/lp1876600-sd-bus-deal-with-cookie-overruns.patch:
    - deal with dbus cookie overruns (LP: #1876600)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=8dc9ef188944349169044d8974a4ffa55c66be9d

 -- Heitor Alves de Siqueira <email address hidden> Tue, 07 Jul 2020 17:28:11 -0400