dhclient triggers systemd-resolved start limit when processing more than 5 interfaces at once

I have tested this on both Focal and Bionic VMs. I ensured that the systemd-resolved unit was in a failed state, restarted it, added the PPA, updated the systemd packages, and rebooted. After every reboot the systemd-resolved was in an "active" state.

I tested this with at least 10 reboots on each affected Ubuntu release (b and f).

Hello Dan, or anyone else affected,

Accepted systemd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/237-3ubuntu10.52 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted systemd (237-3ubuntu10.52) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

casync/2+61.20180112-1 (s390x)
nftables/0.8.2-1 (amd64)
lxc/3.0.3-0ubuntu1~18.04.1 (amd64, i386, s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Hello Dan, or anyone else affected,

Accepted systemd into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/245.4-4ubuntu3.12 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted systemd (245.4-4ubuntu3.12) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

flatpak/1.6.5-0ubuntu0.3 (amd64)
gvfs/1.44.1-1ubuntu1 (amd64, arm64)
munin/2.0.56-1ubuntu1 (s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

ubuntu@lp1939255-f:~$ dpkg -l systemd|grep systemd
ii systemd 245.4-4ubuntu3.11 amd64 system and service manager
ubuntu@lp1939255-f:~$ sudo dhclient ens8 ens9 ens10 ens11 ens12 ens13 ens14 ens15
ubuntu@lp1939255-f:~$ journalctl --no-pager -b -u systemd-resolved
-- Logs begin at Sun 2021-09-05 21:35:56 UTC, end at Sun 2021-09-05 21:40:13 UTC. --
...
Sep 05 21:40:13 lp1939255-f systemd[1]: systemd-resolved.service: Start request repeated too quickly.
Sep 05 21:40:13 lp1939255-f systemd[1]: systemd-resolved.service: Failed with result 'start-limit-hit'.
Sep 05 21:40:13 lp1939255-f systemd[1]: Failed to start Network Name Resolution.

ubuntu@lp1939255-f:~$ dpkg -l systemd|grep systemd
ii systemd 245.4-4ubuntu3.12 amd64 system and service manager
ubuntu@lp1939255-f:~$ sudo dhclient ens8 ens9 ens10 ens11 ens12 ens13 ens14 ens15
ubuntu@lp1939255-f:~$ journalctl --no-pager -b -u systemd-resolved
-- Logs begin at Sun 2021-09-05 21:35:56 UTC, end at Sun 2021-09-05 21:44:25 UTC. --
...
Sep 05 21:44:23 lp1939255-f systemd[1]: Started Network Name Resolution.

ubuntu@lp1939255-b:~$ dpkg -l systemd|grep systemd
ii systemd 237-3ubuntu10.51 amd64 system and service manager
ubuntu@lp1939255-b:~$ sudo dhclient ens8 ens9 ens10 ens11 ens12 ens13 ens14 ens15
ubuntu@lp1939255-b:~$ journalctl --no-pager -b -u systemd-resolved
-- Logs begin at Sun 2021-09-05 21:36:12 UTC, end at Sun 2021-09-05 21:41:06 UTC. --
...
Sep 05 21:40:57 lp1939255-b systemd[1]: systemd-resolved.service: Start request repeated too quickly.
Sep 05 21:40:57 lp1939255-b systemd[1]: systemd-resolved.service: Failed with result 'start-limit-hit'.
Sep 05 21:40:57 lp1939255-b systemd[1]: Failed to start Network Name Resolution.

ubuntu@lp1939255-b:~$ dpkg -l systemd|grep systemd
ii systemd 237-3ubuntu10.52 amd64 system and service manager
ubuntu@lp1939255-b:~$ sudo dhclient ens8 ens9 ens10 ens11 ens12 ens13 ens14 ens15
ubuntu@lp1939255-b:~$ journalctl --no-pager -b -u systemd-resolved
-- Logs begin at Sun 2021-09-05 21:36:12 UTC, end at Sun 2021-09-05 21:50:57 UTC. --
...
Sep 05 21:50:57 lp1939255-b systemd[1]: Started Network Name Resolution.

Hello Dan, or anyone else affected,

Accepted systemd into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/245.4-4ubuntu3.13 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted systemd (245.4-4ubuntu3.13) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

gvfs/1.44.1-1ubuntu1 (amd64, ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

as systemd was respun (for focal) due to bug 1942899, and only the one udev (hwdb) patch was reverted which shouldn't affect this at all, I'm remarking this as verified still based on testing for the previous version above.

The verification of the Stable Release Update for systemd has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package systemd - 237-3ubuntu10.52

---------------
systemd (237-3ubuntu10.52) bionic; urgency=medium

  * d/extra/dhclient-enter-resolved-hook:
    Reset start limit counter for systemd-resolved in dhclient hook
    (LP: #1939255)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=ea6710476dde78e8595274c3c4ba7acca6d5162c
  * d/p/lp1934147/0001-core-add-a-new-unit-method-catchup.patch,
    d/p/lp1934147/0002-cgroup-do-catchup-for-unit-cgroup-inotify-watch-file.patch,
    d/p/lp1934147/0003-core-Make-sure-cgroup_oom_queue-is-flushed-on-manage.patch:
    Catch up on cgroup empty inotify after reexec/reload (LP: #1934147)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=5ef61bd930612a90ce3ed9105cbadc5ff97b6ffc
  * d/p/lp1934981-correct-suspend-then-sleep-string.patch:
    Fix sleep verb used by logind during suspend-then-hibernate
    (LP: #1934981)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=1ade873a41ad018a5e07f10775738c6eb8c82310
  * d/extra/dhclient-enter-resolved-hook:
    Check is-enabled systemd-resolved in dhclient hook (LP: #1853164)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=774c2f82a39a88fa0fd8b2adbfa0b8a8c3cd1fb5

 -- Dan Streetman <email address hidden> Thu, 26 Aug 2021 10:20:40 -0400

This bug was fixed in the package systemd - 245.4-4ubuntu3.13

---------------
systemd (245.4-4ubuntu3.13) focal; urgency=medium

  * d/p/dell-clamshell-accel-location-base-with-sku.patch:
    Revert incorrect patch (LP: #1942899)

systemd (245.4-4ubuntu3.12) focal; urgency=medium

  [ Yao Wei ]
  * d/p/dell-clamshell-accel-location-base.patch:
    Add ACCEL_LOCATION=base property for Dell clamshell models (LP: #1938259)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=5c1be33900edee94da0dc9a4ade8edcd079b4c85

  [ Lukas Märdian ]
  * Add d/p/lp1934221-resolved-disable-event-sources-before-unreffing-them.patch
    - Fix segfault in systemd-resolve (LP: #1934221)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=6c401900c70962052f56c7108fdc02fe7f84c9bf

  [ Simon Chopin ]
  * d/p/lp1914740-network-enable-DHCP-broadcast-flag-if-required-by-in.patch:
    - Apply upstream patch to fix Hipersocket DHCP mode (LP: #1914740)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=326ae43b7966d9e7c5f7124027185a79a07fa276

  [ Dan Streetman ]
  * d/p/lp1934981-correct-suspend-then-sleep-string.patch:
    Fix sleep verb used by logind during suspend-then-hibernate
    (LP: #1934981)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=997f3a7da3d5db22e3c63626c3f7dc3dff0830b0
  * d/p/lp1937238-util-return-the-correct-correct-wd-from-inotify-help.patch:
    Fix watch for time sync (LP: #1937238)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=dbabff8a03eb232c19174eff1335cd7cb7d7860c
  * d/extra/dhclient-enter-resolved-hook:
    Reset start limit counter for systemd-resolved in dhclient hook
    (LP: #1939255)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=9d3a91a0b70a4b2bcc166f366cd0a880fd494812
  * d/p/lp1935051-shared-unit-file-make-sure-the-old-hashmaps-and-sets.patch:
    Fix memory leak in path cache (LP: #1935051)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=12d6bdeb35f309158fe8d4242c6dd9be4d067604
  * d/p/lp1934147/0001-cgroup-do-catchup-for-unit-cgroup-inotify-watch-file.patch,
    d/p/lp1934147/0002-core-Make-sure-cgroup_oom_queue-is-flushed-on-manage.patch:
    Catchup cgroup inotify watch after reexec/reload (LP: #1934147)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=63eabc88b8e0005eb40b15b543538ce35377bdbd

 -- Dan Streetman <email address hidden> Tue, 07 Sep 2021 14:37:22 -0400

Hi. It seems this issue may occur not only when receiving addresses on multiple interfaces, but also when dhcp requests on multiple interfaces times out and the hook script was called with 'reason' == FAIL.
Found on Ubuntu 20.04.3 LTS, systemd is 245.4-4ubuntu3.15.

> It seems this issue may occur not only when receiving addresses on multiple interfaces, but also when dhcp requests on multiple interfaces times out and the hook script was called with 'reason' == FAIL.

yep, the try-reload-or-restart is done for those as well and this should be fixed for those $reason cases; since this bug is already resolved, I suggest you open a new bug.