systemd-networkd fails to apply static IPv4 when the static IP is the same as previously configured by DHCP

I should add that if I set "dhcp4: no" explicit when confiugring static addressing, it behaves as expected, but according to the current documentation, this parameter should default to "no", so this should not be neccescary.

Can you paste an example networkd config file, and repro instructions, that reproduces this problem?

Repro instructions:
- (Enable systemd-networkd debug logging)
- Start a server with its interface configured with DHCP
- In the netplan config, configure the interface wtih static addressing, using the exact same address as it already has gotten from DHCP
- Either wait for the initial DHCP lease time to expire, and observe that the interface loses its address; OR observe in the debug log that networkd applied the address with a lease time, even though you told netplan to use static addressing

Networkd config with static addressing:
cat /run/systemd/network/10-netplan-ens3.network
[Match]
MACAddress=<redacted>

[Network]
LinkLocalAddressing=ipv6
Address=10.212.132.107/24
Gateway=10.212.132.1
DNS=8.8.8.8
DNS=8.8.4.4
Domains=<redacted>

Networkd config with DHCP:
cat /run/systemd/network/10-netplan-ens3.network
[Match]
MACAddress=<redacted>
[Network]
DHCP=ipv4
LinkLocalAddressing=ipv6

[DHCP]
RouteMetric=100
UseMTU=true

Debug log output from systemd-networkd when netplan tells it to apply static addressing when the interface already has an address from DHCP:
Sep 30 10:36:15 nettest1 systemd[1]: Stopped Network Service.
Sep 30 10:36:16 nettest1 systemd[1]: Starting Network Service...
<snip>
Sep 30 10:36:16 nettest1 systemd-networkd[6086]: ens3: Adding address: 10.212.132.107/24 (valid for 11h 52min 55s)

Thanks! that clarifies it - I think this is a bug upstream with systemd.

> if I set "dhcp4: no" explicit when confiugring static addressing, it behaves as expected

are you sure? that isn't what I would expect, it seems the problem is networkd doesn't adjust the address lifetime when networkd is restarted and the address changes from dhcp->static; i would not expect the DHCP=no param it fix that. Although that's a netplan config, so possibly netplan does something additional under the covers, other than what systemd-networkd does.

> It seems like upon a restart, systemd-networkd will allways add whatever IP config it had before the service stopped, and then apply changes

not exactly, what's happening is when you stop networkd, it does not remove network configuration that it manages, so in this case it would leave the dhcp address on the interface. then when you (re)start networkd, it does not change existing addresses that it manages, so it sees that the interface already has the address that it's been configured to use as a static address, and does not change it. It looks like networkd needs to also check the address's lifetime to make sure static addrs have their lifetime set to 'forever' (I have not actually checked the code yet to verify this).

If you reboot, or manually remove the address, or just bring the interface down and back up, networkd will re-add the address with 'forever' lft, so this should only be a problem when you have manually changed the config for an address from dhcp->static using the same ip address, and then restart networkd without rebooting, at least IIUC.

Hmm looks like https://github.com/systemd/systemd/issues/11458

Yes that's the same issue. And it seems to be fixed in some version as well? I just tested it on Eoan, using systemd 242-6, and the valid__lft and preferred_lft did indeed change to 'forever'. Don't know which version it was fixed in though. Do you think it kan be backported to Bionic?

The discussion in the GitHub issue was rather interesting as well. I kind of "agree" that this is a strange thing to do, but I don't think that should be any of systemd's business..

> I just tested it on Eoan, using systemd 242-6, and the valid__lft and preferred_lft did indeed change to 'forever'.

well that's interesting...I tested with the latest upstream systemd code and i *was* able to reproduce the problem...maybe something reverted the fix since 242. Unless, of course, I messed up my test :-) I'll backport the fix for that and see if it fixes bionic, then retest the latest upstream to make sure it hasn't reverted.

> The discussion in the GitHub issue was rather interesting as well. I kind of "agree" that this is a strange thing to do, but I don't think that should be any of systemd's business..

yes, I agree, and it is an entertaining discussion on what systemd should prevent users from doing ;)

Sorry for the delay; I need to re-test this again with upstream code to verify it does reproduce there (or not) before working to backport any fix.

Update: I retested today, and I must have made a mistake in my previous upstream tests; it does appear fixed with upstream systemd, as well as being fixed in Ubuntu E/F.

as disco is EOL next week, marking this won't fix for disco.

Hello Lars, or anyone else affected,

Accepted systemd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/237-3ubuntu10.34 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted systemd (237-3ubuntu10.34) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

php7.2/7.2.24-0ubuntu0.18.04.2 (armhf)
openssh/1:7.6p1-4ubuntu0.3 (arm64, armhf, ppc64el, amd64, s390x, i386)
dovecot/1:2.2.33.2-1ubuntu4.5 (armhf)
gvfs/1.36.1-0ubuntu1.3.3 (ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Hi!

Tested with systemd=237-3ubuntu10.34 from proposed (and libsystemd0=237-3ubuntu10.34) and it seems to fix the problem.

What's interesting though, is that a fresh install of bionic came with a newer version (systemd=237-3ubuntu10.38) - which has the bug. This test effectively ended up in a downgrade.

> What's interesting though, is that a fresh install of bionic came with a newer version (systemd=237-3ubuntu10.38) - which has the bug. This test effectively ended up in a downgrade.

yes, a security release for systemd overrode the version with this fix in -proposed, so I have to rebase these patches and re-upload. These things sometimes happen, unfortunately.

This SRU needs to be reuploaded, due to security update that trumped this in progress SRU.

Hello Lars, or anyone else affected,

Accepted systemd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/237-3ubuntu10.39 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted systemd (237-3ubuntu10.39) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

php7.2/7.2.24-0ubuntu0.18.04.2 (armhf)
gvfs/1.36.1-0ubuntu1.3.3 (ppc64el)
lxc/3.0.3-0ubuntu1~18.04.1 (amd64)
systemd/237-3ubuntu10.39 (i386)
netplan.io/0.98-0ubuntu1~18.04.1 (i386, amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Hi!

Re-tested with systemd=237-3ubuntu10.39 from proposed (and libsystemd0=237-3ubuntu10.39), and it works.

The verification of the Stable Release Update for systemd has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package systemd - 237-3ubuntu10.39

---------------
systemd (237-3ubuntu10.39) bionic; urgency=medium

  [ Dariusz Gadomski ]
  * d/p/lp1762391/0001-Call-getgroups-to-know-size-of-supplementary-groups-.patch,
    d/p/lp1762391/0002-user-util-tweak-to-in_gid.patch,
    d/p/lp1762391/0003-user-util-Add-helper-functions-for-gid-lists-operati.patch,
    d/p/lp1762391/0004-execute-Restore-call-to-pam_setcred.patch,
    d/p/lp1762391/0005-execute-Detect-groups-added-by-PAM-and-merge-them-wi.patch,
    d/p/lp1762391/0006-test-Add-tests-for-gid-list-ops.patch,
    d/p/lp1762391/0007-execute-add-const-to-array-parameters-where-possible.patch,
    d/p/lp1762391/0008-execute-allow-pam_setcred-to-fail-ignore-errors.patch:
    - Restore call to pam_setcred (LP: #1762391)

  [ Ioanna Alifieraki ]
  * d/p/lp1860548/0001-Revert-Replace-use-of-snprintf-with-xsprintf.patch,
    d/p/lp1860548/0002-job-truncate-unit-description.patch:
    - use snprintf instead of xsprintf (LP: #1860548)

  [ Dan Streetman ]
  * d/p/lp1833193-network-update-address-when-static-address-was-alrea.patch:
    - Update lft when static addr was cfg by dhcp (LP: #1833193)
  * d/p/lp1849261/0001-core-when-we-can-t-enqueue-OnFailure-job-show-full-e.patch,
    d/p/lp1849261/0002-core-don-t-trigger-OnFailure-deps-when-a-unit-is-goi.patch:
    - Only trigger OnFailure= if Restart= is not in effect (LP: #1849261)
  * d/p/lp1671951-network-set-ipv6-mtu-after-link-up-or-device-mtu-cha.patch:
    - set ipv6 mtu at correct time (LP: #1671951)
  * d/p/lp1845909/0001-networkd-honour-LinkLocalAddressing.patch,
    d/p/lp1845909/0002-networkd-fix-link_up-12505.patch,
    d/p/lp1845909/0003-network-do-not-send-ipv6-token-to-kernel.patch,
    d/p/lp1845909/0004-network-rename-linux_configure_after_setting_mtu-to-linux.patch,
    d/p/lp1845909/0005-network-add-link-setting_genmode-flag.patch,
    d/p/lp1845909/0006-network-if-ipv6ll-is-disabled-enumerate-tentative-ipv6-ad.patch,
    d/p/lp1845909/0007-network-drop-foreign-config-after-addr_gen_mode-has-been-.patch,
    d/p/lp1845909/0008-network-drop-IPv6LL-address-when-LinkLocalAddressing.patch:
    - if LinkLocalAddressing=no prevent creation of ipv6ll (LP: #1845909)
  * d/p/lp1859862-network-Do-not-disable-IPv6-by-writing-to-sysctl.patch:
    - enable ipv6 when needed (LP: #1859862)
  * d/p/lp1836695-networkd-Add-back-static-routes-after-DHCPv4-lease-e.patch:
    - (re)add static routes after getting dhcp4 addr (LP: #1836695)
  * d/t/storage:
    - fix buggy test (LP: #1831459)
    - without scsi_debug, skip test (LP: #1847816)

 -- Dan Streetman <email address hidden> Thu, 06 Feb 2020 10:00:49 -0500