Networking broken after systemd update

Bug #1891716 reported by David Krauser
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
systemd (Debian)
Fix Released
Unknown
systemd (Ubuntu)
Fix Released
High
Balint Reczey

Bug Description

# Description:

After this upgrade in groovy, networking stopped working:

Start-Date: 2020-08-14 17:02:50
Commandline: apt-get dist-upgrade --assume-yes
Requested-By: ubuntu (1000)
Install: motd-news-config:amd64 (11ubuntu11, automatic)
Upgrade: libaudit-common:amd64 (1:2.8.5-2ubuntu6, 1:2.8.5-3ubuntu1), alsa-ucm-conf:amd64 (1.2.2-1ubuntu1, 1.2.2-1ubuntu2), libmpfr6:amd64 (4.0.2-1build1, 4.1.0-3), python3-software-properties:amd64 (0.98.10, 0.99.2), cloud-initramfs-dyn-netconf:amd64 (0.45ubuntu1, 0.46ubuntu1), libsystemd0:amd64 (245.7-1ubuntu1, 246-2ubuntu1), libpam-cap:amd64 (1:2.36-1, 1:2.42-2), libgmp10:amd64 (2:6.2.0+dfsg-6, 2:6.2.0+dfsg-6ubuntu1), ubuntu-standard:amd64 (1.452, 1.453), nano:amd64 (4.9.3-1, 5.1-1), udev:amd64 (245.7-1ubuntu1, 246-2ubuntu1), ubuntu-server:amd64 (1.452, 1.453), libpipeline1:amd64 (1.5.2-2build1, 1.5.3-1), isc-dhcp-common:amd64 (4.4.1-2.1ubuntu6, 4.4.1-2.1ubuntu9), libudev1:amd64 (245.7-1ubuntu1, 246-2ubuntu1), systemd-timesyncd:amd64 (245.7-1ubuntu1, 246-2ubuntu1), ubuntu-minimal:amd64 (1.452, 1.453), libcap2:amd64 (1:2.36-1, 1:2.42-2), systemd-sysv:amd64 (245.7-1ubuntu1, 246-2ubuntu1), libcap2-bin:amd64 (1:2.36-1, 1:2.42-2), libpam-systemd:amd64 (245.7-1ubuntu1, 246-2ubuntu1), systemd:amd64 (245.7-1ubuntu1, 246-2ubuntu1), libnss-systemd:amd64 (245.7-1ubuntu1, 246-2ubuntu1), ethtool:amd64 (1:5.4-1, 1:5.8-1), overlayroot:amd64 (0.45ubuntu1, 0.46ubuntu1), cloud-initramfs-copymods:amd64 (0.45ubuntu1, 0.46ubuntu1), isc-dhcp-client:amd64 (4.4.1-2.1ubuntu6, 4.4.1-2.1ubuntu9), libaudit1:amd64 (1:2.8.5-2ubuntu6, 1:2.8.5-3ubuntu1), base-files:amd64 (11ubuntu10, 11ubuntu11), software-properties-common:amd64 (0.98.10, 0.99.2)
End-Date: 2020-08-14 17:03:10

# Reproduction steps:

$ multipass launch --name groovy-update-test daily:groovy
$ multipass connect groovy-update-test
# connected to VM
$ sudo apt-get update && sudo apt-get dist-upgrade --assume-yes
# works fine
$ sudo apt-get update
Err:1 http://security.ubuntu.com/ubuntu groovy-security InRelease
  Temporary failure resolving 'security.ubuntu.com'
Err:2 http://archive.ubuntu.com/ubuntu groovy InRelease
  Temporary failure resolving 'archive.ubuntu.com'
Err:3 http://archive.ubuntu.com/ubuntu groovy-updates InRelease
  Temporary failure resolving 'archive.ubuntu.com'
Err:4 http://archive.ubuntu.com/ubuntu groovy-backports InRelease
  Temporary failure resolving 'archive.ubuntu.com'
Reading package lists... Done
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/groovy/InRelease Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/groovy-updates/InRelease Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/groovy-backports/InRelease Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/groovy-security/InRelease Temporary failure resolving 'security.ubuntu.com'
W: Some index files failed to download. They have been ignored, or old ones used instead.

$ sudo systemctl list-units --failed
  UNIT LOAD ACTIVE SUB DESCRIPTION
● systemd-networkd.service loaded failed failed Network Service
● systemd-networkd.socket loaded failed failed Network Service Netlink Socket

LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.

2 loaded units listed.
$ systemctl status systemd-networkd.service
● systemd-networkd.service - Network Service
     Loaded: loaded (/lib/systemd/system/systemd-networkd.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Fri 2020-08-14 17:03:01 EDT; 12min ago
TriggeredBy: ● systemd-networkd.socket
       Docs: man:systemd-networkd.service(8)
   Main PID: 2745 (code=exited, status=1/FAILURE)

Aug 14 17:03:01 fulfilling-hornet systemd[1]: systemd-networkd.service: Main process exited, code=exited, status=1/FAILURE
Aug 14 17:03:01 fulfilling-hornet systemd[1]: systemd-networkd.service: Failed with result 'exit-code'.
Aug 14 17:03:01 fulfilling-hornet systemd[1]: Failed to start Network Service.
Aug 14 17:03:01 fulfilling-hornet systemd[1]: systemd-networkd.service: Scheduled restart job, restart counter is at 5.
Aug 14 17:03:01 fulfilling-hornet systemd[1]: Stopped Network Service.
Aug 14 17:03:01 fulfilling-hornet systemd[1]: systemd-networkd.service: Start request repeated too quickly.
Aug 14 17:03:01 fulfilling-hornet systemd[1]: systemd-networkd.service: Failed with result 'exit-code'.
Aug 14 17:03:01 fulfilling-hornet systemd[1]: Failed to start Network Service.

description: updated
Revision history for this message
Joshua Powers (powersj) wrote :

Here are my notes from using a Groovy LXD container:

Expected Results
---
$ resolvectl status | grep "DNS Server"
  Current DNS Server: 192.168.100.1
         DNS Servers: 192.168.100.1

Actual Results
---
$ resolvectl status
Global
       LLMNR setting: no
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 24 (eth0)
      Current Scopes: none
DefaultRoute setting: no
       LLMNR setting: yes
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Steps to reproduce
---
$ lxc launch ubuntu-daily:g g
$ lxc exec g bash
# cat /etc/cloud/build.info
build_name: server
serial: 20200813
# sudo apt update -qq
# apt-cache policy systemd
systemd:
  Installed: 245.7-1ubuntu1
  Candidate: 246-2ubuntu1
  Version table:
     246-2ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu groovy/main amd64 Packages
 *** 245.7-1ubuntu1 100
        100 /var/lib/dpkg/status
# resolvectl status | grep "DNS Server"
  Current DNS Server: 192.168.100.1
         DNS Servers: 192.168.100.1
# sudo apt install --yes systemd
...
Job for systemd-networkd.service failed because the control process exited with error code.
See "systemctl status systemd-networkd.service" and "journalctl -xe" for details.
Removing obsolete conffile /etc/dhcp/dhclient-enter-hooks.d/resolved ...
...
# resolvectl status
Global
       LLMNR setting: no
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 24 (eth0)
      Current Scopes: none
DefaultRoute setting: no
       LLMNR setting: yes
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Revision history for this message
Joshua Powers (powersj) wrote :
Revision history for this message
Joshua Powers (powersj) wrote :
description: updated
Revision history for this message
Joshua Powers (powersj) wrote :

# journalctl -u systemd-networkd
https://paste.ubuntu.com/p/9M3xwgwgMj/

# systemctl status systemd-networkd.service
● systemd-networkd.service - Network Service
     Loaded: loaded (/lib/systemd/system/systemd-networkd.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Fri 2020-08-14 20:59:46 UTC; 19min ago
TriggeredBy: ● systemd-networkd.socket
       Docs: man:systemd-networkd.service(8)
   Main PID: 1107 (code=exited, status=1/FAILURE)

Aug 14 20:59:46 g systemd[1]: systemd-networkd.service: Scheduled restart job, restart counter is at 5.
Aug 14 20:59:46 g systemd[1]: Stopped Network Service.
Aug 14 20:59:46 g systemd[1]: systemd-networkd.service: Start request repeated too quickly.
Aug 14 20:59:46 g systemd[1]: systemd-networkd.service: Failed with result 'exit-code'.
Aug 14 20:59:46 g systemd[1]: Failed to start Network Service.

Balint Reczey (rbalint)
Changed in systemd (Ubuntu):
importance: Undecided → Critical
assignee: nobody → Balint Reczey (rbalint)
Revision history for this message
David Krauser (davidkrauser) wrote :

It looks like this may only be an upgrade failure:

If you grab the 20200812 groovy image with systemd v245.7-1ubuntu1:
https://cloud-images.ubuntu.com/minimal/daily/groovy/20200812/groovy-minimal-cloudimg-amd64.img

And update systemd to v246-2ubuntu1, you see the breakage described in this bug.

If you grab the 20200817.1 groovy image with systemd v246-2ubuntu1:
https://cloud-images.ubuntu.com/minimal/daily/groovy/20200817.1/groovy-minimal-cloudimg-amd64.img

You won't see the breakage.

Balint Reczey (rbalint)
Changed in systemd (Ubuntu):
status: New → In Progress
Revision history for this message
Balint Reczey (rbalint) wrote :
Revision history for this message
Balint Reczey (rbalint) wrote :

Since this affects only upgrades from pre-246 versions the severity should be High only. I'm testing the fix in Bileto and will upload it if the tests found no regression:

https://bileto.ubuntu.com/#/ticket/3801

Changed in systemd (Ubuntu):
importance: Critical → High
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (3.4 KiB)

This bug was fixed in the package systemd - 246.2-1ubuntu1

---------------
systemd (246.2-1ubuntu1) groovy; urgency=medium

  * Merge from Debian unstable
  * Refresh patches
  * test: accept that char device 0/0 can now be created witout privileges.
    This fixes tests with Linux 5.8 and later (LP: #1891527)
    Files:
    - debian/patches/debian/Add-env-variable-for-machine-ID-path.patch
    - debian/patches/test-accept-that-char-device-0-0-can-now-be-created-witou.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=57909a0fa4999ba9ff918cfe6bb3fa53c230dd36
  * debian/tests/systemd-fsckd: Fail plymouth-start with overriding ExecStart=
    The 'process-killer' stopped working with plymouth 0.9.5 (LP: #1886886)
    File: debian/tests/systemd-fsckd
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=efcc25b6011f23b9f90841f4ca0ee29ee674157f
  * cherry-pick from stable: network: only process non-error message
    File: debian/patches/network-only-process-non-error-message.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=dc97e3f5330ae64909ef640ad1ac401374f18c16
  * network: wait for previous address removal before configuring static addresses.
    Fixes systemd-networkd to establish connection after resume from suspend
    Files:
    - debian/patches/network-wait-for-previous-address-removal-before-configur.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=31390403bc000892ada9cb0cbedb198dd83fe7a2
  * network: do not fail if UseMTU=yes on DHCP lease lost
    File: debian/patches/network-do-not-fail-if-UseMTU-yes-on-DHCP-lease-lost.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=21c23966e8dc5fd0858664a5b89d6a9460c41f61
  * debian/systemd.postinst: Restart systemd-networkd.socket on incompatible change
    (Closes: #968589) (LP: #1891716)
    File: debian/systemd.postinst
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=e144a644488c6ff52c2bb7cee71f50aa95c4efd8
  * debian/tests/control: Mark systemd-fsckd test as flaky.
    It is flaky on Ubuntu CI infrastructure (LP: #1892358)
    File: debian/tests/control
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=8c55289624c6f44beb2c5bbae78aabc074d36b1a

systemd (246.2-1) unstable; urgency=medium

  * New upstream version 246.2
  * Remove resolvconf.conf drop-in, resolved integration moved to resolvconf
    package
  * Rebase patches
  * Add versioned Breaks against resolvconf (<< 1.83~) to systemd.
    The PathExists= directive was changed in v246 to match the documented
    behaviour but now causes resolvconf-pull-resolved.service to be
    continuously triggered by resolvconf-pull-resolved.path.
    This requires a fix in the resolvconf package, see #968015.
    (Closes: #967906)
  * Keep journal files compatible with older versions.
    Disable the KEYED-HASH journal feature by default and keep LZ4 (instead
    of ZSTD) as default compression for new journal files. Otherwise journal
    files are incompatible and can't be read by older journalctl
    implementation...

Read more...

Changed in systemd (Ubuntu):
status: In Progress → Fix Released
tags: added: id-5f3afdc64e09838ff1afe222
tags: added: fr-188
Changed in systemd (Debian):
status: Unknown → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.