Ubuntu
systemd package

netlink: 'systemd-network': attribute type 5 has an invalid length.

Bug #1877159 reported by Simon Déziel
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
systemd (Ubuntu)
Fix Released
Undecided
Unassigned
Bionic
Fix Released
Low
Dan Streetman
Eoan
Fix Released
Undecided
Unassigned
Focal
Fix Released
Undecided
Unassigned

Bug Description

[impact]

systemd-networkd uses incorrect netlink attribute length for wireguard's persistent keepalive interval, which logs error messages from the kernel, and may incorrectly set the parameter.

[test case]

Only 1 Bionic VM is required to reproduce the problem:

$ lxc launch images:ubuntu/bionic --vm -c security.secureboot=false foo
$ sleep 10 # allow booting
$ lxc exec foo -- apt install -y software-properties-common
$ lxc exec foo -- add-apt-repository -y ppa:wireguard/wireguard
$ lxc exec foo -- apt install -y wireguard-tools

$ cat << EOF | lxc exec foo -- tee /etc/systemd/network/wg0.netdev
# foo
[NetDev]
Name=wg0
Kind=wireguard

[WireGuard]
ListenPort=5555
PrivateKey=cBkljQSKhtEe/U8GZmCAk2MBbKWL4TLC9PVtbMFyCVQ=

[WireGuardPeer]
PublicKey=emfIuZ3hZ+AnWIrKex/EqCp2mfzip8AxJu6RuweyRGc=
AllowedIPs=192.168.255.2
Endpoint=bar.lxd:5555
EOF

$ cat << EOF | lxc exec foo -- tee /etc/systemd/network/wg0.network
# foo
[Match]
Name=wg0

[Network]
Address=192.168.255.1/24
EOF

$ lxc exec foo -- systemctl restart systemd-networkd

# notice the invalid length in dmesg
$ lxc exec foo -- journalctl -kn 8
-- Logs begin at Mon 2020-05-11 16:56:40 UTC, end at Mon 2020-05-11 17:03:46 UTC. --
May 11 16:58:25 foo kernel: nf_tables: (c) 2007-2009 Patrick McHardy <email address hidden>
May 11 17:01:57 foo kernel: PKCS#7 signature not signed with a trusted key
May 11 17:01:57 foo kernel: wireguard: module verification failed: signature and/or required key missing - tainting kernel
May 11 17:01:57 foo kernel: wireguard: WireGuard 1.0.20200429 loaded. See www.wireguard.com for information.
May 11 17:01:57 foo kernel: wireguard: Copyright (C) 2015-2019 Jason A. Donenfeld <email address hidden>. All Rights Reserved.
May 11 17:01:57 foo kernel: netlink: 'systemd-network': attribute type 5 has an invalid length.
May 11 17:01:57 foo kernel: netlink: 'systemd-network': attribute type 5 has an invalid length.
May 11 17:02:23 foo kernel: netlink: 'systemd-network': attribute type 5 has an invalid length.

[regression potential]

this adjusts the length of the specific netlink parameter, so any regression would likely relate to incorrectly setting the persistent keepalive interval parameter, or failure to set the parameter.

[scope]

this is needed only for Bionic.

this was fixed upstream in commit 7d0b26a027118ca063780421cb31c74e9d2664ee which was first included in v240, so this is fixed in Eoan and later. Xenial does not include support for wireguard, so this does not apply there.

[original description]

This morning, our 2 Bionic machine configured with the wireguard's PPA and using systemd-networkd to configure the wireguard tunnel started misbehaving. Why this started just now is unclear ATM but their dmesg was filled with this:

validate_nla: 100 callbacks suppressed
netlink: 'systemd-network': attribute type 5 has an invalid length.
netlink: 'systemd-network': attribute type 5 has an invalid length.
netlink: 'systemd-network': attribute type 5 has an invalid length.
netlink: 'systemd-network': attribute type 5 has an invalid length.
netlink: 'systemd-network': attribute type 5 has an invalid length.
netlink: 'systemd-network': attribute type 5 has an invalid length.
netlink: 'systemd-network': attribute type 5 has an invalid length.
netlink: 'systemd-network': attribute type 5 has an invalid length.
netlink: 'systemd-network': attribute type 5 has an invalid length.
netlink: 'systemd-network': attribute type 5 has an invalid length.

Folks in #systemd mentioned https://github.com/systemd/systemd/issues/11575 which points to 2 commits missing from Bionic's systemd version:

https://github.com/systemd/systemd/commit/7d0b26a027118ca063780421cb31c74e9d2664ee
https://github.com/systemd/systemd/commit/624a47694cad4c87b2e807c32db656f3e9d679c5

Focal's systemd have the above commits. Would it be possible to backport those 2 commits to Bionic?

Additional information:

# uname -a
Linux noc-eu1 4.15.0-99-generic #100-Ubuntu SMP Wed Apr 22 20:32:56 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

# apt-cache policy systemd wireguard{,-tools,-dkms}
systemd:
  Installed: 237-3ubuntu10.39
  Candidate: 237-3ubuntu10.39
  Version table:
 *** 237-3ubuntu10.39 500
        500 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     237-3ubuntu10.38 500
        500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
     237-3ubuntu10 500
        500 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages
wireguard:
  Installed: 1.0.20200319-1ubuntu1~18.04
  Candidate: 1.0.20200319-1ubuntu1~18.04
  Version table:
 *** 1.0.20200319-1ubuntu1~18.04 500
        500 http://ppa.launchpad.net/wireguard/wireguard/ubuntu bionic/main amd64 Packages
        500 http://ppa.launchpad.net/wireguard/wireguard/ubuntu bionic/main i386 Packages
        100 /var/lib/dpkg/status
wireguard-tools:
  Installed: 1.0.20200319-1ubuntu1~18.04
  Candidate: 1.0.20200319-1ubuntu1~18.04
  Version table:
 *** 1.0.20200319-1ubuntu1~18.04 500
        500 http://ppa.launchpad.net/wireguard/wireguard/ubuntu bionic/main amd64 Packages
        100 /var/lib/dpkg/status
wireguard-dkms:
  Installed: 1.0.20200429-2~18.04
  Candidate: 1.0.20200429-2~18.04
  Version table:
 *** 1.0.20200429-2~18.04 500
        500 http://ppa.launchpad.net/wireguard/wireguard/ubuntu bionic/main amd64 Packages
        500 http://ppa.launchpad.net/wireguard/wireguard/ubuntu bionic/main i386 Packages
        100 /var/lib/dpkg/status

See original description
Balint Reczey (rbalint)
Changed in systemd (Ubuntu):
status: New → Fix Released
Changed in systemd (Ubuntu Eoan):
status: New → Fix Released
Changed in systemd (Ubuntu Focal):
status: New → Fix Released
Revision history for this message
Simon Déziel (sdeziel) wrote :

Here is a strace of systemd-networkd when it was consuming 100% CPU: https://paste.ubuntu.com/p/2XwxWwW99q/

Dan Streetman (ddstreet)
description: updated
Changed in systemd (Ubuntu Bionic):
assignee: nobody → Dan Streetman (ddstreet)
importance: Undecided → Low
status: New → In Progress
Revision history for this message
Dan Streetman (ddstreet) wrote :

> https://github.com/systemd/systemd/commit/7d0b26a027118ca063780421cb31c74e9d2664ee

this is fine to include, it's clearly correct (I checked both the upstream kernel as well as wireguard-linux-compat from the ppa, and the param is u16 in both places)

I assume you can easily reproduce this, just by configuring a non-zero WireGuardPeer.PersistentKeepalive value? If so, can you update the SRU template [test case] section? That will need to be filled out before the patch can be sru'ed.

> https://github.com/systemd/systemd/commit/624a47694cad4c87b2e807c32db656f3e9d679c5

I don't think this is needed, it only changes use of sd_netlink_message_append_data() to sd_netlink_message_append_sockaddr_in() - or _in6() - which aren't included in the bionic version of systemd, and from looking at their upstream addition they don't seem to functionally change anything, just add more checking (and simplify the call params). Unless you can point out what the patch specifically fixes, I think we can leave this patch out.

Revision history for this message
Simon Déziel (sdeziel) wrote :

Steps to reproduce:

lxc launch images:ubuntu/bionic --vm -c security.secureboot=false foo
sleep 10 # allow booting
lxc exec foo -- apt install -y software-properties-common
lxc exec foo -- add-apt-repository -y ppa:wireguard/wireguard
lxc exec foo -- apt install -y wireguard-tools

cat << EOF | lxc exec foo -- tee /etc/systemd/network/wg0.netdev
# foo
[NetDev]
Name=wg0
Kind=wireguard

[WireGuard]
ListenPort=5555
PrivateKey=cBkljQSKhtEe/U8GZmCAk2MBbKWL4TLC9PVtbMFyCVQ=

[WireGuardPeer]
PublicKey=emfIuZ3hZ+AnWIrKex/EqCp2mfzip8AxJu6RuweyRGc=
AllowedIPs=192.168.255.2
Endpoint=bar.lxd:5555
EOF

cat << EOF | lxc exec foo -- tee /etc/systemd/network/wg0.network
# foo
[Match]
Name=wg0

[Network]
Address=192.168.255.1/24
EOF

lxc exec foo -- systemctl restart systemd-networkd

lxc launch images:ubuntu/bionic --vm -c security.secureboot=false bar
sleep 10 # allow booting
lxc exec bar -- apt install -y software-properties-common
lxc exec bar -- add-apt-repository -y ppa:wireguard/wireguard
lxc exec bar -- apt install -y wireguard-tools

cat << EOF | lxc exec bar -- tee /etc/systemd/network/wg0.netdev
# bar
[NetDev]
Name=wg0
Kind=wireguard

[WireGuard]
ListenPort=5555
PrivateKey=AHNwUJjVO939UYnp+SjrxYDa1ZlU1uIToCF9CHUitXE=

[WireGuardPeer]
PublicKey=7TJBZdnkY8zMRVPACZSxT6xL2pAi7/IL4R1DGeThEhY=
AllowedIPs=192.168.255.1
Endpoint=foo.lxd:5555
EOF

cat << EOF | lxc exec bar -- tee /etc/systemd/network/wg0.network
# bar
[Match]
Name=wg0

[Network]
Address=192.168.255.2/24
EOF

lxc exec bar -- systemctl restart systemd-networkd

# test connectivity
lxc exec foo -- ping -qc2 192.168.255.2

# notice the invalid length in dmesg
$ lxc exec foo -- journalctl -kn 8
-- Logs begin at Mon 2020-05-11 16:56:40 UTC, end at Mon 2020-05-11 17:03:46 UTC. --
May 11 16:58:25 foo kernel: nf_tables: (c) 2007-2009 Patrick McHardy <email address hidden>
May 11 17:01:57 foo kernel: PKCS#7 signature not signed with a trusted key
May 11 17:01:57 foo kernel: wireguard: module verification failed: signature and/or required key missing - tainting kernel
May 11 17:01:57 foo kernel: wireguard: WireGuard 1.0.20200429 loaded. See www.wireguard.com for information.
May 11 17:01:57 foo kernel: wireguard: Copyright (C) 2015-2019 Jason A. Donenfeld <email address hidden>. All Rights Reserved.
May 11 17:01:57 foo kernel: netlink: 'systemd-network': attribute type 5 has an invalid length.
May 11 17:01:57 foo kernel: netlink: 'systemd-network': attribute type 5 has an invalid length.
May 11 17:02:23 foo kernel: netlink: 'systemd-network': attribute type 5 has an invalid length.

Revision history for this message
Simon Déziel (sdeziel) wrote :

@ddstreet, PersistentKeepalive is not needed as you'll see in the steps to reproduce.

description: updated
Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Simon, or anyone else affected,

Accepted systemd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/237-3ubuntu10.41 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in systemd (Ubuntu Bionic):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-bionic
Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (systemd/237-3ubuntu10.41)

All autopkgtests for the newly accepted systemd (237-3ubuntu10.41) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

python-dbusmock/unknown (armhf)
policykit-1/unknown (armhf)
multipath-tools/unknown (armhf)
debci/unknown (ppc64el)
netplan.io/0.99-0ubuntu3~18.04.1 (i386)
pdns-recursor/unknown (armhf)
umockdev/0.11.1-1 (armhf)
sssd/unknown (armhf)
linux-raspi2-5.3/unknown (armhf)
suricata/unknown (armhf)
lxc/unknown (armhf)
casync/2+61.20180112-1 (s390x)
openssh/1:7.6p1-4ubuntu0.3 (arm64, s390x, amd64, i386, ppc64el, armhf)
python-systemd/unknown (armhf)
puppet/unknown (armhf)
prometheus-postgres-exporter/unknown (armhf)
lxc/3.0.3-0ubuntu1~18.04.1 (arm64)
postgresql-10/unknown (armhf)
polkit-qt-1/unknown (armhf)
munin/unknown (armhf)
systemd/237-3ubuntu10.41 (armhf)
pulseaudio/unknown (armhf)
php7.2/unknown (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Simon Déziel (sdeziel) wrote :

It tested fine with 237-3ubuntu10.41:

(Reading database ... 54686 files and directories currently installed.)
Preparing to unpack libnss-systemd_237-3ubuntu10.41_amd64.deb ...
Unpacking libnss-systemd:amd64 (237-3ubuntu10.41) over (237-3ubuntu10.40) ...
Preparing to unpack libpam-systemd_237-3ubuntu10.41_amd64.deb ...
Unpacking libpam-systemd:amd64 (237-3ubuntu10.41) over (237-3ubuntu10.40) ...
Preparing to unpack libsystemd0_237-3ubuntu10.41_amd64.deb ...
Unpacking libsystemd0:amd64 (237-3ubuntu10.41) over (237-3ubuntu10.40) ...
Preparing to unpack libudev1_237-3ubuntu10.41_amd64.deb ...
Unpacking libudev1:amd64 (237-3ubuntu10.41) over (237-3ubuntu10.40) ...
Preparing to unpack systemd_237-3ubuntu10.41_amd64.deb ...
Unpacking systemd (237-3ubuntu10.41) over (237-3ubuntu10.40) ...
Preparing to unpack systemd-sysv_237-3ubuntu10.41_amd64.deb ...
Unpacking systemd-sysv (237-3ubuntu10.41) over (237-3ubuntu10.40) ...
Preparing to unpack udev_237-3ubuntu10.41_amd64.deb ...
Unpacking udev (237-3ubuntu10.41) over (237-3ubuntu10.40) ...
Setting up libsystemd0:amd64 (237-3ubuntu10.41) ...
Setting up libudev1:amd64 (237-3ubuntu10.41) ...
Setting up systemd (237-3ubuntu10.41) ...
Setting up systemd-sysv (237-3ubuntu10.41) ...
Setting up udev (237-3ubuntu10.41) ...

tags: added: verification-done verification-done-bionic
removed: verification-needed verification-needed-bionic
Revision history for this message
Łukasz Zemczak (sil2100) wrote :

This is fine right now, but please be sure to be a bit more verbose about what kind of testing has been performed on the selected package!

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package systemd - 237-3ubuntu10.41

---------------
systemd (237-3ubuntu10.41) bionic; urgency=medium

  [ Dan Streetman ]
  * d/p/lp1867375/0001-network-Allow-to-configure-GW-even-UseRoutes-false.patch,
    d/p/lp1867375/0002-network-add-a-flag-to-ignore-gateway-provided-by-DHC.patch,
    d/p/lp1867375/0003-network-change-UseGateway-default-to-UseRoutes-setti.patch:
    - Move gateway ignoring from UseRoutes= to UseGateway= (LP: #1867375)
   * d/p/lp1873607/0002-core-make-sure-to-restore-the-control-command-id-too.patch:
     - Avoid segfault during serialization (LP: #1873607)
   * d/p/lp1529152/0001-bash-completion-systemctl-use-systemctl-no-pager.patch,
     d/p/lp1529152/0002-bash-completion-systemctl-pass-current-partial-unit-.patch,
     d/p/lp1529152/0003-shell-completion-systemctl-pass-current-word-to-all-.patch,
     d/p/lp1529152/0004-bash-completion-systemctl-re-implement-__filter_unit.patch,
     d/p/lp1529152/0005-strip-value-from-property-names.patch:
     - fix slow systemctl tab completion (LP: #1529152)
   * d/p/lp1877159-networkd-fix-attribute-length-for-wireguard-10380.patch:
     - avoid kernel err msg setting wireguard param (LP: #1877159)

  [ Heitor Alves de Siqueira ]
  * d/p/lp1876600-sd-bus-deal-with-cookie-overruns.patch:
    - deal with dbus cookie overruns (LP: #1876600)

 -- Heitor Alves de Siqueira <email address hidden> Sun, 03 May 2020 11:30:25 +0000

Changed in systemd (Ubuntu Bionic):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for systemd has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Simon Déziel (sdeziel) wrote : Re: [Bug 1877159] Re: netlink: 'systemd-network': attribute type 5 has an invalid length.

On 2020-05-25 4:17 a.m., Łukasz Zemczak wrote:
> This is fine right now, but please be sure to be a bit more verbose
> about what kind of testing has been performed on the selected package!

I went through the [test case] steps before and after the -proposed
update. Should I simply explicitly mention me going trough it or do I
need to copy it again?

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.