systemd segv coredump, reboot/poweroff ignored.

crash log

This has been happening perhaps 3 times every 7 days, no obvious cause I can see. The system is a virtual machine that's just a router. No user activity, nobody logged in to the mate gui. Wish I had more to offer by way of a clue but that's it.

Are you able to manually trigger the segfault by running 'sudo systemctl daemon-reload' ?

Just tried it three times, all normal, nothing in the logs except a few netfilter reports of correctly dropped packets.

Status changed to 'Confirmed' because the bug affects multiple users.

same here but on a PC

kernel: [58217.847336] systemctl[2787]: segfault at 0 ip 0000000000000000 sp 00007ffc4b4f9df8 error 14 in systemctl[55a481039000+10000]

crash log

And again:
root@noc1:~#
Broadcast message from <email address hidden> (Tue 2020-04-28 12:32:15 CDT):

systemd[1]: Caught <SEGV>, dumped core as pid 25084.

Broadcast message from <email address hidden> (Tue 2020-04-28 12:32:15 CDT):

systemd[1]: Freezing execution.

root@noc1:~#

from the syslog:

Apr 28 12:31:46 noc1 rssmonitor.py[18369]: rssmain: Running: systemctl daemon-reload
Apr 28 12:31:46 noc1 systemd[1]: /lib/systemd/system/dbus.socket:4: ListenStream= references a path below legacy directory /var/run/, updating /var/run/dbus/system_bus_socket → /run/dbus/system_bus_socket; please update the unit file accordingly.
Apr 28 12:31:46 noc1 systemd[1]: /lib/systemd/system/virtlockd.socket:5: ListenStream= references a path below legacy directory /var/run/, updating /var/run/libvirt/virtlockd-sock → /run/libvirt/virtlockd-sock; please update the unit file accordingly.
Apr 28 12:31:46 noc1 systemd[1]: /lib/systemd/system/virtlockd-admin.socket:5: ListenStream= references a path below legacy directory /var/run/, updating /var/run/libvirt/virtlockd-admin-sock → /run/libvirt/virtlockd-admin-sock; please update the unit file accordingly.
Apr 28 12:31:46 noc1 systemd[1]: /lib/systemd/system/virtlogd.socket:5: ListenStream= references a path below legacy directory /var/run/, updating /var/run/libvirt/virtlogd-sock → /run/libvirt/virtlogd-sock; please update the unit file accordingly.
Apr 28 12:31:46 noc1 systemd[1]: /lib/systemd/system/virtlogd-admin.socket:5: ListenStream= references a path below legacy directory /var/run/, updating /var/run/libvirt/virtlogd-admin-sock → /run/libvirt/virtlogd-admin-sock; please update the unit file accordingly.
Apr 28 12:31:46 noc1 systemd[1]: /lib/systemd/system/fail2ban.service:12: PIDFile= references a path below legacy directory /var/run/, updating /var/run/fail2ban/fail2ban.pid → /run/fail2ban/fail2ban.pid; please update the unit file accordingly.
Apr 28 12:31:46 noc1 systemd[1]: Reloading.
Apr 28 12:31:46 noc1 kernel: [37890.433153] systemd[1]: segfault at 50 ip 000055c2dd32d6f0 sp 00007ffef173c7b0 error 4 in systemd[55c2dd2d4000+ae000]
Apr 28 12:31:46 noc1 kernel: [37890.433172] Code: a8 48 8b 75 a0 c7 45 ac 00 00 00 00 48 8b 94 c7 a0 04 00 00 48 89 45 88 48 89 f0 48 39 d6 74 17 66 2e 0f 1f 84 00 00 00 00 00 <48> 8b 40 50 83 45 ac 01 48 39 c2 75 f3 48 8b 45 a0 31 db 4c 8d 7d
Apr 28 12:31:47 noc1 rssmonitor.py[18369]: rssmain: Ran OK: systemctl daemon-reload

The last entry means the systemctl daemon-reload command returned a normal 0 exit code.

Fixed in ubuntu/245.5-2ubuntu1 in Groovy.

Hello Harry, or anyone else affected,

Accepted systemd into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/245.4-4ubuntu3.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted systemd (245.4-4ubuntu3.1) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

hddemux/unknown (armhf)
pyudev/unknown (armhf)
python-uinput/unknown (armhf)
prometheus/2.15.2+ds-2 (armhf, amd64, arm64, ppc64el, s390x)
gvfs/unknown (armhf)
indicator-session/unknown (armhf)
gpsd/unknown (armhf)
samba/unknown (armhf)
python-systemd/unknown (armhf)
golang-github-coreos-go-systemd/unknown (armhf)
pystemd/unknown (armhf)
umockdev/0.14.1-1 (armhf)
remctl/unknown (armhf)
icinga2/unknown (armhf)
gnome-desktop3/unknown (armhf)
munin/2.0.56-1ubuntu1 (arm64)
haproxy/unknown (armhf)
python-dbusmock/unknown (armhf)
gvfs/1.44.1-1ubuntu1 (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

@hcoin are you still on Eoan or have you moved up to Focal yet? If you're on Focal, could you test the version from -proposed?

Still on eoan.  Staying with Ubuntu depends on freeipa support.
Presently only the freeipa client is supported past eoan, so the future
is unclear.  Can the fix be backported?

On 5/13/20 3:03 PM, Dan Streetman wrote:
> @hcoin are you still on Eoan or have you moved up to Focal yet? If
> you're on Focal, could you test the version from -proposed?
>

> Still on eoan. Staying with Ubuntu depends on freeipa support.
> Presently only the freeipa client is supported past eoan, so the future
> is unclear.

oh my, i wasn't aware of bug 1875114. that does not seem good at all.

> Can the fix be backported?

yep, it's in the upload queue for eoan, so should make it into eoan-proposed sometime soon.

Hello Harry, or anyone else affected,

Accepted systemd into eoan-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/242-7ubuntu3.9 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-eoan to verification-done-eoan. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-eoan. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Harry, or anyone else affected,

Accepted systemd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/237-3ubuntu10.41 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Trying it now on 1 of 4 previously affected systems.  Accepted all
'proposed' items, including changes to libvirt.  Will advise.

On 5/14/20 3:19 PM, Brian Murray wrote:
> Hello Harry, or anyone else affected,
>
> Accepted systemd into eoan-proposed. The package will build now and be
> available at https://launchpad.net/ubuntu/+source/systemd/242-7ubuntu3.9
> in a few hours, and then in the -proposed repository.
>
> Please help us by testing this new package. See
> https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how
> to enable and use -proposed. Your feedback will aid us getting this
> update out to other Ubuntu users.
>
> If this package fixes the bug for you, please add a comment to this bug,
> mentioning the version of the package you tested, what testing has been
> performed on the package and change the tag from verification-needed-
> eoan to verification-done-eoan. If it does not fix the bug for you,
> please add a comment stating that, and change the tag to verification-
> failed-eoan. In either case, without details of your testing we will not
> be able to proceed.
>
> Further information regarding the verification process can be found at
> https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in
> advance for helping!
>
> N.B. The updated package will be released to -updates after the bug(s)
> fixed by this package have been verified and the package has been in
> -proposed for a minimum of 7 days.
>
> ** Changed in: systemd (Ubuntu Eoan)
> Status: In Progress => Fix Committed
>
> ** Tags added: verification-needed-eoan
>

All autopkgtests for the newly accepted systemd (242-7ubuntu3.9) for eoan have finished running.
The following regressions have been reported in tests triggered by the package:

nghttp2/unknown (armhf)
pulseaudio/unknown (armhf)
docker.io/unknown (armhf)
csync2/unknown (armhf)
network-manager/unknown (armhf)
dovecot/unknown (armhf)
open-iscsi/unknown (armhf)
netplan.io/unknown (armhf)
asterisk/1:16.2.1~dfsg-2build2 (arm64)
tinyssh/unknown (armhf)
puppet/unknown (armhf)
corosync/unknown (armhf)
python-uinput/unknown (armhf)
nextepc/unknown (armhf)
debci/unknown (armhf)
pyudev/unknown (armhf)
cups/unknown (armhf)
umockdev/0.13.2-1 (armhf)
cockpit/unknown (ppc64el)
network-manager/1.20.4-2ubuntu2.2 (arm64)
nftables/unknown (armhf)
prometheus/unknown (armhf)
corosync-qdevice/unknown (armhf)
prometheus-bind-exporter/unknown (armhf)
python-systemd/unknown (armhf)
prometheus-alertmanager/unknown (armhf)
ndctl/unknown (armhf)
prometheus-apache-exporter/unknown (armhf)
dpdk/unknown (armhf)
nextepc/0.3.10+nods-3ubuntu1 (ppc64el)
dbus/unknown (armhf)
openssh/1:8.0p1-6build1 (armhf, amd64, i386, ppc64el, arm64, s390x)
openrazer/unknown (armhf)
gvfs/1.42.1-1ubuntu1 (arm64)
snapd/2.42.1+19.10 (i386)
prometheus-node-exporter/unknown (armhf)
prometheus-postgres-exporter/unknown (armhf)
python-dbusmock/unknown (armhf)
systemd/242-7ubuntu3.9 (ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/eoan/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

All autopkgtests for the newly accepted systemd (237-3ubuntu10.41) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

python-dbusmock/unknown (armhf)
policykit-1/unknown (armhf)
multipath-tools/unknown (armhf)
debci/unknown (ppc64el)
netplan.io/0.99-0ubuntu3~18.04.1 (i386)
pdns-recursor/unknown (armhf)
umockdev/0.11.1-1 (armhf)
sssd/unknown (armhf)
linux-raspi2-5.3/unknown (armhf)
suricata/unknown (armhf)
lxc/unknown (armhf)
casync/2+61.20180112-1 (s390x)
openssh/1:7.6p1-4ubuntu0.3 (arm64, s390x, amd64, i386, ppc64el, armhf)
python-systemd/unknown (armhf)
puppet/unknown (armhf)
prometheus-postgres-exporter/unknown (armhf)
lxc/3.0.3-0ubuntu1~18.04.1 (arm64)
postgresql-10/unknown (armhf)
polkit-qt-1/unknown (armhf)
munin/unknown (armhf)
systemd/237-3ubuntu10.41 (armhf)
pulseaudio/unknown (armhf)
php7.2/unknown (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

@hcoin I assume no more segfaults with the updated pkg? can we mark this verified for eoan?

No issues so far. Thanks for checking.

On May 20, 2020 5:30:40 PM CDT, Dan Streetman <email address hidden> wrote:
>@hcoin I assume no more segfaults with the updated pkg? can we mark
>this verified for eoan?
>
>--
>You received this bug notification because you are subscribed to the
>bug
>report.
>https://bugs.launchpad.net/bugs/1873607
>
>Title:
> systemd segv coredump, reboot/poweroff ignored.
>
>Status in systemd:
> Fix Released
>Status in systemd package in Ubuntu:
> Fix Released
>Status in systemd source package in Bionic:
> Fix Committed
>Status in systemd source package in Eoan:
> Fix Committed
>Status in systemd source package in Focal:
> Fix Committed
>Status in systemd source package in Groovy:
> Fix Released
>
>Bug description:
> [impact]
>
> during systemctl daemon-reload, systemd sometimes segfaults while
> serializing a service.
>
> [test case]
>
> problem is intermittent, but the upstream bug suggests a reproducer.
> Also see original description.
>
> [regression potential]
>
> this adjusts serialization, so any regression would likely occur
> during serialization/deserialization, e.g. at daemon-reload.
>
> [scope]
>
> This is needed for Bionic and later.
>
> this was introduced by commit e266c068b5597e18b2299f9c9d3ee6cf04198c41
> which is included starting in v234. The commit to fix it is
> e9da62b18af647bfa73807e1c7fc3bfa4bb4b2ac which is not yet included in
> any release.
>
> The PR to fix this is https://github.com/systemd/systemd/pull/15546
>
> [other info]
>
>there is an additional PR that further avoids the error, but has not
>yet been accepted, and may not be accepted as it may not be necessary.
> https://github.com/systemd/systemd/pull/15370
>
> [original description]
>
> Proximate syslog:
>
>Apr 18 16:26:56 gate2 rssmonitor.py[2001]: settings: Running: ip -all
>netns del
>Apr 18 16:26:56 gate2 systemd[1]: /lib/systemd/system/dbus.socket:4:
>ListenStream= references a path below legacy directory /var/run/,
>updating /var/run/dbus/system_bus_socket → /r
> un/dbus/system_bus_socket; please update the unit file accordingly.
>Apr 18 16:26:56 gate2 systemd[1]: /lib/systemd/system/sssd.service:11:
>PIDFile= references a path below legacy directory /var/run/, updating
>/var/run/sssd.pid → /run/sssd.pid; plea
> se update the unit file accordingly.
>Apr 18 16:26:56 gate2 systemd[1]:
>/lib/systemd/system/oddjobd.service:6: PIDFile= references a path below
>legacy directory /var/run/, updating /var/run/oddjobd.pid →
>/run/oddjobd.p
> id; please update the unit file accordingly.
>Apr 18 16:26:56 gate2 systemd[1]:
>/lib/systemd/system/fail2ban.service:12: PIDFile= references a path
>below legacy directory /var/run/, updating
>/var/run/fail2ban/fail2ban.pid → /r
> un/fail2ban/fail2ban.pid; please update the unit file accordingly.
>Apr 18 16:26:56 gate2 systemd[1]:
>/lib/systemd/system/certmonger.service:6: PIDFile= references a path
>below legacy directory /var/run/, updating /var/run/certmonger.pid →
>/run/cer
> tmonger.pid; please update the unit file accordingly.
> Apr 18 16:26:56 gate2 systemd[1]: Reloading.
>Apr 18 16:26:56 gate2 kernel: [ 454.966336] systemd[1]: segfault at 50
>ip ...

I looped daemon-reexec and daemon-reload for a while on both bionic and focal and saw no segfaults with the updated packages. As @hcoin was able to reproduce this on Eoan and the updated pkg appears to have fixed it there, and I was unable to reproduce any segfaults or other issues, I'm marking as verified for b and f also.

The verification of the Stable Release Update for systemd has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package systemd - 245.4-4ubuntu3.1

---------------
systemd (245.4-4ubuntu3.1) focal; urgency=medium

  * d/p/lp1867375/0001-network-add-a-flag-to-ignore-gateway-provided-by-DHC.patch,
    d/p/lp1867375/0002-test-network-add-a-test-case-for-DHCPv4.UseGateway-n.patch,
    d/p/lp1867375/0003-network-change-UseGateway-default-to-UseRoutes-setti.patch,
    d/p/lp1867375/0004-test-modify-add-tests-for-UseRoutes-and-UseGateway-c.patch,
    d/p/lp1867375/0005-network-honor-SetDNSRoutes-even-if-UseGateway-False.patch,
    d/p/lp1867375/0006-test-verify-RoutesToDNS-is-independent-of-UseGateway.patch:
    - Add UseGateway= parameter and default to value of UseRoutes, to restore
      backwards compatibility with old UseRoutes= behavior (LP: #1867375)
  * d/p/lp1860926-network-Change-IgnoreCarrierLoss-default-to-value-of.patch:
    - default ignore_carrier_loss to value of configure_without_carrier,
      so carrier drop during configuration doesn't break networking
      (LP: #1860926)
   * d/e/initramfs-tools/hooks/udev:
     - Follow symlinks when finding link files to copy into initramfs
       (LP: #1868892)
   * d/p/lp1873607/0001-core-some-minor-clean-ups-modernizations.patch,
     d/p/lp1873607/0002-core-make-sure-to-restore-the-control-command-id-too.patch:
     - Avoid segfault during serialization (LP: #1873607)

 -- Dan Streetman <email address hidden> Thu, 07 May 2020 09:21:22 -0400

This bug was fixed in the package systemd - 242-7ubuntu3.9

---------------
systemd (242-7ubuntu3.9) eoan; urgency=medium

  * d/p/lp1867375/0001-network-Allow-to-configure-GW-even-UseRoutes-false.patch,
    d/p/lp1867375/0002-network-add-a-flag-to-ignore-gateway-provided-by-DHC.patch,
    d/p/lp1867375/0003-network-change-UseGateway-default-to-UseRoutes-setti.patch,
    d/p/lp1867375/0004-network-honor-SetDNSRoutes-even-if-UseGateway-False.patch:
    - Move gateway ignoring from UseRoutes= to UseGateway= (LP: #1867375)
   * d/p/lp1873607/0001-core-some-minor-clean-ups-modernizations.patch,
     d/p/lp1873607/0002-core-make-sure-to-restore-the-control-command-id-too.patch:
     - Avoid segfault during serialization (LP: #1873607)
   * d/p/lp1877271-network-drop-all-checks-of-ipv6_disabled-sysctl.patch:
     - enable ipv6 for interface when needed (LP: #1877271)
   * d/p/lp1860926-network-Change-IgnoreCarrierLoss-default-to-value-of.patch:
     - IgnoreCarrierLoss default to ConfigureWithoutCarrier (LP: #1860926)

 -- Dan Streetman <email address hidden> Fri, 08 May 2020 12:28:53 -0400

This bug was fixed in the package systemd - 237-3ubuntu10.41

---------------
systemd (237-3ubuntu10.41) bionic; urgency=medium

  [ Dan Streetman ]
  * d/p/lp1867375/0001-network-Allow-to-configure-GW-even-UseRoutes-false.patch,
    d/p/lp1867375/0002-network-add-a-flag-to-ignore-gateway-provided-by-DHC.patch,
    d/p/lp1867375/0003-network-change-UseGateway-default-to-UseRoutes-setti.patch:
    - Move gateway ignoring from UseRoutes= to UseGateway= (LP: #1867375)
   * d/p/lp1873607/0002-core-make-sure-to-restore-the-control-command-id-too.patch:
     - Avoid segfault during serialization (LP: #1873607)
   * d/p/lp1529152/0001-bash-completion-systemctl-use-systemctl-no-pager.patch,
     d/p/lp1529152/0002-bash-completion-systemctl-pass-current-partial-unit-.patch,
     d/p/lp1529152/0003-shell-completion-systemctl-pass-current-word-to-all-.patch,
     d/p/lp1529152/0004-bash-completion-systemctl-re-implement-__filter_unit.patch,
     d/p/lp1529152/0005-strip-value-from-property-names.patch:
     - fix slow systemctl tab completion (LP: #1529152)
   * d/p/lp1877159-networkd-fix-attribute-length-for-wireguard-10380.patch:
     - avoid kernel err msg setting wireguard param (LP: #1877159)

  [ Heitor Alves de Siqueira ]
  * d/p/lp1876600-sd-bus-deal-with-cookie-overruns.patch:
    - deal with dbus cookie overruns (LP: #1876600)

 -- Heitor Alves de Siqueira <email address hidden> Sun, 03 May 2020 11:30:25 +0000