systemd-logind crashed with SIGSEGV

StacktraceTop:
 logind_wall_tty_filter ()
 utmp_wall.constprop ()
 warn_wall.lto_priv ()
 method_schedule_shutdown.lto_priv ()
 object_find_and_run.lto_priv ()

Status changed to 'Confirmed' because the bug affects multiple users.

I'm on a xenial ubuntu-desktop and have snappy dimension on classic installed. I was working on other things and apport popped up. The trigger seems to be the background snappy update task for ubuntu-core that triggers a reboot of ubuntu-core. Looking at my logs I see:

Mar 21 10:00:30 <hostname> systemd[1]: Started Ubuntu Core Snappy AutoUpdate.
Mar 21 10:01:02 <hostname> systemd[1]: Mounting Squashfs mount unit for ubuntu-core...
Mar 21 10:01:02 <hostname> systemd[1]: Mounted Squashfs mount unit for ubuntu-core.
Mar 21 10:01:03 <hostname> snappy[25381]: Name Date Version Developer
Mar 21 10:01:03 <hostname> snappy[25381]: ubuntu-core 2016-03-21 16.04-20160321-05-01 canonical!
Mar 21 10:01:03 <hostname> snappy[25381]: Reboot to use ubuntu-core version 16.04-20160321-05-01.
Mar 21 10:01:03 <hostname> snappy[25381]: Rebooting to satisfy updates for ubuntu-core
Mar 21 10:01:03 <hostname> kernel: [85783.825747] systemd-logind[1101]: segfault at 0 ip 00007fc5cafcfd9e sp 00007ffc1cf 580b8 error 4 in libc-2.21.so[7fc5caf30000+1c0000]
Mar 21 10:01:03 <hostname> systemd[1]: systemd-logind.service: Main process exited, code=dumped, status=11/SEGV
Mar 21 10:01:03 <hostname> systemd[1]: systemd-logind.service: Unit entered failed state.
Mar 21 10:01:03 <hostname> /usr/bin/snappy[25381]: main.go:50: DEBUG: [/usr/bin/snappy update --automatic-reboot] failed : failed to auto reboot: Failed to call ScheduleShutdown in logind, proceeding with immediate shutdown: Message re cipient disconnected from message bus without replying
Mar 21 10:01:03 <hostname> snappy[25381]: failed to auto reboot: Failed to call ScheduleShutdown in logind, proceeding w ith immediate shutdown: Message recipient disconnected from message bus without replying
Mar 21 10:01:03 <hostname> systemd[1]: systemd-logind.service: Failed with result 'core-dump'.
Mar 21 10:01:03 <hostname> systemd[1]: snappy-autopilot.service: Main process exited, code=exited, status=1/FAILURE
Mar 21 10:01:03 <hostname> systemd[1]: snappy-autopilot.service: Unit entered failed state.
Mar 21 10:01:03 <hostname> systemd[1]: snappy-autopilot.service: Failed with result 'exit-code'.
Mar 21 10:01:03 <hostname> systemd[1]: systemd-logind.service: Service has no hold-off time, scheduling restart.
Mar 21 10:01:03 <hostname> systemd[1]: Stopped Login Service.
Mar 21 10:01:03 <hostname> dbus[1163]: [system] Activating via systemd: service name='org.freedesktop.login1' unit='dbus -org.freedesktop.login1.service'
Mar 21 10:01:03 <hostname> systemd[1]: Starting Login Service...
Mar 21 10:01:03 <hostname> dbus[1163]: [system] Successfully activated service 'org.freedesktop.login1'
Mar 21 10:01:03 <hostname> systemd[1]: Started Login Service.
Mar 21 10:01:03 <hostname> systemd[1]: Started User Manager for UID 1000.
Mar 21 10:01:03 <hostname> systemd[1]: Started User Manager for UID 119.

This appears to be happening on the hour. I imagine a workaround is to uninstall ubuntu-core, then install it again. /me goes to do that now.

Can't currently 'snappy remove ubuntu-core'. I'm going to hack on /lib/systemd/system/snappy-autopilot.timer as a workaround.

Unfortunately the stack trace isn't particularly useful. If you can reproduce this, it would be helpful if you could install systemd-dbg and look at the generated stack trace then?

This crashes in:

bool logind_wall_tty_filter(const char *tty, void *userdata) {

        Manager *m = userdata;

        assert(m);

        if (!startswith(tty, "/dev/"))
                return true;

        return !streq(tty + 5, m->scheduled_shutdown_tty);
}

startswith() does a strlen() and a strncmp(), so I don't think it's that as it would already segfault in strlen(). m is also not NULL (due to the assert), so I figure it's the m->scheduled_shutdown_tty that is NULL.

I'll dig through the code to see where this is being set.

I tried to reproduce this with "nohup shutdown" and similar, but wasn't able to. However, I do understand what's going on.

Upstream fix sent: https://github.com/systemd/systemd/pull/2887

My fix landed upstream, I cherry-picked it into our packaging git.

This bug was fixed in the package systemd - 229-3ubuntu2

---------------
systemd (229-3ubuntu2) xenial; urgency=medium

  * udev.postinst: Quiesce error message if /sys/class/net/eno* does not
    exist. (LP: #1560112)
  * Fix assertion crash when processing a (broken) device without a sysfs
    path. (Closes: #819290, LP: #1560695)
  * Fix crash when shutdown is issued from a non-tty. (LP: #1553040)
  * networkd: Stay running while any non-loopback interface is up.
    (Closes: #819414)
  * udev: Don't kill peer processes if we don't run in a cgroup. This happens
    when running under/upgrading from upstart. udevd and everything
    else run in the root hierarchy on all controllers then, and the alleged
    cleanup of "our" cgroup becomes a system-wide killing spree.
    (LP: #1555237)

 -- Martin Pitt <email address hidden> Thu, 31 Mar 2016 23:11:19 +0200