boot scripts hang in plymouth --ping (aka: Plymouth's bug returns)

Hi,

Summary:
========

My system is not able to boot unattended anymore.

Problem seems similar to https://bugs.launchpad.net/ubuntu/+source/plymouth/+bug/564360

History:
========

The history of this issue is the following:

1- On 2016.06.14, I "aptitude safe-upgrade" my system. The kernel upgraded from 3.13.0-85-generic to 3.13.0-88-generic. After the upgrade, the system won't boot anymore. After some troubleshooting, suspecting some kernel/low-level issue, I decided to reboot it on kernel 3.13.0-85-generic, and systems boots OK.
2- on 2016.07.26, I "aptitude safe-upgrade" my system. The kernel upgraded from 3.13.0-85-generic to 3.13.0-92-generic. After the upgrade, the system won't finish booting up. Fortunatelly, OpenSSH had started, and I could investigate the issue a little bit further. Hereunder are my findings about this issue.

Findings:
=========

On the console, the last boot message I can see is the following:

* Starting configure virtual network devices [ OK ]
[ 16.192284] bnx2 000:01:00.1 eth1 NIC Copper Link is Down

my /var/log/boot.log file is the following:

root@otrs-prod:/etc/init# cat /var/log/boot.log
 * Starting Read required files in advance [ OK ]
 * Starting Mount filesystems on boot [ OK ]
 * Starting Fix-up sensitive /proc filesystem entries [ OK ]
 * Starting Populate /dev filesystem [ OK ]
 * Starting Populate and link to /run filesystem [ OK ]
 * Stopping Populate /dev filesystem [ OK ]
 * Stopping Fix-up sensitive /proc filesystem entries [ OK ]
 * Stopping Populate and link to /run filesystem [ OK ]
 * Stopping Track if upstart is running in a container [ OK ]
 * Starting Signal sysvinit that the rootfs is mounted [ OK ]
 * Starting Clean /tmp directory [ OK ]
 * Stopping Read required files in advance (for other mountpoints) [ OK ]
 * Stopping Read required files in advance (for other mountpoints) [ OK ]
 * Stopping Clean /tmp directory [ OK ]
 * Starting Initialize or finalize resolvconf [ OK ]
 * Starting set console keymap [ OK ]
 * Starting Signal sysvinit that virtual filesystems are mounted [ OK ]
 * Starting Signal sysvinit that virtual filesystems are mounted [ OK ]
 * Starting set sysctls from /etc/sysctl.conf [ OK ]
 * Starting Bridge udev events into upstart [ OK ]
 * Starting Signal sysvinit that local filesystems are mounted [ OK ]
 * Starting Signal sysvinit that remote filesystems are mounted [ OK ]
 * Starting device node and kernel event manager [ OK ]
 * Starting GNU Screen Cleanup [ OK ]
 * Starting Flush boot log to disk [ OK ]
 * Starting flush early job output to logs [ OK ]
 * Stopping Mount filesystems on boot [ OK ]
 * Stopping set sysctls from /etc/sysctl.conf [ OK ]
 * Stopping set console keymap [ OK ]
 * Stopping Flush boot log to disk [ OK ]
 * Stopping flush early job output to logs [ OK ]
 * Starting load modules from /etc/modules [ OK ]
 * Starting cold plug devices [ OK ]
 * Starting log initial device creation [ OK ]
 * Stopping GNU Screen Cleanup [ OK ]
 * Starting Bridge file events into upstart [ OK ]
 * Stopping load modules from /etc/modules [ OK ]
 * Starting system logging daemon [ OK ]
 * Starting D-Bus system message bus [ OK ]
 * Starting SystemD login management service [ OK ]
 * Stopping cold plug devices [ OK ]
 * Stopping log initial device creation [ OK ]
 * Starting configure network device security [ OK ]
 * Starting configure network device security [ OK ]
 * Starting configure network device security [ OK ]
 * Starting configure network device security [ OK ]
 * Starting configure network device security [ OK ]
 * Starting load fallback graphics devices [ OK ]
 * Starting configure network device security [ OK ]
 * Stopping load fallback graphics devices [ OK ]
 * Starting save udev log and update rules [ OK ]
 * Starting set console font [ OK ]
 * Stopping save udev log and update rules [ OK ]
 * Stopping set console font [ OK ]
 * Starting userspace bootsplash [ OK ]
 * Starting configure network device [ OK ]
 * Starting Send an event to indicate plymouth is up [ OK ]
 * Stopping Send an event to indicate plymouth is up [ OK ]
 * Starting configure network device security [ OK ]
 * Stopping userspace bootsplash [ OK ]
 * Starting Mount network filesystems [ OK ]
 * Stopping Mount network filesystems [ OK ]
 * Starting Mount network filesystems [ OK ]
 * Starting configure network device [ OK ]
 * Stopping Mount network filesystems [ OK ]
 * Starting Mount network filesystems [ OK ]
 * Starting configure network device [ OK ]
 * Starting System V initialisation compatibility [ OK ]
 * Stopping Mount network filesystems [ OK ]
 * Starting Uncomplicated firewall [ OK ]
Skipping profile in /etc/apparmor.d/disable: usr.sbin.rsyslogd
 * Starting configure network device [ OK ]
 * Starting Mount network filesystems [ OK ]
 * Starting Failsafe Boot Delay [ OK ]
 * Stopping Mount network filesystems [ OK ]
 * Starting Bridge socket events into upstart [ OK ]
 * Starting AppArmor profiles [ OK ]
 * Setting up X socket directories... [ OK ]
 * Stopping System V initialisation compatibility [ OK ]
 * Starting System V runlevel compatibility [ OK ]
 * Starting save kernel messages [ OK ]
 * Starting regular background program processing daemon [ OK ]
 * Starting deferred execution scheduler [ OK ]
 * Starting ACPI daemon [ OK ]
 * Starting automatic crash report generation [ OK ]
 * Stopping save kernel messages [ OK ]
 * Starting CPU interrupts balancing daemon [ OK ]
 * Starting OpenSSH server [ OK ]
Starting Systems Management Device Drivers:
Starting dell_rbu: *
Starting ipmi driver: * Already started
Starting Systems Management Data Engine:
Starting dsm_sa_datamgrd: * Starting MySQL Server [ OK ]
 *
Starting dsm_sa_eventmgrd: *
Starting dsm_sa_snmpd: *
/etc/rc2.d/S20dataeng: 566: /etc/rc2.d/S20dataeng: /opt/dell/srvadmin//lib64/srvadmin-deng/dcfwsnmp.sh: Permission denied

WARNING: The SNMP port appears to be closed by the firewall.
You may not be able to manage this system remotely using SNMP.
See Firewall Configuration in the product user's guide.
 * Starting Postfix Mail Transport Agent postfix [ OK ]
 * Starting ipmi drivers [ OK ]
 * Stopping OpenSSH server [ OK ]
 * Starting OpenSSH server [ OK ]
 * Starting Mount network filesystems [ OK ]
 * Stopping Mount network filesystems [ OK ]
 * Starting configure network device [ OK ]
 * Stopping OpenSSH server [ OK ]
 * Starting OpenSSH server [ OK ]
 * Starting Mount network filesystems [ OK ]
 * Starting configure network device [ OK ]
 * Stopping Mount network filesystems [ OK ]
 * Starting configure virtual network devices [ OK ]
root@otrs-prod:/etc/init#

When listing the processes running, I see some plymouth-related processes which seem hung:

...
root 2944 2715 0 08:34 ? 00:00:00 sshd: support [priv]
support 3021 2944 0 08:34 ? 00:00:00 sshd: support@pts/3
support 3022 3021 0 08:34 pts/3 00:00:00 -bash
otrs 3147 1 0 08:35 ? 00:00:01 /usr/bin/perl /opt/otrs//bin/otrs.Scheduler.pl -w 1
root 3153 1 0 08:35 ? 00:00:00 plymouth --ping
root 3406 2 0 08:39 ? 00:00:00 [kworker/u66:0]
root 3506 3022 0 08:42 pts/3 00:00:00 sudo su
root 3507 3506 0 08:42 pts/3 00:00:00 su
root 3508 3507 0 08:42 pts/3 00:00:00 bash
root 3540 1679 0 08:43 ? 00:00:00 /bin/sh /etc/rc2.d/S70dns-clean start
root 3651 3540 0 08:48 ? 00:00:00 plymouth --ping
postfix 3667 2458 0 08:49 ? 00:00:00 smtpd -n smtp -t inet -u -c -o stress= -s 2
postfix 3668 2458 0 08:49 ? 00:00:00 proxymap -t unix -u
postfix 3669 2458 0 08:49 ? 00:00:00 anvil -l -t unix -u -c
root 3673 1584 0 08:49 ? 00:00:00 /bin/plymouth message --text=Booting system without full network configuration...
root 3681 3508 0 08:49 pts/3 00:00:00 ps -eaf
root@otrs-prod:/home/support#

root@otrs-prod:/home/support# ps -eaf | grep ply
root 367 1 0 08:33 ? 00:00:00 @sbin/plymouthd --mode=boot --attach-to-session
root 368 1 0 08:33 ? 00:00:00 plymouth-upstart-bridge
root 3153 1 0 08:35 ? 00:00:00 plymouth --ping
root 3651 1 0 08:48 ? 00:00:00 plymouth --ping
root 3822 3815 0 08:52 ? 00:00:00 plymouth --ping
root 4042 4022 0 08:56 ? 00:00:00 plymouth --ping
root 4050 3508 0 08:56 pts/3 00:00:00 grep --color=auto ply
root@otrs-prod:/home/support

root@otrs-prod:/home/support# strace -p 367
Process 367 attached
write(12, "Waiting for network configuratio"..., 36

Workaround:
===========

After a few more troubleshooting, failing to understand what the problem is, I decided to kill a "plymouth --ping" process. Eventually, it allowed the system to boot a little further (ie: I can see in the list of processes that initd/systemd/whatever launched a new startup script. For example, in the processes list here above, the S70dns-clean startup script would have vanished and a new one, eg. S70ppd-dns, would have shown up), before it hung again on a "plymouth --ping" process.

I thus continued to kill those "plymouth --ping" each time allowing the system to boot a little further, until no "plymouth --ping" process would show up anymore and the system appeared to be completly started.

A that time, I also killed "plymouthd" for the sake of completeness.

Expected result:
================

The system should boot without human assistance.

Supportive information:
=======================

My system is a physicall server Dell PowerEdge R710 running Ubuntu 14.04 now running linux kernel 3.13.0-92-generic:

root@otrs-prod:/etc/init# lsb_release -rd
Description: Ubuntu 14.04.4 LTS
Release: 14.04

root@otrs-prod:/etc/init# uname -a
Linux otrs-prod 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
root@otrs-prod:/etc/init#

It has 3 physical ethernet interfaces, with 2 bounded together. Here's my (edited) interfaces file:

root@otrs-prod:/etc/init# cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto bond0
iface bond0 inet static
        address a.b.c.d
        netmask 255.255.255.224
        gateway a.b.c.1
        bond-slaves none
        bond_mode 1
        bond_miimon 100

iface bond0 inet6 static
        address a:b:c:d::e
        netmask 64
        gateway a:b:c:d::1
        # Impletement split access
        up ip -f inet6 rule add from a:b:c:d::e table local_v6
        up ip -f inet6 route add default via a:b:c:d::1 dev bond0 table local_v6

auto eth0
iface eth0 inet manual
        bond-master bond0
        bond_mode 1
        bond_miimon 100

auto eth1
iface eth1 inet manual
        bond-master bond0
        bond_mode 1
        bond_miimon 100

# The management interface
auto eth2
iface eth2 inet static
        address f.g.h.i
        netmask 255.255.0.0
        network f.g.0.0
        broadcast f.g.255.255
        # dns-* options are implemented by the resolvconf package, if installed
        dns-nameservers s.t.u.v w.x.y.z
        dns-search fw.belnet.be
        up route add -net f.0.0.0/8 gw f.g.0.1 dev eth2

iface eth2 inet6 static
        address a:b:c:f::g
        netmask 64
        up route -A inet6 add a:b:c:h::/56 gw a:b:c:f::1 dev eth2
root@otrs-prod:/etc/init#

Finally, the system runs the OpenManage stuff:

root@otrs-prod:/etc/init# dpkg -l "srvadmin*"
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-===============================-====================-====================-===================================================================
ii srvadmin-base 8.3.0 amd64 Meta package for installing the Server Agent
ii srvadmin-deng 8.3.0-1 amd64 Dell OpenManage Data Engine
ii srvadmin-deng-snmp 8.3.0-1 amd64 Dell OpenManage Data Engine SNMP
ii srvadmin-hapi 8.3.0-1 amd64 Dell OpenManage Hardware Abstraction Programming Interface
ii srvadmin-idrac-snmp 8.2.0-1 amd64 CLI utils from the management station to the iDRAC
ii srvadmin-isvc 8.3.0-1 amd64 Dell OpenManage Instrumentation Services
ii srvadmin-isvc-snmp 8.3.0-1 amd64 Dell OpenManage Instrumentation Services SNMP
ii srvadmin-nvme 8.3.0 amd64 This package provides the libaries to manage NVMe devices.
ii srvadmin-omacore 8.3.0-1 amd64 Server Administrator CLI
ii srvadmin-omacs 8.3.0-2 amd64 Dell OpenManage Server Administrator OMACS
ii srvadmin-omcommon 8.3.0-2 amd64 Dell OpenManage Server Administrator Common Framework
ii srvadmin-omilcore 8.3.0 all Dell OpenManage Server Administrator Install Core
ii srvadmin-ominst 8.3.0-1 amd64 OMINST
ii srvadmin-realssd 7.4.0-1 amd64 RealSSD package for storage management
ii srvadmin-rnasoap 7.4.0-1 amd64 Fluid Cache Management
ii srvadmin-server-cli 8.3.0 amd64 Meta package for installing the Server Administrator server cli com
ii srvadmin-server-snmp 8.3.0 amd64 Meta package for installing the Server Administrator snmp component
ii srvadmin-smcommon 8.3.0 amd64 Storage Management common files for GUI and CLI
ii srvadmin-storage 8.3.0 amd64 Storage Management accessors package
ii srvadmin-storage-cli 8.3.0 amd64 Storage Management cli component
un srvadmin-storage-populator <none> <none> (no description available)
ii srvadmin-storage-snmp 8.3.0 amd64 Storage Management SNMP component
ii srvadmin-storageservices 7.4.0 amd64 Meta package for installing the Server Administrator Storage Servic
ii srvadmin-storageservices-cli 7.4.0 amd64 Meta package for storageservices-cli
ii srvadmin-storageservices-snmp 7.4.0 amd64 Meta package for storageservices-snmp
ii srvadmin-storelib 8.3.0 amd64 StoreLib package for storage management
ii srvadmin-storelib-sysfs 8.3.0 amd64 Metapackage for libsysfs2
ii srvadmin-xmlsup 8.3.0-1 amd64 Dell OpenManage XML Support SDK
root@otrs-prod:/etc/init#

root@otrs-prod:/etc/init# cat /etc/apt/sources.list.d/openmanage.ubuntu.com.sources.list
deb http://linux.dell.com/repo/community/ubuntu trusty openmanage
root@otrs-prod:/etc/init#

Plymouth is at latest available version:

root@otrs-prod:/etc/init# dpkg -l "plymouth*"
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-===============================-====================-====================-===================================================================
ii plymouth 0.8.8-0ubuntu17.1 amd64 graphical boot animation and logger - main package
un plymouth-theme <none> <none> (no description available)
ii plymouth-theme-ubuntu-text 0.8.8-0ubuntu17.1 amd64 graphical boot animation and logger - ubuntu-logo theme
root@otrs-prod:/etc/init#