nginx systemd service fails to autostart at boot with libssl update

Asked by Dietmar May

After updating libssl and related packages, nginx will no longer autostart at system boot.

Immediately after boot, nginx.service is in a failed state.

# service nginx status
● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: failed (Result: timeout) since Fri 2018-08-24 21:27:51 UTC; 32min ago
     Docs: man:nginx(8)

systemd[1]: Starting A high performance web server and a reverse proxy server...
systemd[1]: nginx.service: Start-pre operation timed out. Terminating.
systemd[1]: nginx.service: Failed with result 'timeout'.
systemd[1]: Failed to start A high performance web server and a reverse proxy server.

The service can be manually started after boot.

# service nginx start
# service nginx status
● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2018-08-24 22:02:06 UTC; 2s ago
     Docs: man:nginx(8)
  Process: 2704 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
  Process: 2703 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
 Main PID: 2705 (nginx)
   CGroup: /system.slice/nginx.service
           ├─2705 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
           └─2706 nginx: worker process

systemd[1]: Starting A high performance web server and a reverse proxy server...
systemd[1]: nginx.service: Failed to parse PID from file /run/nginx.pid: Invalid argument
systemd[1]: Started A high performance web server and a reverse proxy server.

This happens on an ARMHF based microcontroller running ubuntu 18.04.2 raspi server distribution with a stock kernel.org 4.9-181 kernel.

Ubuntu repositories are not accessible from the device, so packages are copied to the device, and apt install is used to upgrade them:

apt install --no-install-recommends $dir/updates/system/*.deb | logger 2>&1

The following is a list of packages that, when upgraded, cause the nginx systemd service to fail to autostart at boot.

201,205c201,205
< ii libpython2.7:armhf 2.7.15-4ubuntu4~18.04 armhf Shared Python runtime library (version 2.7)
< ii libpython2.7-minimal:armhf 2.7.15-4ubuntu4~18.04 armhf Minimal subset of the Python language (version 2.7)
< ii libpython2.7-stdlib:armhf 2.7.15-4ubuntu4~18.04 armhf Interactive high-level object-oriented language (standard library, version 2.7)
< ii libpython3.6-minimal:armhf 3.6.8-1~18.04.1 armhf Minimal subset of the Python language (version 3.6)
< ii libpython3.6-stdlib:armhf 3.6.8-1~18.04.1 armhf Interactive high-level object-oriented language (standard library, version 3.6)
---
> ii libpython2.7:armhf 2.7.15~rc1-1ubuntu0.1 armhf Shared Python runtime library (version 2.7)
> ii libpython2.7-minimal:armhf 2.7.15~rc1-1ubuntu0.1 armhf Minimal subset of the Python language (version 2.7)
> ii libpython2.7-stdlib:armhf 2.7.15~rc1-1ubuntu0.1 armhf Interactive high-level object-oriented language (standard library, version 2.7)
> ii libpython3.6-minimal:armhf 3.6.7-1~18.04 armhf Minimal subset of the Python language (version 3.6)
> ii libpython3.6-stdlib:armhf 3.6.7-1~18.04 armhf Interactive high-level object-oriented language (standard library, version 3.6)
225c225
< ii libssl1.1:armhf 1.1.1-1ubuntu2.1~18.04.2 armhf Secure Sockets Layer toolkit - shared libraries
---
> ii libssl1.1:armhf 1.1.0g-2ubuntu4.3 armhf Secure Sockets Layer toolkit - shared libraries
272c272
< ii openssl 1.1.1-1ubuntu2.1~18.04.2 armhf Secure Sockets Layer toolkit - cryptographic utility
---
> ii openssl 1.1.0g-2ubuntu4.3 armhf Secure Sockets Layer toolkit - cryptographic utility
282,283c282,283
< ii python3.6 3.6.8-1~18.04.1 armhf Interactive high-level object-oriented language (version 3.6)
< ii python3.6-minimal 3.6.8-1~18.04.1 armhf Minimal subset of the Python language (version 3.6)
---
> ii python3.6 3.6.7-1~18.04 armhf Interactive high-level object-oriented language (version 3.6)
> ii python3.6-minimal 3.6.7-1~18.04 armhf Minimal subset of the Python language (version 3.6)

nginx is used primarily as an https front-end for web services on the device.

libssl is the core dependency for all of the packages in the group that, when upgraded, causes nginx to fail.

The nginx configuration includes the following SSL settings:

http {
        ##
        # SSL Settings
        ##

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
        ssl_prefer_server_ciphers on;
}

server {
  listen 443 ssl;
  ssl_certificate /etc/certs/cert.crt;
  ssl_certificate_key /etc/certs/cert.key;
  ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
  ssl_ciphers HIGH:!aNULL:!MD5;
}

Question information

Language:
English Edit question
Status:
Expired
For:
Ubuntu Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Launchpad Janitor (janitor) said :
#1

This question was expired because it remained in the 'Open' state without activity for the last 15 days.

Revision history for this message
Dietmar May (dietmar.may) said :
#2

This appears to be due to openssl requests blocking or failing until sufficient entropy is available for random number generation.

The target device is based on the TI AM335X (Sitara) ARM Cortex A8 SOC. The SOC (system on a chip) has a hardware random number generator, which requires a kernel driver to be built.

Though the kernel driver was being loaded, that's not enough for the hardware RNG to be used by the OS.

After installing the rng-tools package, which connects the hardware RNG / kernel driver to the OS layer, entropy at boot went up 100-fold; and nginx started normally at boot.