==== Begin SRU Template [cloud-init] ====
[Impact]
One of cloud-init's features is to upgrade the system during first boot so that it is fully up to date when the user code starts running.
[Test Case]
launch an old instance of 16.04 that will need an update to snapd with
user-data that indicates a package upgrade should be done.
$ lxc image show ubuntu:74a491804877
autoupdate: false
properties:
aliases: 16.04,default,lts,x,xenial
architecture: amd64
description: ubuntu 16.04 LTS amd64 (release) (20160830)
label: release
os: ubuntu
release: xenial
serial: "20160830"
version: "16.04"
public: true
$ printf "#%s\n%s\n" cloud-config "packages: [snapd]" > user-data
$ lxc launch ubuntu:74a491804877 xrecreate "--config=user.user-data=$(cat user-data)"
$ lxc exec xrecreate -- tail -f /var/log/cloud-init-output.log
# you will see the output log hang at:
# Setting up snapd (2.14.2~16.04) ...
## Now get new container and patch in cloud-init
$ lxc launch ubuntu:74a491804877 xpatched
# let it boot, with no user-data saying to update.
$ sleep 10
# update the container to new cloud-init, then clean it to make
# it look like first boot again.
$ lxc file push - xpatched/etc/cloud/cloud.cfg.d/update.cfg < user-data
$ lxc exec xpatched -- sh -c '
p=/etc/apt/sources.list.d/proposed.list
echo deb http://archive.ubuntu.com/ubuntu xenial-proposed main > "$p" &&
apt-get update -q && apt-get -qy install cloud-init'
$ lxc exec xpatched -- sh -c '
cd /var/lib/cloud && for d in *; do [ "$d" = "seed" ] || rm -Rf "$d"; done
rm -Rf /var/log/cloud-init*'
$ lxc exec xpatched reboot
$ lxc exec xpatched -- tail -f /var/log/cloud-init-output.log
# snapd installed and a 'Cloud-init finished' message.
[Regression Potential]
The change to running package installation later in boot will likely affect some things. However, previously a larger set of things were unreliable. This will make things over all more reliable.
==== End SRU Template [cloud-init] ====
I reproducibly run into an eternal hang when deploying services with Juju, when it prepares a new xenial testbed. The current xenial cloud image does not have the latest snapd, so snapd gets dist-upgraded:
Preparing to unpack .../snapd_2.14.2~16.04_amd64.deb ...
Warning: Stopping snapd.service, but it can still be activated by:
snapd.socket
Unpacking snapd (2.14.2~16.04) over (2.13) ...
Setting up snapd (2.14.2~16.04) ...
[...] hangs
The postinst tries to start snapd.boot-ok.service on upgrade:
|-cloud-init(311)-+-apt-get(577)---dpkg(845)---snapd.postinst(846)---perl(919)---systemctl(922)
| `-sh(354)---tee(355)
root 922 0.0 0.0 25316 1412 pts/0 S+ 06:09 0:00 /bin/systemctl start snapd.boot-ok.service
This hangs eternally because:
- cloud-init's dist-upgrade runs *during* the boot process, so that the system is not fully booted yet when this happens (see bug 1576692); thus multi-user.target is *not* yet active
- snapd.boot-ok.service is After=multi-user.target
- "systemctl start" is synchronous by default, i. e. it waits until the service is started unless you use --no-block.
Thus snapd.postinst waits on snapd.boot-ok.service waits on multi-user.target waits on cloud-init to finish waits on snapd.postinst to finish.
I think conceptually you shouldn't start snapd.boot-ok.service in the postinst; if the system is already booted (manual dist-upgrade) it should already be running, and if it does get upgraded during boot (with cloud-init) then you shouldn't pretend that booting is already finished. So I suggest to use dh_installinit with --no-scripts for snapd.boot-ok.service.
Status changed to 'Confirmed' because the bug affects multiple users.