Ubuntu

update containerd:amd64 1.3.3-0 stops docker daemon

Asked by Bernd

hello we have had in several vms, the problem that after updating to "containerd:amd64 1.3.3-0ubuntu1~18.04.2/" the docker daemon is stopped and so are all running containers

the vms run with ubuntu 18.04.1/18.04.2

journal log during the update:

```
-- Logs begin at Tue 2020-01-14 09:58:27 CET, end at Fri 2020-04-03 11:30:39 CEST. --
Apr 03 06:09:09 server dockerd[1751]: time="2020-04-03T06:09:09+02:00" level=info msg="level=info ts=2020-04-03T04:09:09.648358396Z caller=loki.go:55 container_id=1eefb971e84cdaa4eadb0ac890fe9adf5275a4e9cbcf9635cf941b3aa469dd1f msg=\"ignoring empty line\" line=" plugin=255e05f2d4535991812f30e4cb5ea39fe0b498ca612c26dd89909de318b7f68d
Apr 03 06:09:09 server dockerd[1751]: time="2020-04-03T06:09:09+02:00" level=info msg="level=info ts=2020-04-03T04:09:09.660781933Z caller=loki.go:55 container_id=1eefb971e84cdaa4eadb0ac890fe9adf5275a4e9cbcf9635cf941b3aa469dd1f msg=\"ignoring empty line\" line=\" \"" plugin=255e05f2d4535991812f30e4cb5ea39fe0b498ca612c26dd89909de318b7f68d
Apr 03 06:09:14 server interface_rules[1485]: {"time":"2020-04-03 06:09:14+02:00","message":"interface enp0s3 works fine"}
Apr 03 06:09:14 server interface_rules[1485]: {"time":"2020-04-03 06:09:14+02:00","message":"interface enp0s8 works fine"}
Apr 03 06:09:24 server dockerd[1751]: time="2020-04-03T06:09:24+02:00" level=info msg="level=info ts=2020-04-03T04:09:24.650166885Z caller=loki.go:55 container_id=1eefb971e84cdaa4eadb0ac890fe9adf5275a4e9cbcf9635cf941b3aa469dd1f msg=\"ignoring empty line\" line=" plugin=255e05f2d4535991812f30e4cb5ea39fe0b498ca612c26dd89909de318b7f68d
Apr 03 06:09:24 server dockerd[1751]: time="2020-04-03T06:09:24+02:00" level=info msg="level=info ts=2020-04-03T04:09:24.663239678Z caller=loki.go:55 container_id=1eefb971e84cdaa4eadb0ac890fe9adf5275a4e9cbcf9635cf941b3aa469dd1f msg=\"ignoring empty line\" line=\" \"" plugin=255e05f2d4535991812f30e4cb5ea39fe0b498ca612c26dd89909de318b7f68d
Apr 03 06:09:31 server systemd[1]: Starting Daily apt upgrade and clean activities...
Apr 03 06:09:39 server dockerd[1751]: time="2020-04-03T06:09:39+02:00" level=info msg="level=info ts=2020-04-03T04:09:39.64981528Z caller=loki.go:55 container_id=1eefb971e84cdaa4eadb0ac890fe9adf5275a4e9cbcf9635cf941b3aa469dd1f msg=\"ignoring empty line\" line=" plugin=255e05f2d4535991812f30e4cb5ea39fe0b498ca612c26dd89909de318b7f68d
Apr 03 06:09:39 server dockerd[1751]: time="2020-04-03T06:09:39+02:00" level=info msg="level=info ts=2020-04-03T04:09:39.665086076Z caller=loki.go:55 container_id=1eefb971e84cdaa4eadb0ac890fe9adf5275a4e9cbcf9635cf941b3aa469dd1f msg=\"ignoring empty line\" line=\" \"" plugin=255e05f2d4535991812f30e4cb5ea39fe0b498ca612c26dd89909de318b7f68d
Apr 03 06:09:43 server systemd[1]: Stopping Docker Application Container Engine...
Apr 03 06:09:43 server dockerd[1751]: time="2020-04-03T06:09:43.571408106+02:00" level=info msg="Processing signal 'terminated'"
Apr 03 06:09:43 server dockerd[1751]: time="2020-04-03T06:09:43.614841270+02:00" level=info msg="Daemon shutdown complete"
Apr 03 06:09:43 server dockerd[1751]: time="2020-04-03T06:09:43.614972021+02:00" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=moby
Apr 03 06:09:43 server systemd[1]: Stopped Docker Application Container Engine.
Apr 03 06:09:43 server systemd[1]: Stopping containerd container runtime...
Apr 03 06:09:43 server systemd[1]: Closed Docker Socket for the API.
Apr 03 06:09:43 server containerd[1736]: time="2020-04-03T06:09:43.896546147+02:00" level=info msg="Stop CRI service"
Apr 03 06:09:43 server containerd[1736]: time="2020-04-03T06:09:43.897819071+02:00" level=info msg="Stop CRI service"
Apr 03 06:09:43 server containerd[1736]: time="2020-04-03T06:09:43.899864471+02:00" level=info msg="Event monitor stopped"
Apr 03 06:09:43 server containerd[1736]: time="2020-04-03T06:09:43.900291027+02:00" level=info msg="Stream server stopped"
Apr 03 06:09:43 server systemd[1]: Stopped containerd container runtime.
Apr 03 06:09:47 server systemd[1]: Reloading.
Apr 03 06:09:48 server systemd[1]: Reloading.
Apr 03 06:09:49 server systemd[1]: Reloading.
Apr 03 06:09:49 server systemd[1]: Starting containerd container runtime...
Apr 03 06:09:49 server containerd[1904]: time="2020-04-03T06:09:49.951538613+02:00" level=info msg="starting containerd" revision= version="1.3.3-0ubuntu1~18.04.2"
Apr 03 06:09:49 server systemd[1]: Started containerd container runtime.
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.021260455+02:00" level=info msg="loading plugin \"io.containerd.content.v1.content\"..." type=io.containerd.content.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.021461398+02:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." type=io.containerd.snapshotter.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.022876671+02:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.devmapper\"..." type=io.containerd.snapshotter.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.022992123+02:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.023036103+02:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.aufs\"..." type=io.containerd.snapshotter.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.031683852+02:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.native\"..." type=io.containerd.snapshotter.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.031948994+02:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd.snapshotter.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.032592571+02:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.zfs\"..." type=io.containerd.snapshotter.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.034455570+02:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="path /var/lib/containerd/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.034808794+02:00" level=info msg="loading plugin \"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.034914325+02:00" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.034952775+02:00" level=info msg="metadata content store policy set" policy=shared
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.035299108+02:00" level=info msg="loading plugin \"io.containerd.differ.v1.walking\"..." type=io.containerd.differ.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.035362929+02:00" level=info msg="loading plugin \"io.containerd.gc.v1.scheduler\"..." type=io.containerd.gc.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.035585401+02:00" level=info msg="loading plugin \"io.containerd.service.v1.containers-service\"..." type=io.containerd.service.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.035642662+02:00" level=info msg="loading plugin \"io.containerd.service.v1.content-service\"..." type=io.containerd.service.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.035676962+02:00" level=info msg="loading plugin \"io.containerd.service.v1.diff-service\"..." type=io.containerd.service.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.035751653+02:00" level=info msg="loading plugin \"io.containerd.service.v1.images-service\"..." type=io.containerd.service.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.035797984+02:00" level=info msg="loading plugin \"io.containerd.service.v1.leases-service\"..." type=io.containerd.service.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.035899825+02:00" level=info msg="loading plugin \"io.containerd.service.v1.namespaces-service\"..." type=io.containerd.service.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.035955175+02:00" level=info msg="loading plugin \"io.containerd.service.v1.snapshots-service\"..." type=io.containerd.service.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.035994665+02:00" level=info msg="loading plugin \"io.containerd.runtime.v1.linux\"..." type=io.containerd.runtime.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.504599571+02:00" level=info msg="loading plugin \"io.containerd.runtime.v2.task\"..." type=io.containerd.runtime.v2
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.504847245+02:00" level=info msg="loading plugin \"io.containerd.monitor.v1.cgroups\"..." type=io.containerd.monitor.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.507240519+02:00" level=info msg="loading plugin \"io.containerd.service.v1.tasks-service\"..." type=io.containerd.service.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.514716375+02:00" level=info msg="loading plugin \"io.containerd.internal.v1.restart\"..." type=io.containerd.internal.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.515138699+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.containers\"..." type=io.containerd.grpc.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.515385472+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.content\"..." type=io.containerd.grpc.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.515439083+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.diff\"..." type=io.containerd.grpc.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.515474183+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.events\"..." type=io.containerd.grpc.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.515521113+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.healthcheck\"..." type=io.containerd.grpc.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.515569144+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.images\"..." type=io.containerd.grpc.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.515602784+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.leases\"..." type=io.containerd.grpc.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.515645985+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.namespaces\"..." type=io.containerd.grpc.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.515812346+02:00" level=info msg="loading plugin \"io.containerd.internal.v1.opt\"..." type=io.containerd.internal.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.515960658+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.snapshots\"..." type=io.containerd.grpc.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.516016449+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.tasks\"..." type=io.containerd.grpc.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.516062909+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.version\"..." type=io.containerd.grpc.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.516218490+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.cri\"..." type=io.containerd.grpc.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.516437722+02:00" level=info msg="Start cri plugin with config {PluginConfig:{ContainerdConfig:{Snapshotter:overlayfs DefaultRuntimeName:runc DefaultRuntime:{Type: Engine: PodAnnotations:[] Root: Options:<nil> PrivilegedWithoutHostDevices:false} UntrustedWorkloadRuntime:{Type: Engine: PodAnnotations:[] Root: Options:<nil> PrivilegedWithoutHostDevices:false} Runtimes:map[runc:{Type:io.containerd.runc.v1 Engine: PodAnnotations:[] Root: Options:<nil> PrivilegedWithoutHostDevices:false}] NoPivot:false} CniConfig:{NetworkPluginBinDir:/opt/cni/bin NetworkPluginConfDir:/etc/cni/net.d NetworkPluginMaxConfNum:1 NetworkPluginConfTemplate:} Registry:{Mirrors:map[docker.io:{Endpoints:[https://registry-1.docker.io]}] Configs:map[] Auths:map[]} DisableTCPService:true StreamServerAddress:127.0.0.1 StreamServerPort:0 StreamIdleTimeout:4h0m0s EnableSelinux:false SandboxImage:k8s.gcr.io/pause:3.1 StatsCollectPeriod:10 SystemdCgroup:false EnableTLSStreaming:false X509KeyPairStreaming:{TLSCertFile: TLSKeyFile:} MaxContainerLogLineSize:16384 DisableCgroup:false DisableApparmor:false RestrictOOMScoreAdj:false MaxConcurrentDownloads:3 DisableProcMount:false} ContainerdRootDir:/var/lib/containerd ContainerdEndpoint:/run/containerd/containerd.sock RootDir:/var/lib/containerd/io.containerd.grpc.v1.cri StateDir:/run/containerd/io.containerd.grpc.v1.cri}"
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.516732916+02:00" level=info msg="Connect containerd service"
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.517528134+02:00" level=info msg="Get image filesystem path \"/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs\""
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.517888717+02:00" level=error msg="Failed to load cni during init, please check CRI plugin status before setting up network for pods" error="cni config load failed: no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config"
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.518436663+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.518837507+02:00" level=info msg="Start subscribing containerd event"
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.519057900+02:00" level=info msg="Start recovering state"
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.519208531+02:00" level=info msg=serving... address=/run/containerd/containerd.sock.ttrpc
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.519444914+02:00" level=info msg=serving... address=/run/containerd/containerd.sock
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.519481864+02:00" level=info msg="Start event monitor"
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.519573005+02:00" level=info msg="Start snapshots syncer"
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.519657075+02:00" level=info msg="Start streaming server"
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.519497564+02:00" level=info msg="containerd successfully booted in 0.569453s"
Apr 03 06:10:08 server systemd[1]: Reloading.
Apr 03 06:10:08 server systemd[1]: Stopping LSB: automatic crash report generation...
Apr 03 06:10:08 server apport[2377]: * Stopping automatic crash report generation: apport
Apr 03 06:10:08 server apport[2377]: ...done.
Apr 03 06:10:08 server systemd[1]: Stopped LSB: automatic crash report generation.
Apr 03 06:10:09 server systemd[1]: Reloading.
Apr 03 06:10:09 server systemd[1]: Reloading.
Apr 03 06:10:11 server systemd[1]: Reloading.
Apr 03 06:10:11 server systemd[1]: Reloading.
Apr 03 06:10:12 server systemd[1]: Reloading.
Apr 03 06:10:12 server systemd[1]: Starting LSB: automatic crash report generation...
Apr 03 06:10:12 server apport[2666]: * Starting automatic crash report generation: apport
Apr 03 06:10:12 server apport[2666]: ...done.
Apr 03 06:10:12 server systemd[1]: Started LSB: automatic crash report generation.
Apr 03 06:10:12 server systemd[1]: Reloading.
Apr 03 06:10:18 server systemd[1]: Started Daily apt upgrade and clean activities.

```

Question information

Language:
English Edit question
Status:
Answered
For:
Ubuntu Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Manfred Hampl (m-hampl) said : 		#1

From which version to which version did you upgrade?

There is no version "containerd:amd64 1.3.3" for bionic, but 1.3.3-0ubuntu1~18.04.1 and 1.3.3-0ubuntu1~18.04.2

Revision history for this message
Bernd (badsmoke) said : 		#2

i have update the specific version "containerd:amd64 1.3.3-0ubuntu1~18.04.2"

Revision history for this message
actionparsnip (andrew-woodhead666) said : 		#3

What is the output of:

apt-cache policy containerd; lsb_release -a; uname -a

Thanks

Revision history for this message
actionparsnip (andrew-woodhead666) said : 		#4

I'm smelling that it's either Focal, or you installed the deb from Focal into Bionic to "upgrade" the application and broke it

Revision history for this message
actionparsnip (andrew-woodhead666) said : 		#5

or a PPA

Revision history for this message
Bernd (badsmoke) said : 		#6

user@server:~$ apt-cache policy containerd; lsb_release -a; uname -a
containerd:
  Installiert: 1.3.3-0ubuntu1~18.04.2
  Installationskandidat: 1.3.3-0ubuntu1~18.04.2
  Versionstabelle:
 *** 1.3.3-0ubuntu1~18.04.2 500
        500 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages
        100 /var/lib/dpkg/status
     1.2.6-0ubuntu1~18.04.2 500
        500 http://archive.ubuntu.com/ubuntu bionic-security/universe amd64 Packages
     0.2.5-0ubuntu2 500
        500 http://archive.ubuntu.com/ubuntu bionic/universe amd64 Packages
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.4 LTS
Release: 18.04
Codename: bionic
Linux server 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Revision history for this message
Bernd (badsmoke) said : 		#7

one of the vms runs unchanged in the aws cloud, so no ppa was changed

Revision history for this message
Manfred Hampl (m-hampl) said : 		#8

A similar question as been asked in https://answers.launchpad.net/ubuntu/+question/689690

Revision history for this message
actionparsnip (andrew-woodhead666) said : 		#9

Looks like the official packages....I can be wrong :)

Revision history for this message
lrkwz (luca-orlandi) said : 		#10

SAme here:

$ apt-cache policy containerd; lsb_release -a; uname -a
containerd:
  Installed: 1.3.3-0ubuntu1~18.04.2
  Candidate: 1.3.3-0ubuntu1~18.04.2
  Version table:
 *** 1.3.3-0ubuntu1~18.04.2 500
        500 http://ubuntu.mirrors.ovh.net/ubuntu bionic-updates/universe amd64 Packages
        100 /var/lib/dpkg/status
     1.2.6-0ubuntu1~18.04.2 500
        500 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages
     0.2.5-0ubuntu2 500
        500 http://ubuntu.mirrors.ovh.net/ubuntu bionic/universe amd64 Packages
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.4 LTS
Release: 18.04
Codename: bionic
Linux srv1 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Revision history for this message
Manfred Hampl (m-hampl) said : 		#11

If the same problem has been experienced by more than one person, then this seems to be a general problem and somebody affected should create a bug report.

Revision history for this message
lrkwz (luca-orlandi) said : 		#12

Resolved with a `sudo reboot now` :-|

Can you help with this problem?

Provide an answer of your own, or ask Bernd for more information if necessary.

To post a message you must log in.