unable to acces guest cgroup after systemd daemon-reload and libvirtd restart

Asked by Erwin Hager on 2018-03-02

I am trying to read CPU stats from a running VM. It works fine when I try it on a VM that I have just started, however when I try to read the stats after systemd reloaded and libvirtd has been restarted while the VM is still running I get the following error:

error: Requested operation is not valid: cgroup CPUACCT controller is not mounted

I can reproduce this error by taking the following steps:

start VM by running:
$ virsh start webstuff

request CPU stats by running:
$ virsh cpu-stats webstuff
CPU0:
 cpu_time 74.028426322 seconds
 vcpu_time 66.172828369 seconds
CPU1:
 cpu_time 245.630499935 seconds
 vcpu_time 240.445737437 seconds
CPU2:
 cpu_time 480.159937313 seconds
 vcpu_time 471.825403803 seconds
CPU3:
 cpu_time 223.615986499 seconds
 vcpu_time 217.718292798 seconds
Total:
 cpu_time 1023.445384757 seconds
 user_time 14.190000000 seconds
 system_time 26.340000000 seconds

reload systemd:
$ sudo systemctl daemon-reload

request CPU stats again:
$ virsh cpu-stats webstuff
CPU0:
 cpu_time 81.244675853 seconds
 vcpu_time 72.600668269 seconds
CPU1:
 cpu_time 259.391406546 seconds
 vcpu_time 253.761752135 seconds
CPU2:
 cpu_time 487.465607796 seconds
 vcpu_time 478.549083841 seconds
CPU3:
 cpu_time 231.854335519 seconds
 vcpu_time 225.344671622 seconds
Total:
 cpu_time 1059.956025714 seconds
 user_time 15.110000000 seconds
 system_time 34.390000000 seconds

restart libvirtd:
$ sudo systemctl restart libvirtd

request CPU stats again by running:
$ virsh cpu-stats webstuff
error: Failed to retrieve CPU statistics for domain 'webstuff'
error: Requested operation is not valid: cgroup CPUACCT controller is not mounted

start another VM to try if it works:
$ virsh start webstuff-clone

request CPU stats from second VM:
$ virsh cpu-stats webstuff-clone
CPU0:
 cpu_time 0.077817002 seconds
 vcpu_time 0.026559548 seconds
CPU1:
 cpu_time 0.023116769 seconds
 vcpu_time 0.001427647 seconds
CPU2:
 cpu_time 2.526200306 seconds
 vcpu_time 2.456538659 seconds
CPU3:
 cpu_time 2.460251504 seconds
 vcpu_time 2.460251504 seconds
Total:
 cpu_time 5.091386434 seconds
 user_time 0.160000000 seconds
 system_time 1.100000000 seconds

I expect to be able to retrieve the CPU stats of a running VM even after a daemon reload. Libvirtd usually gets restarted after an upgrade.

software versions:

Distributor ID: Ubuntu
Description: Ubuntu 16.04.4 LTS
Release: 16.04
Codename: xenial

Linux hostname 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

systemd 229
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN

libvirtd (libvirt) 1.3.1

I can provide more information if needed.

Question information

Language:
English Edit question
Status:
Expired
For:
Ubuntu Edit question
Assignee:
No assignee Edit question
Last query:
2018-03-02
Last reply:
2018-03-18
Launchpad Janitor (janitor) said : #1

This question was expired because it remained in the 'Open' state without activity for the last 15 days.

Same behavior on Debian/Stretch (systemd 232 / libvirt 3.0.0)

Noticed the cpu,cpuacct cgroup from an underlying QEMU process jumps from:

A.
# fgrep cpu,cpuacct /proc/<pid>/cgroup
2:cpu,cpuacct:/machine/qemu-3-<domain>.libvirt-qemu/emulator

To:

B.
# fgrep cpu,cpuacct /proc/<pid>/cgroup
2:cpu,cpuacct:/system.slice

After the systemctl daemon-reload + service libvirtd restart actions

Netsearching around, I stumbled on:
1. https://www.freedesktop.org/wiki/Software/systemd/ControlGroupInterface/
2. https://github.com/moby/moby/pull/20633 (Docker-related)

cat << EOF > /etc/systemd/system/libvirtd.service.d/cgroup-delegation.conf
[Service]
Delegate=yes
EOF

Does solve the problem!