qemu process got killed by oom killer while there are plenty of reclaimable slab

Asked by norman shen on 2021-06-06

I saw strange behavior on production where a qemu process got killed by OOM killer while there are plenty of reclaimable slab available.

the compute node got around 372 GB memory, but when OOM happens from kernel log only around 200 GB are used by user programs while anther 100 GB is taken by reclaimable slab...

system is ubunt 18.04 with kernel 4.15.0.

[Thu Jun 3 15:32:38 2021] CPU 18/KVM invoked oom-killer: gfp_mask=0x14280ca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), nodemask=(null), order=0, oom_score_adj=999
[Thu Jun 3 15:32:38 2021] CPU 18/KVM cpuset=vcpu18 mems_allowed=0-1
[Thu Jun 3 15:32:38 2021] CPU: 7 PID: 3416489 Comm: CPU 18/KVM Not tainted 4.15.0-72-generic #81-Ubuntu
[Thu Jun 3 15:32:38 2021] Call Trace:
[Thu Jun 3 15:32:38 2021] dump_stack+0x63/0x8e
[Thu Jun 3 15:32:38 2021] dump_header+0x71/0x285
[Thu Jun 3 15:32:38 2021] oom_kill_process+0x21f/0x420
[Thu Jun 3 15:32:38 2021] out_of_memory+0x116/0x4e0
[Thu Jun 3 15:32:38 2021] __alloc_pages_slowpath+0xa53/0xe00
[Thu Jun 3 15:32:38 2021] __alloc_pages_nodemask+0x29a/0x2c0
[Thu Jun 3 15:32:38 2021] alloc_pages_vma+0x88/0x1f0
[Thu Jun 3 15:32:38 2021] __handle_mm_fault+0x8b7/0x1290
[Thu Jun 3 15:32:38 2021] handle_mm_fault+0xb1/0x210
[Thu Jun 3 15:32:38 2021] __get_user_pages+0x1ee/0x720
[Thu Jun 3 15:32:38 2021] get_user_pages+0x42/0x50
[Thu Jun 3 15:32:38 2021] __gfn_to_pfn_memslot+0x126/0x410 [kvm]
[Thu Jun 3 15:32:38 2021] try_async_pf+0x66/0x1f0 [kvm]
[Thu Jun 3 15:32:38 2021] tdp_page_fault+0x138/0x290 [kvm]
[Thu Jun 3 15:32:38 2021] ? vmexit_fill_RSB+0x1c/0x40 [kvm_intel]
[Thu Jun 3 15:32:38 2021] kvm_mmu_page_fault+0x62/0x160 [kvm]
[Thu Jun 3 15:32:38 2021] handle_ept_violation+0xbb/0x150 [kvm_intel]
[Thu Jun 3 15:32:38 2021] vmx_handle_exit+0xb3/0xe80 [kvm_intel]
[Thu Jun 3 15:32:38 2021] ? vmexit_fill_RSB+0x1c/0x40 [kvm_intel]
[Thu Jun 3 15:32:38 2021] ? vmexit_fill_RSB+0x10/0x40 [kvm_intel]
[Thu Jun 3 15:32:38 2021] ? vmexit_fill_RSB+0x1c/0x40 [kvm_intel]
[Thu Jun 3 15:32:38 2021] ? vmexit_fill_RSB+0x10/0x40 [kvm_intel]
[Thu Jun 3 15:32:38 2021] ? vmx_vcpu_run+0x3fa/0x600 [kvm_intel]
[Thu Jun 3 15:32:38 2021] vcpu_enter_guest+0x424/0x1260 [kvm]
[Thu Jun 3 15:32:38 2021] kvm_arch_vcpu_ioctl_run+0x203/0x3e0 [kvm]
[Thu Jun 3 15:32:38 2021] ? kvm_arch_vcpu_ioctl_run+0x203/0x3e0 [kvm]
[Thu Jun 3 15:32:38 2021] kvm_vcpu_ioctl+0x2a6/0x620 [kvm]
[Thu Jun 3 15:32:38 2021] ? do_futex+0x185/0x590
[Thu Jun 3 15:32:38 2021] do_vfs_ioctl+0xa8/0x630
[Thu Jun 3 15:32:38 2021] ? SyS_futex+0x13b/0x180
[Thu Jun 3 15:32:38 2021] SyS_ioctl+0x79/0x90
[Thu Jun 3 15:32:38 2021] do_syscall_64+0x73/0x130
[Thu Jun 3 15:32:38 2021] entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[Thu Jun 3 15:32:38 2021] RIP: 0033:0x7f6f4a1bb6d7
[Thu Jun 3 15:32:38 2021] RSP: 002b:00007f6ea97f9818 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[Thu Jun 3 15:32:38 2021] RAX: ffffffffffffffda RBX: 000000000000ae80 RCX: 00007f6f4a1bb6d7
[Thu Jun 3 15:32:38 2021] RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000044
[Thu Jun 3 15:32:38 2021] RBP: 0000561bbbfe5c70 R08: 0000561bbab97990 R09: 0000561bbb1e44e0
[Thu Jun 3 15:32:38 2021] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[Thu Jun 3 15:32:38 2021] R13: 00007f6f4fe37000 R14: 0000561bbbfe5c70 R15: 0000561bbbfe5c70
[Thu Jun 3 15:32:38 2021] Mem-Info:
[Thu Jun 3 15:32:38 2021] active_anon:65919219 inactive_anon:317 isolated_anon:0
                            active_file:535 inactive_file:677 isolated_file:32
                            unevictable:118187 dirty:0 writeback:0 unstable:0
                            slab_reclaimable:25801716 slab_unreclaimable:5518465
                            mapped:2889 shmem:1280 pagetables:140051 bounce:0
                            free:220415 free_pcp:1997 free_cma:0
[Thu Jun 3 15:32:38 2021] Node 0 active_anon:136866928kB inactive_anon:676kB active_file:612kB inactive_file:644kB unevictable:109156kB isolated(anon):0kB isolated(file):0kB mapped:3512kB dirty:0kB writeback:0kB shmem:2280kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[Thu Jun 3 15:32:38 2021] Node 1 active_anon:126809948kB inactive_anon:592kB active_file:1528kB inactive_file:2064kB unevictable:363592kB isolated(anon):0kB isolated(file):128kB mapped:8044kB dirty:0kB writeback:0kB shmem:2840kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[Thu Jun 3 15:32:38 2021] Node 0 DMA free:15900kB min:0kB low:12kB high:24kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15996kB managed:15900kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[Thu Jun 3 15:32:38 2021] lowmem_reserve[]: 0 1553 192032 192032 192032
[Thu Jun 3 15:32:38 2021] Node 0 DMA32 free:762604kB min:360kB low:1948kB high:3536kB active_anon:871880kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:1713468kB managed:1647064kB mlocked:0kB kernel_stack:32kB pagetables:1684kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[Thu Jun 3 15:32:38 2021] lowmem_reserve[]: 0 0 190479 190479 190479
[Thu Jun 3 15:32:38 2021] Node 0 Normal free:53588kB min:44516kB low:239564kB high:434612kB active_anon:135995128kB inactive_anon:676kB active_file:620kB inactive_file:620kB unevictable:109156kB writepending:0kB present:198180864kB managed:195051028kB mlocked:109156kB kernel_stack:18904kB pagetables:215836kB bounce:0kB free_pcp:4484kB local_pcp:620kB free_cma:0kB
[Thu Jun 3 15:32:38 2021] lowmem_reserve[]: 0 0 0 0 0
[Thu Jun 3 15:32:38 2021] Node 1 Normal free:49244kB min:45224kB low:243368kB high:441512kB active_anon:126810348kB inactive_anon:592kB active_file:1452kB inactive_file:2120kB unevictable:363592kB writepending:0kB present:201326592kB managed:198153736kB mlocked:363592kB kernel_stack:25128kB pagetables:342784kB bounce:0kB free_pcp:2776kB local_pcp:100kB free_cma:0kB
[Thu Jun 3 15:32:38 2021] lowmem_reserve[]: 0 0 0 0 0
[Thu Jun 3 15:32:38 2021] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 2*32kB (U) 3*64kB (U) 0*128kB 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15900kB
[Thu Jun 3 15:32:38 2021] Node 0 DMA32: 166*4kB (UM) 122*8kB (UM) 123*16kB (UM) 61*32kB (UM) 21*64kB (UM) 4*128kB (UM) 0*256kB 1*512kB (U) 1*1024kB (U) 0*2048kB 184*4096kB (ME) = 762616kB
[Thu Jun 3 15:32:38 2021] Node 0 Normal: 2814*4kB (UME) 176*8kB (UME) 55*16kB (UME) 1391*32kB (UME) 0*64kB 1*128kB (E) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 58184kB
[Thu Jun 3 15:32:38 2021] Node 1 Normal: 1671*4kB (UME) 126*8kB (UME) 44*16kB (UME) 1300*32kB (UME) 1*64kB (M) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 50060kB
[Thu Jun 3 15:32:38 2021] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[Thu Jun 3 15:32:38 2021] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[Thu Jun 3 15:32:38 2021] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[Thu Jun 3 15:32:38 2021] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[Thu Jun 3 15:32:38 2021] 5030 total pagecache pages
[Thu Jun 3 15:32:38 2021] 0 pages in swap cache
[Thu Jun 3 15:32:38 2021] Swap cache stats: add 0, delete 0, find 0/0
[Thu Jun 3 15:32:38 2021] Free swap = 0kB
[Thu Jun 3 15:32:38 2021] Total swap = 0kB
[Thu Jun 3 15:32:38 2021] 100309230 pages RAM
[Thu Jun 3 15:32:38 2021] 0 pages HighMem/MovableOnly
[Thu Jun 3 15:32:38 2021] 1592298 pages reserved
[Thu Jun 3 15:32:38 2021] 0 pages cma reserved
[Thu Jun 3 15:32:38 2021] 0 pages hwpoisoned
[Thu Jun 3 15:32:38 2021] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[Thu Jun 3 15:32:38 2021] [ 954] 0 954 11954 1337 126976 0 -1000 systemd-udevd
[Thu Jun 3 15:32:38 2021] [ 1429] 101 1429 17687 731 172032 0 0 systemd-resolve
[Thu Jun 3 15:32:38 2021] [ 1669] 103 1669 12746 843 143360 0 -900 dbus-daemon
[Thu Jun 3 15:32:38 2021] [ 1685] 0 1685 7083 475 98304 0 0 atd
[Thu Jun 3 15:32:38 2021] [ 1691] 0 1691 72030 744 196608 0 0 accounts-daemon
[Thu Jun 3 15:32:38 2021] [ 1717] 0 1717 27853 847 118784 0 0 irqbalance
[Thu Jun 3 15:32:38 2021] [ 1738] 0 1738 42805 2659 245760 0 0 networkd-dispat
[Thu Jun 3 15:32:38 2021] [ 1768] 0 1768 115774 2203 131072 0 0 lxcfs
[Thu Jun 3 15:32:38 2021] [ 1873] 0 1873 72220 679 212992 0 0 polkitd
[Thu Jun 3 15:32:38 2021] [ 1878] 0 1878 46919 2596 258048 0 0 unattended-upgr
[Thu Jun 3 15:32:38 2021] [ 1971] 0 1971 19692 649 196608 0 0 login
[Thu Jun 3 15:32:38 2021] [ 5366] 0 5366 15986 872 172032 0 0 lldpd
[Thu Jun 3 15:32:38 2021] [ 5372] 111 5372 15534 658 155648 0 0 lldpd
[Thu Jun 3 15:32:38 2021] [ 9070] 0 9070 39052 12899 348160 0 0 salt-minion
[Thu Jun 3 15:32:38 2021] [ 9072] 0 9072 157299 19790 577536 0 0 salt-minion
[Thu Jun 3 15:32:38 2021] [ 9080] 0 9080 68881 13123 434176 0 0 salt-minion
[Thu Jun 3 15:32:38 2021] [10399] 0 10399 5625 1307 94208 0 0 ovsdb-server
[Thu Jun 3 15:32:38 2021] [10467] 0 10467 910632 118189 1183744 0 0 ovs-vswitchd
[Thu Jun 3 15:32:38 2021] [12653] 100 12653 18127 916 188416 0 0 systemd-network
[Thu Jun 3 15:32:38 2021] [33030] 106 33030 6711 105 94208 0 0 uuidd
[Thu Jun 3 15:32:38 2021] [26766] 42424 26766 5050729 4314932 36372480 0 1000 qemu-system-x86
[Thu Jun 3 15:32:38 2021] [25421] 0 25421 19486 1138 192512 0 0 systemd
[Thu Jun 3 15:32:38 2021] [25422] 0 25422 64511 886 262144 0 0 (sd-pam)
[Thu Jun 3 15:32:38 2021] [25433] 0 25433 5340 856 86016 0 0 bash
[Thu Jun 3 15:32:38 2021] [ 3558] 0 3558 7563 207 90112 0 0 ipmievd
[Thu Jun 3 15:32:38 2021] [34463] 42424 34463 3065411 2134534 18681856 0 1000 qemu-system-x86
[Thu Jun 3 15:32:38 2021] [15471] 0 15471 18074 664 184320 0 -1000 sshd
[Thu Jun 3 15:32:38 2021] [ 8583] 0 8583 7507 553 102400 0 0 cron
[Thu Jun 3 15:32:38 2021] [1123144] 42424 1123144 5217011 4246721 35741696 0 1000 qemu-system-x86
[Thu Jun 3 15:32:38 2021] [1123149] 42424 1123149 5184453 1527471 14028800 0 1000 qemu-system-x86
[Thu Jun 3 15:32:38 2021] [3092185] 42424 3092185 3008359 2158314 18972672 0 1000 qemu-system-x86
[Thu Jun 3 15:32:38 2021] [1829721] 42424 1829721 2977646 2177346 18886656 0 1000 qemu-system-x86
[Thu Jun 3 15:32:38 2021] [2893751] 42424 2893751 18025680 16902105 137207808 0 1000 qemu-system-x86
[Thu Jun 3 15:32:38 2021] [3034509] 0 3034509 26476 288 106496 0 0 lvmetad
[Thu Jun 3 15:32:38 2021] [852199] 42424 852199 5068296 4209215 35246080 0 1000 qemu-system-x86
[Thu Jun 3 15:32:38 2021] [3444528] 42424 3444528 5082484 4222127 35401728 0 1000 qemu-system-x86
[Thu Jun 3 15:32:38 2021] [3541149] 42424 3541149 18146771 16840351 136617984 0 1000 qemu-system-x86
[Thu Jun 3 15:32:38 2021] [2874733] 0 2874733 17651 738 176128 0 0 systemd-logind
[Thu Jun 3 15:32:38 2021] [3141573] 0 3141573 33467 845 139264 0 0 job-agent
[Thu Jun 3 15:32:38 2021] [3143006] 102 3143006 65767 899 155648 0 0 rsyslogd
[Thu Jun 3 15:32:38 2021] [3143340] 0 3143340 84603 3409 704512 0 0 systemd-journal
[Thu Jun 3 15:32:38 2021] [3155103] 0 3155103 1999562 7101 1175552 0 0 containerd
[Thu Jun 3 15:32:38 2021] [3155104] 0 3155104 1427750 19318 1056768 0 -999 dockerd
[Thu Jun 3 15:32:38 2021] [3156172] 0 3156172 1219926 3969 786432 0 0 docker
[Thu Jun 3 15:32:38 2021] [3156204] 0 3156204 26837 258 69632 0 -999 containerd-shim
[Thu Jun 3 15:32:38 2021] [3156234] 0 3156234 263 1 32768 0 0 init
[Thu Jun 3 15:32:38 2021] [3156261] 0 3156261 6463 157 94208 0 0 ntpd
[Thu Jun 3 15:32:38 2021] [3156667] 0 3156667 27189 225 77824 0 -999 containerd-shim
[Thu Jun 3 15:32:38 2021] [3156695] 0 3156695 256 1 32768 0 -998 pause
[Thu Jun 3 15:32:38 2021] [3156904] 0 3156904 27189 382 77824 0 -999 containerd-shim
[Thu Jun 3 15:32:38 2021] [3156937] 0 3156937 1052 22 53248 0 -998 runsvdir
[Thu Jun 3 15:32:38 2021] [3157144] 0 3157144 1014 16 49152 0 -998 runsv
[Thu Jun 3 15:32:38 2021] [3157145] 0 3157145 1014 17 45056 0 -998 runsv
[Thu Jun 3 15:32:38 2021] [3157146] 0 3157146 1014 17 49152 0 -998 runsv
[Thu Jun 3 15:32:38 2021] [3157147] 0 3157147 1014 16 49152 0 -998 runsv
[Thu Jun 3 15:32:38 2021] [3157150] 0 3157150 37855 2063 184320 0 -998 calico-node
[Thu Jun 3 15:32:38 2021] [3157151] 0 3157151 38623 5184 204800 0 -998 calico-node
[Thu Jun 3 15:32:38 2021] [3157388] 0 3157388 676 292 32768 0 -998 bird
[Thu Jun 3 15:32:38 2021] [3157389] 0 3157389 416 45 28672 0 -998 bird6
[Thu Jun 3 15:32:38 2021] [3165454] 0 3165454 1764848 26071 1445888 0 -999 kubelet
[Thu Jun 3 15:32:38 2021] [3165813] 0 3165813 27189 234 73728 0 -999 containerd-shim
[Thu Jun 3 15:32:38 2021] [3165852] 0 3165852 256 1 32768 0 -998 pause
[Thu Jun 3 15:32:38 2021] [3165929] 0 3165929 27189 224 77824 0 -999 containerd-shim
[Thu Jun 3 15:32:38 2021] [3165957] 0 3165957 861074 9796 933888 0 -998 hyperkube
[Thu Jun 3 15:32:38 2021] [3491460] 0 3491460 27189 286 77824 0 -999 containerd-shim
[Thu Jun 3 15:32:38 2021] [3491502] 42424 3491502 256 1 32768 0 -998 pause
[Thu Jun 3 15:32:38 2021] [3496246] 0 3496246 27189 474 73728 0 -999 containerd-shim
[Thu Jun 3 15:32:38 2021] [3496290] 42424 3496290 58221 27207 512000 0 999 neutron-l3-agen
[Thu Jun 3 15:32:38 2021] [3496511] 0 3496511 32018 8615 286720 0 999 privsep-helper
[Thu Jun 3 15:32:38 2021] [3500227] 42424 3500227 13530 290 126976 0 999 haproxy
[Thu Jun 3 15:32:38 2021] [3500268] 42424 3500268 13530 292 135168 0 999 haproxy
[Thu Jun 3 15:32:38 2021] [3500275] 42424 3500275 13530 289 131072 0 999 haproxy
[Thu Jun 3 15:32:38 2021] [3500313] 42424 3500313 13530 291 135168 0 999 haproxy
[Thu Jun 3 15:32:38 2021] [3501293] 42424 3501293 13530 289 131072 0 999 haproxy
[Thu Jun 3 15:32:38 2021] [3515852] 0 3515852 27189 207 77824 0 -999 containerd-shim
[Thu Jun 3 15:32:38 2021] [3515877] 42424 3515877 256 1 32768 0 -998 pause
[Thu Jun 3 15:32:38 2021] [3518108] 0 3518108 27189 749 77824 0 -999 containerd-shim
[Thu Jun 3 15:32:38 2021] [3518133] 42424 3518133 60714 31689 528384 0 999 neutron-openvsw
[Thu Jun 3 15:32:38 2021] [3518492] 42424 3518492 5286 165 86016 0 999 ovsdb-client
[Thu Jun 3 15:32:38 2021] [3518496] 42424 3518496 5286 159 86016 0 999 ovsdb-client
[Thu Jun 3 15:32:38 2021] [3528524] 0 3528524 27189 261 73728 0 -999 containerd-shim
[Thu Jun 3 15:32:38 2021] [3528561] 42424 3528561 256 1 32768 0 -998 pause
[Thu Jun 3 15:32:38 2021] [3534774] 0 3534774 27189 511 77824 0 -999 containerd-shim
[Thu Jun 3 15:32:38 2021] [3534801] 42424 3534801 666149 33522 1171456 0 999 nova-compute
[Thu Jun 3 15:32:38 2021] [3535021] 0 3535021 918040 9528 716800 0 999 privsep-helper
[Thu Jun 3 15:32:38 2021] [3540909] 0 3540909 27189 269 77824 0 -999 containerd-shim
[Thu Jun 3 15:32:38 2021] [3540936] 0 3540936 256 1 32768 0 -998 pause
[Thu Jun 3 15:32:38 2021] [3541695] 0 3541695 27205 507 77824 0 -999 containerd-shim
[Thu Jun 3 15:32:38 2021] [3541721] 0 3541721 582754 3116 782336 0 999 libvirtd
[Thu Jun 3 15:32:38 2021] [3569210] 0 3569210 27189 211 73728 0 -999 containerd-shim
[Thu Jun 3 15:32:38 2021] [3569236] 42424 3569236 256 1 32768 0 -998 pause
[Thu Jun 3 15:32:38 2021] [3569917] 0 3569917 27189 441 77824 0 -999 containerd-shim
[Thu Jun 3 15:32:38 2021] [3569944] 42424 3569944 48312 19722 421888 0 995 neutron-metadat
[Thu Jun 3 15:32:38 2021] [3570046] 42424 3570046 48573 19983 417792 0 995 neutron-metadat
[Thu Jun 3 15:32:38 2021] [3570047] 42424 3570047 48279 19687 417792 0 995 neutron-metadat
[Thu Jun 3 15:32:38 2021] [3570048] 42424 3570048 48381 19831 417792 0 995 neutron-metadat
[Thu Jun 3 15:32:38 2021] [3570049] 42424 3570049 48356 19729 417792 0 995 neutron-metadat
[Thu Jun 3 15:32:38 2021] [3570050] 42424 3570050 48229 19696 413696 0 995 neutron-metadat
[Thu Jun 3 15:32:38 2021] [3570051] 42424 3570051 48291 19697 417792 0 995 neutron-metadat
[Thu Jun 3 15:32:38 2021] [3570052] 42424 3570052 48501 19898 413696 0 995 neutron-metadat
[Thu Jun 3 15:32:38 2021] [3570053] 42424 3570053 48590 19947 413696 0 995 neutron-metadat
[Thu Jun 3 15:32:38 2021] [3570054] 42424 3570054 48294 19703 413696 0 995 neutron-metadat
[Thu Jun 3 15:32:38 2021] [3570055] 42424 3570055 48258 19712 413696 0 995 neutron-metadat
[Thu Jun 3 15:32:38 2021] [3570056] 42424 3570056 48439 19882 417792 0 995 neutron-metadat
[Thu Jun 3 15:32:38 2021] [3570057] 42424 3570057 48265 19725 413696 0 995 neutron-metadat
[Thu Jun 3 15:32:38 2021] [3570058] 42424 3570058 48280 19704 413696 0 995 neutron-metadat
[Thu Jun 3 15:32:38 2021] [3570059] 42424 3570059 48376 19791 417792 0 995 neutron-metadat
[Thu Jun 3 15:32:38 2021] [3570060] 42424 3570060 48443 19853 417792 0 995 neutron-metadat
[Thu Jun 3 15:32:38 2021] [3570061] 42424 3570061 48292 19710 413696 0 995 neutron-metadat
[Thu Jun 3 15:32:38 2021] [3570062] 42424 3570062 48384 19780 413696 0 995 neutron-metadat
[Thu Jun 3 15:32:38 2021] [3570063] 42424 3570063 48541 19912 417792 0 995 neutron-metadat
[Thu Jun 3 15:32:38 2021] [3570064] 42424 3570064 48397 19850 413696 0 995 neutron-metadat
[Thu Jun 3 15:32:38 2021] [3570065] 42424 3570065 48638 20072 417792 0 995 neutron-metadat
[Thu Jun 3 15:32:38 2021] [3570066] 42424 3570066 48285 19731 417792 0 995 neutron-metadat
[Thu Jun 3 15:32:38 2021] [3570067] 42424 3570067 48489 19871 413696 0 995 neutron-metadat
[Thu Jun 3 15:32:38 2021] [3570068] 42424 3570068 48331 19733 413696 0 995 neutron-metadat
[Thu Jun 3 15:32:38 2021] [3570069] 42424 3570069 48350 19703 417792 0 995 neutron-metadat
[Thu Jun 3 15:32:38 2021] [1016118] 0 1016118 26837 198 73728 0 -999 containerd-shim
[Thu Jun 3 15:32:38 2021] [1016147] 0 1016147 256 1 32768 0 -998 pause
[Thu Jun 3 15:32:38 2021] [1016586] 0 1016586 27189 244 77824 0 -999 containerd-shim
[Thu Jun 3 15:32:38 2021] [1016617] 0 1016617 256 1 32768 0 -998 pause
[Thu Jun 3 15:32:38 2021] [1016927] 0 1016927 27189 226 77824 0 -999 containerd-shim
[Thu Jun 3 15:32:38 2021] [1016954] 0 1016954 1130 16 57344 0 1000 tini
[Thu Jun 3 15:32:38 2021] [1016997] 0 1016997 1242905 10625 999424 0 1000 libvirt-qemu-ex
[Thu Jun 3 15:32:38 2021] [1017395] 0 1017395 27189 204 77824 0 -999 containerd-shim
[Thu Jun 3 15:32:38 2021] [1017436] 0 1017436 1131 17 57344 0 1000 tini
[Thu Jun 3 15:32:38 2021] [1017471] 0 1017471 565022 2408 376832 0 1000 san-multipath-e
[Thu Jun 3 15:32:38 2021] [1017570] 0 1017570 27189 265 77824 0 -999 containerd-shim
[Thu Jun 3 15:32:38 2021] [1017597] 0 1017597 1131 16 53248 0 1000 tini
[Thu Jun 3 15:32:38 2021] [1017624] 0 1017624 1431487 21761 942080 0 1000 telegraf
[Thu Jun 3 15:32:38 2021] [1027196] 0 1027196 26837 237 73728 0 -999 containerd-shim
[Thu Jun 3 15:32:38 2021] [1027222] 0 1027222 256 1 32768 0 -998 pause
[Thu Jun 3 15:32:38 2021] [1028115] 0 1028115 27189 428 77824 0 -999 containerd-shim
[Thu Jun 3 15:32:38 2021] [1028140] 0 1028140 3278 28 69632 0 998 tini
[Thu Jun 3 15:32:38 2021] [1028167] 0 1028167 41242 8029 389120 0 998 fluentd
[Thu Jun 3 15:32:38 2021] [1028209] 0 1028209 113843 13430 868352 0 998 ruby2.5
[Thu Jun 3 15:32:38 2021] [1028210] 0 1028210 44449 9920 397312 0 998 ruby2.5
[Thu Jun 3 15:32:38 2021] [2240279] 0 2240279 217134 8077 385024 0 999 privsep-helper
[Thu Jun 3 15:32:38 2021] [2240485] 0 2240485 198703 8079 352256 0 999 privsep-helper
[Thu Jun 3 15:32:38 2021] [3416294] 42424 3416294 9400600 2912078 24678400 0 999 qemu-system-x86
[Thu Jun 3 15:32:38 2021] [3416305] 42424 3416305 18069141 3605844 30380032 0 999 qemu-system-x86
[Thu Jun 3 15:32:38 2021] [3416858] 0 3416858 12476 133 139264 0 999 sudo
[Thu Jun 3 15:32:38 2021] [3416861] 0 3416861 18869 3922 196608 0 999 neutron-rootwra
[Thu Jun 3 15:32:38 2021] [3416876] 0 3416876 12476 133 139264 0 999 sudo
[Thu Jun 3 15:32:38 2021] [3416878] 0 3416878 18874 3922 192512 0 999 neutron-rootwra
[Thu Jun 3 15:32:38 2021] [3417750] 0 3417750 12476 132 139264 0 999 sudo
[Thu Jun 3 15:32:38 2021] [3417752] 0 3417752 18872 3923 192512 0 999 neutron-rootwra
[Thu Jun 3 15:32:38 2021] [3417827] 0 3417827 3416 32 73728 0 999 ip
[Thu Jun 3 15:32:38 2021] [3417886] 0 3417886 6392 97 98304 0 1000 pgrep
[Thu Jun 3 15:32:38 2021] [3417887] 0 3417887 1431487 21761 942080 0 1000 telegraf
[Thu Jun 3 15:32:38 2021] Out of memory: Kill process 2893751 (qemu-system-x86) score 1171 or sacrifice child
[Thu Jun 3 15:32:38 2021] Killed process 2893751 (qemu-system-x86) total-vm:72102720kB, anon-rss:67608400kB, file-rss:0kB, shmem-rss:20kB

Question information

Language:
English Edit question
Status:
Open
For:
Ubuntu Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:

Can you help with this problem?

Provide an answer of your own, or ask norman shen for more information if necessary.

To post a message you must log in.