kvm_intel (nested) module will not load [Input/output error]

This change was made by a bot.

To show that this issue is actually kernel (rather than host kernel not having nested enabled), i booted into trusty kernel on the same 'system 1' above.
$ uname -r
3.13.0-46-generic
$ dpkg -S /boot/vmlinuz-$(uname -r)
linux-image-3.13.0-46-generic: /boot/vmlinuz-3.13.0-46-generic
$ lsmod | grep kvm
kvm_intel 143187 0
kvm 455835 1 kvm_intel

$ kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used

Also just verified that utopic kernel (3.16) also does not have /dev/kvm here.
So, short summary:

vmlinuz-3.13.0-46-generic - kvm_intel loads
vmlinuz-3.16.0-31-generic - Input/output error
vmlinuz-3.19.0-7-generic - Input/output error

Reproduced with latest upstream kernel from:

http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.0-rc4-vivid/
linux-image-4.0.0-040000rc4-generic 4.0.0-040000rc4.201503152135
Linux vivid-test 4.0.0-040000rc4-generic #201503152135 SMP Mon Mar 16 01:36:37 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Does not reproduce (what I want) with:

http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.13.11.11-trusty/
linux-image-3.13.11-03131111-generic 3.13.11-03131111.201411111336
Linux vivid-test 3.13.11-03131111-generic #201411111336 SMP Tue Nov 11 18:37:52 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

All on Vivid, using cloud image daily build on amd64 serial 20150314.1. Canonical engineers: it seems you can reproduce on lcy01.

$ cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Core(TM)2 Duo CPU T7700 @ 2.40GHz
stepping : 11
microcode : 0x1
cpu MHz : 2400.084
cache size : 4096 KB
physical id : 0
siblings : 1
core id : 0
cpu cores : 1
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc rep_good nopl pni vmx ssse3 cx16 sse4_1 sse4_2 popcnt hypervisor lahf_lm
bogomips : 4800.16
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Core(TM)2 Duo CPU T7700 @ 2.40GHz
stepping : 11
microcode : 0x1
cpu MHz : 2400.084
cache size : 4096 KB
physical id : 1
siblings : 1
core id : 0
cpu cores : 1
apicid : 1
initial apicid : 1
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc rep_good nopl pni vmx ssse3 cx16 sse4_1 sse4_2 popcnt hypervisor lahf_lm
bogomips : 4800.16
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

Workaround:

1. Install http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.13.11.11-trusty/linux-image-3.13.11-03131111-generic_3.13.11-03131111.201411111336_amd64.deb

2. Edit /etc/default/grub and set:

GRUB_DEFAULT="Advanced options for Ubuntu>Ubuntu, with Linux 3.13.11-03131111-generic"

3. Run "sudo update-grub"

4. Reboot.

Can you confirm cpu of system 1 is really a core2duo or whether it plays some tricks there?

Also, can you attach the libvirt XML you are using?

Also, what is the L0 host running? Trusty 3.13 or something else?

I think this is a problem very much tied to the raw host cpu type and the masking of that as core2duo for the guest. I just tried with a current Trusty host with a current Vivid guest masked to core2duo:

host cpu:
processor : 7
vendor_id : GenuineIntel
cpu family : 6
model : 58
model name : Intel(R) Core(TM) i7-3770K CPU @ 3.50GHz
stepping : 9
microcode : 0x19
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms

libvirt setting:
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>core2duo</model>
    <vendor>Intel</vendor>
  </cpu>

guest cpu:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Core(TM)2 Duo CPU T7700 @ 2.40GHz
stepping : 11
microcode : 0x1
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx lm constant_tsc rep_good nopl pni vmx ssse3 cx16 x2apic hypervisor lahf_lm vnmi ept

There is some difference of what the guest sees on Scott's host compared to mine:
--- cpuflags-smb 2015-03-17 10:38:33.477207501 +0100
+++ cpuflags-smoser 2015-03-17 10:38:59.741208228 +0100
@@ -23,6 +23,7 @@
 ss
 syscall
 nx
+rdtscp
 lm
 constant_tsc
 rep_good
@@ -31,8 +32,8 @@
 vmx
 ssse3
 cx16
-x2apic
+sse4_1
+sse4_2
+popcnt
 hypervisor
 lahf_lm
-vnmi
-ept

I should also say that for me the guest was able to load the kvm-intel module.

In reponse to comment 7 and 8:

The host instance-0000e403 runs on is sapodilla. It runs a 12.04 ubuntu with a 3.2.0-60-generic x86_64 kernel.

attached is the /proc/cpuinfo

in response to comment 7 "can you attach the libvirt XML you are using?"

I was able to reproduce the bug on another type of processor (core i7).
/dev/kvm is not available with vivid kernel but available with trusty kernel.

ubuntu@vivid-adt3:~$ cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 2
model name : Intel Core i7 9xx (Nehalem Class Core i7)
stepping : 3
microcode : 0x1
cpu MHz : 2400.084
cache size : 2048 KB
physical id : 0
siblings : 1
core id : 0
cpu cores : 1
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 ss syscall nx rdtscp lm rep_good nopl pni vmx ssse3 cx16 sse4_1 sse4_2 popcnt hypervisor lahf_lm
bugs :
bogomips : 4800.16
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

ubuntu@vivid-adt3:~$ ls -l /dev/kvm
ls: cannot access /dev/kvm: No such file or directory

ubuntu@vivid-adt3:~$ sudo modprobe kvm_intel
modprobe: ERROR: could not insert 'kvm_intel': Input/output error

ubuntu@vivid-adt3:~$ uname -a
Linux vivid-adt3 3.19.0-9-generic #9-Ubuntu SMP Wed Mar 11 17:50:03 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Joseph, Stefan: I can give you access to the guest on lcy01 that exhibits the problem if you need it.

So I think with the info above it should be possible to figure things out. I am pretty sure the secret lies in one of those flags. Or combination of them. But at least we have a small collection of working and not working now.

Oh... maybe for Oleg: what release/kernel is running on your host?

> Oh... maybe for Oleg: what release/kernel is running on your host?
I'm experimenting with canonistack (lcy_02) that's why I don't have such info as well as some other guys in the thread.
Sorry for not mentioning canonistack in my comment.

I did a quick analysis if the flag (doublecheck might be needed).
All systems which WORK CORRECTLY has these guest flags:
x2apic vnmi ept

Systems which fail don't have any of these flags.

x2apic -- improves guest performance by reducing the overhead of APIC access, which is used to program timers and for issuing inter-processor interrupts
vnmi -- Intel Virtual NMI feature which helps with certain sorts of interrupt events in guests
ept -- Intel Extended Page Table

I wonder if it's possible to change livbirt config to block some of these flags even if they are provided by host CPU.

Thanks Oleg. I guess it being canonistack we probably can assume Precise on the host. Based on that assumption I also did some tests. And there found that it is indeed a combination of host kernel / guest kernel which causes this. Looking at the kernel code there are many places that can actually exit with -EIO (sadly all without giving any hint about the specific mismatch that is causing the failure). Those mostly seem to be concerned with checking MSRs about correct configuration. So the cpuid flags may be different but at the same time could be just fluke. There should be ways to influence those in libvirt but now I think this may not help.

So my finding on the same Hashwell/Sharbay system:
- host 3.2: guest 3.2: ok
- host 3.2: guest 3.16: not ok
- host 3.13: guest: 3.16: ok

So this seems to be some mismatch in what the hosts kvm passes into the L1 guests vmx msr which in turn the L1 kvm module validates on load. But since I have now a setup combination that triggers the issue I can prepare a more verbose kvm-intel module to try locally.

Ok, so 3.16 on 3.12 fails (there might be more but first failure exits) with:

MSR_IA32_VMX_EXIT_CTLS (0x0483) which is required to support at least
  VM_EXIT_SAVE_DEBUG_CONTROLS (=0x002)|
  VM_EXIT_HOST_ADDR_SPACE_SIZE (0x0200)
but evaluating the msr returns only support for
  VM_EXIT_HOST_ADDR_SPACE_SIZE

The requirement got introduced in 3.15 by:

KVM: vmx: Allow the guest to run with dirty debug registers

Right now I am not sure where this needs fixing (the older or the newer kernels). I better ask that upstream.

Mirroring the feedback I got on the mailing list. Potentially we could add the missing bit to the required part of the 3.2 kvm-intel module. But then the comments about support and security make me wonder whether we really want to. There seems to be quite a bit of work to get even 3.13 (Trusty) into shape. Leave alone 3.2...

Paolo Bonzini wrote:
> Because if we wanted to make 3.14 nested VMX stable-ish we would need
> several more, at least these:
>
> KVM: nVMX: fix lifetime issues for vmcs02
> KVM: nVMX: clean up nested_release_vmcs12 and code around it
> KVM: nVMX: Rework interception of IRQs and NMIs
> KVM: nVMX: Do not inject NMI vmexits when L2 has a pending
> interrupt
> KVM: nVMX: Disable preemption while reading from shadow VMCS
>
> and for 3.13:
>
> KVM: nVMX: Leave VMX mode on clearing of feature control MSR
>
> There are also several L2-crash-L1 bugs too in Nadav Amit's patches.
>
> Basically, nested VMX was never considered stable-worthy. Perhaps
> that can change soon---but not retroactively.
>
> So I'd rather avoid giving false impressions of the stability of nVMX
> in 3.14.
>
> Even if we considered nVMX stable, I'd _really_ not want to consider
> the L1<->L2 boundary a secure one for a longer time.

Darn, I think I got lost in version numbers. While the change mentioned in comment #21 is causing the module load to fail, it cannot be helping to make this present in L1. Because 3.13 did work with a 3.16 kernel. It was only 3.2 that failed. And the change was in 3.15. So it is something else that causes both features to be set as supported in 3.13 but not in 3.2. Only before 3.15 this cannot be noticed.

"rdmsr -0 0x483" on
- host 3.13: 0x007fffff00036dff guest 3.16: 0x007f6fff00036dff
- host 3.02: 0x007fffff00036dff guest 3.16: 0x0000020000000000

Thats a few too many zeros especially in the high word.

So I found (and could verify) that it is actually this change from 3.10 which fixes nested for L1 guests >=3.15:

commit 33fb20c39e98b90813b5ab2d9a0d6faa6300caca
Author: Jan Kiszka <email address hidden>
Date: Wed Mar 6 15:44:03 2013 +0100

    KVM: nVMX: Fix content of MSR_IA32_VMX_ENTRY/EXIT_CTLS

And that change looks not too intrusive and to be rather a fix than adding any feature. It also cleanly cherry-picks into 3.2.

Brilliant job Stefan!
Thanks for helping.

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-precise' to 'verification-done-precise'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

I will do appropriate verification on Monday.
Thanks for notification.

Kernel from -proposed (3.2.0-80-generic) resolves the bug.

L0: 12.04 running 3.2.0-80-generic
L1: 14.10 (utopic) -> /dev/kvm is available
L2: 14.04 (trusty) -> I was able to bootstrap the instance and ssh into it

L0: 12.04 running 3.2.0-80-generic
L1: 15.04 (vivid) -> /dev/kvm is available
L2: 14.04 (trusty) -> I was able to bootstrap the instance and ssh into it

Results:
(1) /dev/kvm exists for both L1/utopic and L1/vivid
(2) nested virtualization is functional and L2 guests can be bootstrapped

Thanks to everyone!
Special thanks to Stefan for providing us with the fix.

This bug was fixed in the package linux - 3.2.0-80.116

---------------
linux (3.2.0-80.116) precise; urgency=low

  [ Brad Figg ]

  * Release Tracking Bug
    - LP: #1435392

  [ Andy Whitcroft ]

  * [Packaging] generate live watchdog blacklists
    - LP: #1432837

  [ Upstream Kernel Changes ]

  * Drivers: hv: vmbus: incorrect device name is printed when child device
    is unregistered
    - LP: #1417313
  * x86, mm/ASLR: Fix stack randomization on 64-bit systems
    - LP: #1423757
    - CVE-2015-1593
  * net: llc: use correct size for sysctl timeout entries
    - LP: #1425271
    - CVE-2015-2041
  * net: rds: use correct size for max unacked packets and bytes
    - LP: #1425274
    - CVE-2015-2042
  * PCI: quirks: Fix backport of quirk_io()
    - LP: #1434639
  * MIPS: IRQ: Fix disable_irq on CPU IRQs
    - LP: #1434639
  * ASoC: atmel_ssc_dai: fix start event for I2S mode
    - LP: #1434639
  * ALSA: ak411x: Fix stall in work callback
    - LP: #1434639
  * lib/checksum.c: fix carry in csum_tcpudp_nofold
    - LP: #1434639
  * lib/checksum.c: fix build for generic csum_tcpudp_nofold
    - LP: #1434639
  * caif: remove wrong dev_net_set() call
    - LP: #1434639
  * MIPS: Fix kernel lockup or crash after CPU offline/online
    - LP: #1434639
  * gpio: sysfs: fix memory leak in gpiod_export_link
    - LP: #1434639
  * gpio: sysfs: fix memory leak in gpiod_sysfs_set_active_low
    - LP: #1434639
  * net: sctp: fix passing wrong parameter header to param_type2af in
    sctp_process_param
    - LP: #1434639
  * mm: pagewalk: call pte_hole() for VM_PFNMAP during walk_page_range
    - LP: #1434639
  * nilfs2: fix deadlock of segment constructor over I_SYNC flag
    - LP: #1434639
  * staging: comedi: cb_pcidas64: fix incorrect AI range code handling
    - LP: #1434639
  * media/rc: Send sync space information on the lirc device
    - LP: #1434639
  * sched/rt: Reduce rq lock contention by eliminating locking of
    non-feasible target
    - LP: #1434639
  * time: adjtimex: Validate the ADJ_FREQUENCY values
    - LP: #1434639
  * ntp: Fixup adjtimex freq validation on 32-bit systems
    - LP: #1434639
  * ipv6: fib: fix fib dump restart
    - LP: #1434639
  * ipv6: fib: fix fib dump restart
    - LP: #1434639
  * Bluetooth: ath3k: workaround the compatibility issue with xHCI
    controller
    - LP: #1400215, #1434639
  * Linux 3.2.68
    - LP: #1434639
  * KVM: nVMX: Fix content of MSR_IA32_VMX_ENTRY/EXIT_CTLS
    - LP: #1431473
 -- Brad Figg <email address hidden> Mon, 23 Mar 2015 08:41:45 -0700