ami-852718ec fails to come up with c3 and i2 instances

Eugene, thank you for the bug report.

I have recreated and am attaching console log. This is from ubuntu-precise-12.04-amd64-server-20131003 (ami-b93264d0). This ami boots without the sr-iov attribute which allows me to make changes prior to adding the 'simple' sriov attribute and recreating the problem you are seeing. Removal of the xen_emul_unplug kernel cmdline option and addition of the xen-platform-pci kernel module to the initramfs does not provide relief, therefore I do not believe tha this is related to bug #1268012.

Bug recreates with 12.04.3 ami once the sr-iov 'simple' attribute is set, builds after 12.04.3 deploy with this attribute on by default so they hit the problem on first boot. I saw this failure for quantal (ubuntu-quantal-12.10-amd64-server-20130206 - ami-02df496b) as well but AMIs for raring or newer did not recreate the issue.

(AMI's cited are for us-east-1 where I was testing recreates.)

The network issue was not seen if, after booting a 12.04.3 AMI, I installed the 3.8.0-35 kernel packages from raring and booted with the sr-iov simple attribute.

Modinfo for ixgbevf on quantal (last build exhibiting the issue):
        filename: /lib/modules/3.5.0-23-generic/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko
        version: 2.6.0-k
        description: Intel(R) 82599 Virtual Function Driver
        srcversion: 60A709CBB79F6D372029FAB
        alias: pci:v00008086d00001515sv*sd*bc*sc*i*
        alias: pci:v00008086d000010EDsv*sd*bc*sc*i*
        vermagic: 3.5.0-23-generic SMP mod_unload modversions

Modinfo for ixgbevg from raring kernel which allows precise ami to boot:
        filename: /lib/modules/3.8.0-35-generic/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko
        version: 2.7.12-k
        description: Intel(R) 82599 Virtual Function Driver
        srcversion: 43AF6807120E0559452AE57
        alias: pci:v00008086d00001515sv*sd*bc*sc*i*
        alias: pci:v00008086d000010EDsv*sd*bc*sc*i*
        vermagic: 3.8.0-35-generic SMP mod_unload modversions

If this is just a driver issue, the changes between these two driver versions are listed below.
# git log --graph --decorate --pretty=oneline --abbrev-commit \
 9cd9130~..1b3d2d7 drivers/net/ethernet/intel/ixgbevf/

* 1b3d2d7 ixgbevf: Update version string
* 55fdd45 ixgbevf: fix softirq-safe to unsafe splat on internal mbx_lock
* 6132ee8 ixgbevf: Check for error on dma_map_single call
* f447770 ixgbevf: make netif_napi_add and netif_napi_del symmetric
* 56e9409 ixgbevf: Add VF DCB + SR-IOV support
* c88887e ixgbe/ixgbevf: Limit maximum jumbo frame size to 9.5K to avoid Tx hangs
* 91e2b89 ixgbevf: Set the netdev number of Tx queues
* aecdc33 Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next
|\
| * 2ddc7fe ixgbevf: Return error on failure to enable VLAN
| * 5c60f81 ixgbevf: Add fix to VF to handle multi-descriptor buffers
| * ac6ed8f ixgbevf: Fix AIM (Adaptive Interrupt Moderation)
| * b3d58a8 ixgbevf - Remove unused parameter in ixgbevf_receive_skb
| * 4b2cd27 ixgbevf: Fix code for handling timeout
| * 012dc19 ixgbevf: scheduling while atomic in reset hw path
| * 3118678 ixgbevf: Add support for VF API negotiation
| * dd1fe11 ixgbevf: Cleanup handling of configuration for jumbo frames
| * 0ac1e8c ixgbevf: Add suspend and resume support to the VF
* | 3646f0e netdev: make pci_error_handlers const
|/
* 0614002 netvm: propagate page->pfmemalloc from skb_alloc_page to skb
* ce42260 ixgbevf: Fix namespace issue with ixgbe_write_eitr
* 9f19f31 ixgbevf: Add support for PCI error handling
* 1c55ed7 ixgbevf: Add lock around mailbox ops to prevent simultaneous access
* abaa72d Merge...

Attaching console log output for a successful boot of precise using a raring kernel and sriov 'simple' attribute.

Latest daily precise images (tested on the 20140206) build do not have the sriov 'simple' attribute set and will boot on c3/i2 instances. This is a workaround at this time while the root issue is found. Adding the sriov 'simple' attribute to an instance with this build will result in a network failure as seen with the other builds.

Continuing to work the underlying network failure but c3/i2 instances launched with latest daily builds for precise will boot with networking.

Thank you, Robert!

On Thu, Feb 6, 2014 at 12:39 PM, Robert C Jennings <
<email address hidden>> wrote:

> Latest daily precise images (tested on the 20140206) build do not have
> the sriov 'simple' attribute set and will boot on c3/i2 instances. This
> is a workaround at this time while the root issue is found. Adding the
> sriov 'simple' attribute to an instance with this build will result in a
> network failure as seen with the other builds.
>
> Continuing to work the underlying network failure but c3/i2 instances
> launched with latest daily builds for precise will boot with networking.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1274676
>
> Title:
> ami-852718ec fails to come up with c3 and i2 instances
>
> Status in Official Ubuntu Images on EC2:
> Confirmed
>
> Bug description:
> ami-852718ec fails to come up with c3 and i2 instances with the console
> output below.
> AWS support thinks this is happening because c3 and i2 instances want to
> run with SRIOV networking enabled
> (possibly because ami-852718e had image-attribute sriov set to
> 'simple'?), but ixgbevf-2.11.3 drivers are not installed in the ami.
> ---------------------
> Begin: Mounting root file system ... Begin: Running /scripts/local-top
> ... done.
> Begin: Running /scripts/local-premount ... done.
> [ 2.112634] EXT4-fs (sda1): mounted filesystem with ordered data
> mode. Opts: (null)
> Begin: Running /scripts/local-bottom ... done.
> done.
> Begin: Running /scripts/init-bottom ... done.
> [ 11.425236] EXT4-fs (sda1): re-mounted. Opts: (null)
> [ 12.922186] kjournald starting. Commit interval 5 seconds
> [ 12.929710] EXT3-fs (xvdb): using internal journal
> [ 12.932988] EXT3-fs (xvdb): mounted filesystem with ordered data mode
> cloud-init start-local running: Tue, 28 Jan 2014 17:55:22 +0000. up
> 22.99 seconds
> no instance data found in start-local
> cloud-init-nonet waiting 120 seconds for a network device.
> cloud-init-nonet gave up waiting for a network device.
> ci-info: lo : 1 127.0.0.1 255.0.0.0 .
> ci-info: eth0 : 1 . . 12:c1:85:bb:da:87
> route_info failed
> ---------------------
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu-on-ec2/+bug/1274676/+subscriptions
>

Bisecting kernel gave bad results, recording here so that I can get another pair of eyes on things. Started with reverse bisect between v3.5 and v3.8 mainline source limited to changes on the ixgbevf source. Bisection indicated that the "fix" was a cleanup patch. Given that this was not the issue, I used the last tested boundaries from that as the input to a full kernel tree bisection which lead me to another 'fix' which only changes the formatting of debug printk in the tun driver.

Notes from debug:
- using ubuntu/images-testing/hvm/ubuntu-precise-daily-amd64-server-20140210 because it boots without sriov 'simple' attribute
- performing kernel bisect on mainline from v3.5..v3.8 limited to driver source ('git bisect start v3.8 v3.5 -- drivers/net/ethernet/intel/ixgbevf/')

$ git bisect log
# bad: [19f949f52599ba7c3f67a5897ac6be14bfcb1200] Linux 3.8
# good: [28a33cbc24e4256c143dce96c7d93bf423229f92] Linux 3.5
git bisect start 'v3.8' 'v3.5' '--' 'drivers/net/ethernet/intel/ixgbevf/'
# good: [5c60f81a2553213856b3bb80f18003e56a6a110d] ixgbevf: Add fix to VF to handle multi-descriptor buffers
git bisect good 5c60f81a2553213856b3bb80f18003e56a6a110d
# good: [810b6d7638a288216f99bd190470d67061c8bd88] Merge branch 'master' of git://git.kernel.org/pub/scm/linux/kernel/git/jkirsher/net-next
git bisect good 810b6d7638a288216f99bd190470d67061c8bd88
# bad: [6259a01fb2d458b4157918b2da0d0f42242a9d72] ixgbevf: Fix unnecessary dereference where local var is available.
git bisect bad 6259a01fb2d458b4157918b2da0d0f42242a9d72
# bad: [a5f9337bdc45bb8c13037bdb70d16fd9017bb83a] ixgbevf: fix possible use of uninitialized variable
git bisect bad a5f9337bdc45bb8c13037bdb70d16fd9017bb83a
# bad: [46acc460c07b5c74287560a00b6cbc6111136ab6] eth: Make is_link_local() consistent with other address tests
git bisect bad 46acc460c07b5c74287560a00b6cbc6111136ab6
# first bad commit: [46acc460c07b5c74287560a00b6cbc6111136ab6] eth: Make is_link_local() consistent with other address tests
==============================================================================
46acc460c07b5c74287560a00b6cbc6111136ab6 is the first bad commit
commit 46acc460c07b5c74287560a00b6cbc6111136ab6
Author: Ben Hutchings <email address hidden>
Date: Thu Nov 1 09:11:11 2012 +0000

    eth: Make is_link_local() consistent with other address tests

    Function name should include '_ether_addr'.
    Return type should be bool.
    Parameter name should be 'addr' not 'dest' (also matching kernel-doc).

    Signed-off-by: Ben Hutchings <email address hidden>
    Acked-by: John Fastabend <email address hidden>
    Signed-off-by: David S. Miller <email address hidden>

- Used last good/bad from prior search as input to a bisect operation on the full tree source ('git bisect start 46acc46 810b6d7')
$ git bisect log
# bad: [46acc460c07b5c74287560a00b6cbc6111136ab6] eth: Make is_link_local() consistent with other address tests
# good: [810b6d7638a288216f99bd190470d67061c8bd88] Merge branch 'master' of git://git.kernel.org/pub/scm/linux/kernel/git/jkirsher/net-next
git bisect start '46acc46' '810b6d7'
# bad: [9750a3ade7b635a18f04371b4fddad0de0b4e6d8] cpsw: support both silicon versions
git bise...

Hi Robert, it is possible, due to the non-linearity of git trees, that starting with the previously obtained range will not work. Especially as one of them if a merge. So unfortunately it might be necessary to do the bisect over the full tree from the start.
I suspect that the fix, although it results in the driver working or not, is rather in a different place. Could be mmu or pci. In fact there seem to be two patches between 3.5 and 3.8 that mention SR-IOV (and both are in the pci related parts):

commit c3cb4709809e655a4ba5a716086c8bc5bbbbccdb
Author: Jan Beulich <email address hidden>
Date: Tue Sep 18 12:29:03 2012 +0100

    xen-pciback: support wild cards in slot specifications

    Particularly for hiding sets of SR-IOV devices, specifying them all
    individually is rather cumbersome. Therefore, allow function and slot
    numbers to be replaced by a wildcard character ('*').

    Unfortunately this gets complicated by the in-kernel sscanf()
    implementation not being really standard conformant - matching of
    plain text tails cannot be checked by the caller (a patch to overcome
    this will be sent shortly, and a follow-up patch for simplifying the
    code is planned to be sent when that fixed went upstream).

    Signed-off-by: Jan Beulich <email address hidden>
    Signed-off-by: Konrad Rzeszutek Wilk <email address hidden>

commit 8a5248fe10b101104d92d01438f918e899414fd1
Author: Laszlo Ersek <email address hidden>
Date: Wed Oct 17 11:55:55 2012 +0200

    xen PV passthru: assign SR-IOV virtual functions to separate virtual slots

    VFs are reported as single-function devices in PCI_HEADER_TYPE, which
    causes pci_scan_slot() in the PV domU to skip all VFs beyond #0 in the
    pciback-provided slot. Avoid this by assigning each VF to a separate
    virtual slot.

    Acked-by: Jan Beulich <email address hidden>
    Signed-off-by: Laszlo Ersek <email address hidden>
    Signed-off-by: Konrad Rzeszutek Wilk <email address hidden>

Stefan, thanks for the update, that gives me what I need to continue tracking this down.

Thanks to information provided by SkyShard for Question #243761 this issue can be worked around by doing:
  "sudo update-initramfs -c -k all"
after loading the driver with modprobe ixgbevf.

This loads the correct ixgbevf driver on boot - the one you installed with the modprobe command - in my case this worked with version 2.11.3
For a complete list of how to enable the sr-iov see: https://gist.github.com/CBarraford/8850424

The "Enabling Enhanced Networking on Linux Instances in a VPC" information at http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking.html needs to be updated to include the line to update the init ramfs.

Following up on Dan Krent's update in comment #9.

Running 'update-initramfs' will not resolve the issue for 12.04LTS images. On 12.04 the 3.2 kernel will include version 2.2.0-k of the Intel ixgbevf module. 2.2.0-k is insufficient for supporting the SR-IOV setup in this environment. A kernel with a newer version of the driver is required.

Finished bisecting the kernel today and found the culprit:

commit 56e94095efb3d4f749212bf7c0b151843d157f49
Author: Alexander Duyck <email address hidden>
Date: Fri Jul 20 08:10:03 2012 +0000

    ixgbevf: Add VF DCB + SR-IOV support

    This change adds support for DCB and SR-IOV from the VF. With this change
    in place the VF will correctly use a traffic class other than 0 in the case
    that the PF is configured with the default user priority belonging to a
    traffic class other than 0.

    Cc: Greg Rose <email address hidden>
    Signed-off-by: Alexander Duyck <email address hidden>
    Tested-by: Sibai Li <email address hidden>
    Signed-off-by: Jeff Kirsher <email address hidden>

The challenge is that this commit is stacked on 53 additional patches to the driver in 12.04 which is a massive delta.

I am not recommending back-porting this large set of driver patches to the precise kernel. The correct solution would be to install the 'linux-image-generic-lts-saucy' and then modify the ec2 instance attributes to enable SR-IOV. This hardware enablement kernel brings the necessary features for this driver in a supported fashion (https://wiki.ubuntu.com/Kernel/LTSEnablementStack).

Solution: For SR-IOV in an EC2 environment with a 12.04 image, the most recent hardware enablement kernel should be used. As of today, that kernel is linux-image-generic-lts-saucy, see https://wiki.ubuntu.com/Kernel/LTSEnablementStack for updates.

Once the kernel is installed (sudo apt-get install linux-image-generic-lts-saucy) the EC2 instance must be stopped, modified to enable SR-IOV ('ec2-modify-instance-attributes <instance> --sriov simple') and restarted.

Closed as "Won't fix" given this solution.

Appears that the linux-image-generic-lts-saucy kernel will stop receiving security updates soon- is this correct?

 'sudo apt-get install linux-image-generic-lts-trusty' is working for me now.

Jack, that is correct. The latest supported HWE kernel will be linux-image-generic-lts-trusty; the saucy kernel is reaching EOL shortly.