Ubuntu on EC2

Bug #1274676
Comment #2

Comment 2 for bug 1274676

Revision history for this message

Robert C Jennings (rcj) wrote on 2014-02-04:

Bug recreates with 12.04.3 ami once the sr-iov 'simple' attribute is set, builds after 12.04.3 deploy with this attribute on by default so they hit the problem on first boot. I saw this failure for quantal (ubuntu-quantal-12.10-amd64-server-20130206 - ami-02df496b) as well but AMIs for raring or newer did not recreate the issue.

(AMI's cited are for us-east-1 where I was testing recreates.)

The network issue was not seen if, after booting a 12.04.3 AMI, I installed the 3.8.0-35 kernel packages from raring and booted with the sr-iov simple attribute.

Modinfo for ixgbevf on quantal (last build exhibiting the issue):
        filename: /lib/modules/3.5.0-23-generic/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko
        version: 2.6.0-k
        description: Intel(R) 82599 Virtual Function Driver
        srcversion: 60A709CBB79F6D372029FAB
        alias: pci:v00008086d00001515sv*sd*bc*sc*i*
        alias: pci:v00008086d000010EDsv*sd*bc*sc*i*
        vermagic: 3.5.0-23-generic SMP mod_unload modversions

Modinfo for ixgbevg from raring kernel which allows precise ami to boot:
        filename: /lib/modules/3.8.0-35-generic/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko
        version: 2.7.12-k
        description: Intel(R) 82599 Virtual Function Driver
        srcversion: 43AF6807120E0559452AE57
        alias: pci:v00008086d00001515sv*sd*bc*sc*i*
        alias: pci:v00008086d000010EDsv*sd*bc*sc*i*
        vermagic: 3.8.0-35-generic SMP mod_unload modversions

If this is just a driver issue, the changes between these two driver versions are listed below.
# git log --graph --decorate --pretty=oneline --abbrev-commit \
9cd9130~..1b3d2d7 drivers/net/ethernet/intel/ixgbevf/

* 1b3d2d7 ixgbevf: Update version string
* 55fdd45 ixgbevf: fix softirq-safe to unsafe splat on internal mbx_lock
* 6132ee8 ixgbevf: Check for error on dma_map_single call
* f447770 ixgbevf: make netif_napi_add and netif_napi_del symmetric
* 56e9409 ixgbevf: Add VF DCB + SR-IOV support
* c88887e ixgbe/ixgbevf: Limit maximum jumbo frame size to 9.5K to avoid Tx hangs
* 91e2b89 ixgbevf: Set the netdev number of Tx queues
* aecdc33 Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next
|\
| * 2ddc7fe ixgbevf: Return error on failure to enable VLAN
| * 5c60f81 ixgbevf: Add fix to VF to handle multi-descriptor buffers
| * ac6ed8f ixgbevf: Fix AIM (Adaptive Interrupt Moderation)
| * b3d58a8 ixgbevf - Remove unused parameter in ixgbevf_receive_skb
| * 4b2cd27 ixgbevf: Fix code for handling timeout
| * 012dc19 ixgbevf: scheduling while atomic in reset hw path
| * 3118678 ixgbevf: Add support for VF API negotiation
| * dd1fe11 ixgbevf: Cleanup handling of configuration for jumbo frames
| * 0ac1e8c ixgbevf: Add suspend and resume support to the VF
* | 3646f0e netdev: make pci_error_handlers const
|/
* 0614002 netvm: propagate page->pfmemalloc from skb_alloc_page to skb
* ce42260 ixgbevf: Fix namespace issue with ixgbe_write_eitr
* 9f19f31 ixgbevf: Add support for PCI error handling
* 1c55ed7 ixgbevf: Add lock around mailbox ops to prevent simultaneous access
* abaa72d Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net
|\
| * 10cc1bd ixgbevf: Fix panic when loading driver
| * 9365976 ixgbevf: Prevent RX/TX statistics getting reset to zero
* | eb022d0 ixgbevf: Fix multiple issues in ixgbevf_get/set_ringparam
* | 70a10e2 ixgbevf: Consolidate Tx context descriptor creation code
* | fb40195 ixgbevf: Add netdev to ring structure
* | 18c6308 ixgbevf: Do not rewind the Rx ring before bumping tail
* | 5d9a533 ixgbevf: fix VF untagging when 802.1 prio is set
* | 908421f ixgbevf: Update descriptor macros to accept pointers and drop _ADV suffix
* | 5f3600e ixgbevf: Use igb style interrupt masks instead of ixgbe style
* | fa71ae2 ixgbevf: Move Tx clean-up into NAPI context
* | 6b43c44 ixgbevf: Update q_vector to contain ring pointers instead of bitmaps
* | 3595990 ixgbevf: Cleanup accounting for space needed at start of xmit_frame
* | e2c28ce ixgbevf: Drop use of eitr_low and eitr_high for hard coded values
* | 525a940 ixgbevf: Make use of NETIF_F_RXCSUM instead of keeping our own flag
* | fd13a9a ixgbevf: Drop netdev_registered value since that is already stored in netdev
* | 77d5dfc ixgbevf: Drop all dead or unnecessary code
* | 49ce9c2 drivers/net/ethernet: Fix (nearly-)kernel-doc comments for various functions
|/
* 9cd9130 ixgbevf: Update version string

Bug recreates with 12.04.3 ami once the sr-iov 'simple' attribute is set, builds after 12.04.3 deploy with this attribute on by default so they hit the problem on first boot.  I saw this failure for quantal (ubuntu-quantal-12.10-amd64-server-20130206 - ami-02df496b) as well but AMIs for raring or newer did not recreate the issue.

(AMI's cited are for us-east-1 where I was testing recreates.)

The network issue was not seen if, after booting a 12.04.3 AMI, I installed the 3.8.0-35 kernel packages from raring and booted with the sr-iov simple attribute.

Modinfo for ixgbevf on quantal (last build exhibiting the issue):
        filename:       /lib/modules/3.5.0-23-generic/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko
        version:        2.6.0-k
        description:    Intel(R) 82599 Virtual Function Driver
        srcversion:     60A709CBB79F6D372029FAB
        alias:          pci:v00008086d00001515sv*sd*bc*sc*i*
        alias:          pci:v00008086d000010EDsv*sd*bc*sc*i*
        vermagic:       3.5.0-23-generic SMP mod_unload modversions

Modinfo for ixgbevg from raring kernel which allows precise ami to boot:
        filename:       /lib/modules/3.8.0-35-generic/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko
        version:        2.7.12-k
        description:    Intel(R) 82599 Virtual Function Driver
        srcversion:     43AF6807120E0559452AE57
        alias:          pci:v00008086d00001515sv*sd*bc*sc*i*
        alias:          pci:v00008086d000010EDsv*sd*bc*sc*i*
        vermagic:       3.8.0-35-generic SMP mod_unload modversions

If this is just a driver issue, the changes between these two driver versions are listed below.
# git log --graph --decorate --pretty=oneline --abbrev-commit \
 9cd9130~..1b3d2d7 drivers/net/ethernet/intel/ixgbevf/

* 1b3d2d7 ixgbevf: Update version string
* 55fdd45 ixgbevf: fix softirq-safe to unsafe splat on internal mbx_lock
* 6132ee8 ixgbevf: Check for error on dma_map_single call
* f447770 ixgbevf: make netif_napi_add and netif_napi_del symmetric
* 56e9409 ixgbevf: Add VF DCB + SR-IOV support
* c88887e ixgbe/ixgbevf: Limit maximum jumbo frame size to 9.5K to avoid Tx hangs
* 91e2b89 ixgbevf: Set the netdev number of Tx queues
*   aecdc33 Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next
|\
| * 2ddc7fe ixgbevf: Return error on failure to enable VLAN
| * 5c60f81 ixgbevf: Add fix to VF to handle multi-descriptor buffers
| * ac6ed8f ixgbevf: Fix AIM (Adaptive Interrupt Moderation)
| * b3d58a8 ixgbevf - Remove unused parameter in ixgbevf_receive_skb
| * 4b2cd27 ixgbevf: Fix code for handling timeout
| * 012dc19 ixgbevf: scheduling while atomic in reset hw path
| * 3118678 ixgbevf: Add support for VF API negotiation
| * dd1fe11 ixgbevf: Cleanup handling of configuration for jumbo frames
| * 0ac1e8c ixgbevf: Add suspend and resume support to the VF
* | 3646f0e netdev: make pci_error_handlers const
|/
* 0614002 netvm: propagate page->pfmemalloc from skb_alloc_page to skb
* ce42260 ixgbevf: Fix namespace issue with ixgbe_write_eitr
* 9f19f31 ixgbevf: Add support for PCI error handling
* 1c55ed7 ixgbevf: Add lock around mailbox ops to prevent simultaneous access
*   abaa72d Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net
|\
| * 10cc1bd ixgbevf: Fix panic when loading driver
| * 9365976 ixgbevf: Prevent RX/TX statistics getting reset to zero
* | eb022d0 ixgbevf: Fix multiple issues in ixgbevf_get/set_ringparam
* | 70a10e2 ixgbevf: Consolidate Tx context descriptor creation code
* | fb40195 ixgbevf: Add netdev to ring structure
* | 18c6308 ixgbevf: Do not rewind the Rx ring before bumping tail
* | 5d9a533 ixgbevf: fix VF untagging when 802.1 prio is set
* | 908421f ixgbevf: Update descriptor macros to accept pointers and drop _ADV suffix
* | 5f3600e ixgbevf: Use igb style interrupt masks instead of ixgbe style
* | fa71ae2 ixgbevf: Move Tx clean-up into NAPI context
* | 6b43c44 ixgbevf: Update q_vector to contain ring pointers instead of bitmaps
* | 3595990 ixgbevf: Cleanup accounting for space needed at start of xmit_frame
* | e2c28ce ixgbevf: Drop use of eitr_low and eitr_high for hard coded values
* | 525a940 ixgbevf: Make use of NETIF_F_RXCSUM instead of keeping our own flag
* | fd13a9a ixgbevf: Drop netdev_registered value since that is already stored in netdev
* | 77d5dfc ixgbevf: Drop all dead or unnecessary code
* | 49ce9c2 drivers/net/ethernet: Fix (nearly-)kernel-doc comments for various functions
|/
* 9cd9130 ixgbevf: Update version string