Fuel for OpenStack

InfiniBand ConnectX-3 not detected by Fuel 6.1 (nightly) bootstrap

Asked by Gennaro Capuozzo

I've tried to install the latest nightly build of 6.1, since it was supposed to support native IB networking.

My hardware is: DELL C6220 II nodes with ConnectX-3, Mellanox SX6036 (IB only) switch, GbE networking.
I'm running opensmd on a spare node, as suggested in a couple of tutorials found on Mellanox community website.

After bootstrapping the nodes, when I click the cog wheel on Fuel dashboard, I just get the two GbE adapters with no evidence of "ib0" nor "eth_ipoib".

If I manually configure the "ib0" adapters (insmod ib_ipoib and ifconfig), I can ping the hosts over IB without any problem.

It could be a "bootstrap" issue in detecting/configuring my adapters (15b3:1003 - Mellanox Technologies MT27500 Family [ConnectX-3]) .

Thank you in advance for any useful advice.
GC

Question information

Language:
English Edit question
Status:
Solved
For:
Fuel for OpenStack Edit question
Assignee:
No assignee Edit question
Solved by:
Gennaro Capuozzo
Solved:
Last query:
Last reply:
Revision history for this message
Evgeny Kozhemyakin (ekozhemyakin) said : 		#1

Hello,

The module is not managed by udev.
And it seems to be reasonable that IPoIB isn't loaded by default.

In order to get it loaded in boot time you should rebuild bootstrap image
adding instruction to load the module and so on.

You can find how to customize bootstrap here:
https://docs.fuel-infra.org/fuel-dev/develop/custom-bootstrap-node.html
Please ping us if you need any assistance with customization.

BWT, after setting IPoIB manually does fuel recognize the adapter?
Do you see ib0 in node's configuration in fuel UI?

Regards,

Revision history for this message
Gennaro Capuozzo (gencap) said : 		#2

Hi there,

It seems that the solution for my problem will be the "Mellanox Fuel Plugin" for Fuel 6.1.
While the plugin hasn't been released (yet) and it's due after the scheduled 6.1 GA, I was interested in early checking the correct handling of my ConnectX-3 (IB only) adapters.

I haven't compiled the entire plugin (https://github.com/stackforge/fuel-plugin-mellanox), but I've managed to replace the initramfs.img available in the standard bootstrap with a custom one, downloaded from
http://www.mellanox.com/repository/solutions/openstack/mellanox_fuel_plugin/
The kernel present in the same directory won't boot, but the original one is OK.

The new bootstrap is able to detect my IB adapter and here's the "Cog Wheel -> Interfaces" output (note the eth2 interface):

name eth0
mac 34:17:EB:xx:xx:xx
state up
ip 10.20.0.3
netmask 255.255.255.0
current speed 1.0 Gbps
max speed 1.0 Gbps
driver igb
bus info 0000:02:00.0

name eth1
mac 34:17:EB:xx:xx:xx
state down
current speed N/A
max speed 1.0 Gbps
driver igb
bus info 0000:02:00.3

name eth2
mac F4:52:14:xx:xx:xx
state up
current speed 56.0 Gbps
max speed 10.0 Gbps
driver eth_ipoib
bus info ib0

The list of the available interfaces, when logged in the test node, includes several ib0.8xxx.1 entries, one for each PKEY/VLAN configured on my spare machine running opensmd.

It should be the right direction...

Anyway, I am not interested in going further with this nightly build, since the stable 6.1 will be ready in a few days.
I'm looking forward to installing the stable "Mellanox Fuel Plugin" too...

GC