Infiniband support for storage network

You will need to add the adapters firmware to the bootstrap image that is used to discover the node, you can do this by building your own iso from https://github.com/stackforge/fuel-main and editing this https://github.com/stackforge/fuel-main/blob/master/bootstrap/module.mk to install the firmware. You will also want to add this package to requirements-rpm.txt and requirements-deb.txt so that they are included in the mirrors that the iso produces. Finally, you will need to tweak the cobbler preseed file (or anaconda) so that they will include the driver during OS install time. These are found under build/repos/fuellib/deployment/puppet/cobbler/templates/preseed or build/repos/fuellib/deployment/puppet/cobbler/templates/kickstart after you do a 'make build_repos' from the root of the git folder.

If you run into any problems making the iso please feel free to reach out to us on IRC

Thank you for the prompt reply!

So you mean that I have to rebuild the Fuel install ISO and reinstall my master node with the new one?

Actually both bootstrap and install PXE images of the current Fuel ISO already have all required driver modules:

mlx4_ib
ib_sa
ib_cm
ib_umad
ib_addr
ib_uverbs
ib_ipoib

So can I patch the PXE images to modprobe them at boot without rebuilding the whole ISO and keeping my existing master node installation?

And I just wonder if this will be enough for the Fuel GUI or CLI to see and manage those Infiniband adapters afterwards to use for the storage traffic.

I'm asking because Infiniband is not ethernet compatible, it is just an IP over Infiniband L3 device with its own 20-bytes hardware address and so can not be L2-bridged by linux-bridge or OVS. Though it works perfectly with usual routing or GRE tunnels in OpenStack.

Regards,
Dmitry

Thanks Andrey Korolyov, that solved my question.

Andrey, one more question please:

you suggest the EoIPoIB solution with the link to a driver implementation at LWN. Did you just googled for it or tried this driver yourself?

I was already thinking of using it but unfortunately have no glue what is the source of the driver and where it can be downloaded for compilation :( There are some patches mentioned at the link but with no source.

Any ideas?..

Yes, I tried it, but amount of work is relatively high - you`ll need to patch kernel with those patches(can be found http://thread.gmane.org/gmane.linux.network/238895 or at patchwork ). EoIB is a faster and more straightforward way to achieve the same functionality but its functionality depends on your cards version(though they are not mlx, they have same capabilities as a regular connect-x mellanox card).

"EoIB is a faster and more straightforward way to achieve the same functionality ... though they are not mlx, they have same capabilities as a regular connect-x mellanox card" - would you be more specific please?..

I use an older model of mellanox connectX which has no hardware support for ethernet switching, so there are no official EoIB drivers from Mellanox for my adapters.

Thank you!
Dmitry

Ehm, I thought that your sentence 'not ConnectX-3' cards related to the card vendor, so if yours are just ConnectX(not even -2, which can work with EoIB just fine) you may try to use patchset from above and consider address reassignment for plain IPoIB as a fallback.

OK, thank you very much for your explanatory answer!

I was just surprised no one here asked the same yet but now things getting more clear...

I'm having the same 'bridging' problem with CEPH / CloudStack combo as it has very limited storage capabilities where I can't use RBD as a secondary storage backend. So I wanted to shift to OpenStack where RBD can be used for both Glance and Cinder without this annoying bridging requirement. It was just a bit unclear for me how does the Fuel OpenStack distro manage storage traffic.

I would prefer to avoid this EoIPoIB solution as it supposed to have huge overhead in resources and latency and I would like to stay with plain IPoIB model.

Am I right that everything I need is to leave this OVS br-storage as is, assign another IP range to IB network and patch the ceph.conf and hosts files to this new address space on all nodes? As far as I can understand no other configuration required?..

Thank you!
Dmitry

Yes, but I suggest you to stay on same addresses. Address change may require ceph.conf change and of course not easy task for a MON quorum in case if 'public' Ceph network with 'storage' one.

Good point, thank you!

If I stay with the same IP addresses, to avoid the IP conflict with preconfigured OVS br-storage interface, is it enough to change the /etc/network/interfaces.d/ifcfg-br-storage IP to a different one? Or this will require more complex reconfiguration of OVS?

Sorry for bothering, I just never used OVS before and don't feel comfortable with its configuration yet.

Dmitry

Until all physical nodes will be able to reach this subnet directly there will be absolutely no problem, so you may just move addresses in the config from br-storage to ibX. Bridge may remain, but feel free to destroy it at any time after address migration(and delete an appropriate network config).

THANK YOU VERY MUCH!

You saved my day!

Everything works as expected, though I also had to change the public_network parameter in ceph.conf to the storage (infiniband) subnet.

Without this change all the traffic between compute <-> CEPH was coming through the mgmt ethernet network and only CEPH internal traffic was running through infiniband:

http://ceph.com/docs/master/rados/configuration/network-config-ref/

Hello, I'm a little stuck trying to get this working. I have loaded the modules on the kernel but fuel still dosent see the infiniband cards. These are a couple of things I tried.

1. I modified the initramfs.img and loaded the modules for inifinband. Fuel still does not see it
2. I downloaded the GIT source and looked for where the modules are loaded and did not find it.

Any help is appreciated.

Thanks,
Roberto