use the 40GB mellanox switch at backend
tuocmd
54 Posts
May 28, 2018, 5:54 pmQuote from tuocmd on May 28, 2018, 5:54 pmDoes petasan support the use of a 40GB switch?
Does petasan support the use of a 40GB switch?
admin
2,921 Posts
May 29, 2018, 11:25 amQuote from admin on May 29, 2018, 11:25 amWe support 40G ethernet. We do not support infiniband since we do not test it, some users got working by some manual effort.
We support 40G ethernet. We do not support infiniband since we do not test it, some users got working by some manual effort.
Shiori
86 Posts
June 6, 2019, 2:22 amQuote from Shiori on June 6, 2019, 2:22 amthe short answer is yes, but there are caveats.
the longer answer is, the Mellanox switch in question must support the protocols being used. In the case of supported configurations: Ethernet.
If you are lucky enough to have a switch that understands Ethernet then you would not need to do anything but plug it in. If your switch only understands Infiniband then you will have some challenges to use that switch. Though the switches are fundamentally the same, Infiniband uses a much longer MAC address and a different frame format but can carry Ethernet in an encapsulated form(IPoverIB). Check with Mellanox's support pages for additional firmware to change to Ethernet if available.
My simple solution was just to use Infiniband on my nodes for iSCSI and backend networks. I have a fairly clear how-to in these forums on how to do this if you are interested.
the short answer is yes, but there are caveats.
the longer answer is, the Mellanox switch in question must support the protocols being used. In the case of supported configurations: Ethernet.
If you are lucky enough to have a switch that understands Ethernet then you would not need to do anything but plug it in. If your switch only understands Infiniband then you will have some challenges to use that switch. Though the switches are fundamentally the same, Infiniband uses a much longer MAC address and a different frame format but can carry Ethernet in an encapsulated form(IPoverIB). Check with Mellanox's support pages for additional firmware to change to Ethernet if available.
My simple solution was just to use Infiniband on my nodes for iSCSI and backend networks. I have a fairly clear how-to in these forums on how to do this if you are interested.
Kurti2k
16 Posts
June 6, 2019, 12:30 pmQuote from Kurti2k on June 6, 2019, 12:30 pm
Hi you must change Port Type from your MLX Card
InfiniBand (1)
Ethernet (2)
AutoSense (3)
default set is 3
set port type to 2 (edit /dev/mtXXXX_pciconfX)
mst status
mlxconfig -d /dev/mt4121_pciconf0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2
reboot
Best Regards
Hi you must change Port Type from your MLX Card
InfiniBand (1)
Ethernet (2)
AutoSense (3)
default set is 3
set port type to 2 (edit /dev/mtXXXX_pciconfX)
mst status
mlxconfig -d /dev/mt4121_pciconf0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2
reboot
Best Regards
Last edited on June 6, 2019, 1:37 pm by Kurti2k · #4
Shiori
86 Posts
June 13, 2019, 9:09 pmQuote from Shiori on June 13, 2019, 9:09 pmKurti2k,
That is the procedure for MLX cards, the question was to the use of a switch without any word on which network card was to be used. It is true however that most of the Mellanox Infiniband cards can be configured to connect to an Ethernet based switch or direct connected Ethernet card, the most notable exceptions are the IB series cards that do not support Ethernet at the hardware. Any ConnectX card supports both Ethernet and Infiniband.
Any network card with QSFP+ connections should work with an Infiniband switch, but if the switch does not recognize the much shorter MAC addresses used by Ethernet then it will not work. I had this issue with a Voltare switch that did not support Ethernet.
It also must be noted that Ethernet switches with QSFP+ connectors do not support Infiniband connections, again it has to do with the MAC address length.
Kurti2k,
That is the procedure for MLX cards, the question was to the use of a switch without any word on which network card was to be used. It is true however that most of the Mellanox Infiniband cards can be configured to connect to an Ethernet based switch or direct connected Ethernet card, the most notable exceptions are the IB series cards that do not support Ethernet at the hardware. Any ConnectX card supports both Ethernet and Infiniband.
Any network card with QSFP+ connections should work with an Infiniband switch, but if the switch does not recognize the much shorter MAC addresses used by Ethernet then it will not work. I had this issue with a Voltare switch that did not support Ethernet.
It also must be noted that Ethernet switches with QSFP+ connectors do not support Infiniband connections, again it has to do with the MAC address length.
Kurti2k
16 Posts
June 19, 2019, 8:09 amQuote from Kurti2k on June 19, 2019, 8:09 amok sry missunderstood
you mean IPoverIB ? you need a driver forit to see the device in interface for setting up the cluster
ok sry missunderstood
you mean IPoverIB ? you need a driver forit to see the device in interface for setting up the cluster
Shiori
86 Posts
January 28, 2020, 11:42 amQuote from Shiori on January 28, 2020, 11:42 amsorry, I didn't get notified you asked this question, so for clarity and to provide a fuller understanding:
No, IPoIB is a IB function to use a IB network as a IP network (we do this on our clusters). the Ethernet function is simply a pure Ethernet setup using a compatible IB card in Ethernet mode. The drivers will differ for the extended functions but are not directly needed as the standard Ethernet drivers found in most systems are workable.
If you are using IPoIB then you must load the IB driver set and the IPoIB module. In Linux just "modprobe mlx4" (change number for differing cards, 4 is Connectx2, 5 is connectx3) then "modprobe ipoib"
then setup your ip addresses as normal but instead of eth? it will be ib0 and ib1 (ports are numbered individually as if they are separate cards)
sorry, I didn't get notified you asked this question, so for clarity and to provide a fuller understanding:
No, IPoIB is a IB function to use a IB network as a IP network (we do this on our clusters). the Ethernet function is simply a pure Ethernet setup using a compatible IB card in Ethernet mode. The drivers will differ for the extended functions but are not directly needed as the standard Ethernet drivers found in most systems are workable.
If you are using IPoIB then you must load the IB driver set and the IPoIB module. In Linux just "modprobe mlx4" (change number for differing cards, 4 is Connectx2, 5 is connectx3) then "modprobe ipoib"
then setup your ip addresses as normal but instead of eth? it will be ib0 and ib1 (ports are numbered individually as if they are separate cards)
dh0253
15 Posts
November 14, 2022, 4:08 amQuote from dh0253 on November 14, 2022, 4:08 amHi Shiori,
We're interested in setting up PetaSAN with Mellanox InfiniBand switches SX6036 and HPE 544+QSFP (MT27520) adapters. Could you share your instructions on how to setup PetaSAN for IB only.
Thank you
Hi Shiori,
We're interested in setting up PetaSAN with Mellanox InfiniBand switches SX6036 and HPE 544+QSFP (MT27520) adapters. Could you share your instructions on how to setup PetaSAN for IB only.
Thank you
use the 40GB mellanox switch at backend
tuocmd
54 Posts
Quote from tuocmd on May 28, 2018, 5:54 pmDoes petasan support the use of a 40GB switch?
Does petasan support the use of a 40GB switch?
admin
2,921 Posts
Quote from admin on May 29, 2018, 11:25 amWe support 40G ethernet. We do not support infiniband since we do not test it, some users got working by some manual effort.
We support 40G ethernet. We do not support infiniband since we do not test it, some users got working by some manual effort.
Shiori
86 Posts
Quote from Shiori on June 6, 2019, 2:22 amthe short answer is yes, but there are caveats.
the longer answer is, the Mellanox switch in question must support the protocols being used. In the case of supported configurations: Ethernet.
If you are lucky enough to have a switch that understands Ethernet then you would not need to do anything but plug it in. If your switch only understands Infiniband then you will have some challenges to use that switch. Though the switches are fundamentally the same, Infiniband uses a much longer MAC address and a different frame format but can carry Ethernet in an encapsulated form(IPoverIB). Check with Mellanox's support pages for additional firmware to change to Ethernet if available.
My simple solution was just to use Infiniband on my nodes for iSCSI and backend networks. I have a fairly clear how-to in these forums on how to do this if you are interested.
the short answer is yes, but there are caveats.
the longer answer is, the Mellanox switch in question must support the protocols being used. In the case of supported configurations: Ethernet.
If you are lucky enough to have a switch that understands Ethernet then you would not need to do anything but plug it in. If your switch only understands Infiniband then you will have some challenges to use that switch. Though the switches are fundamentally the same, Infiniband uses a much longer MAC address and a different frame format but can carry Ethernet in an encapsulated form(IPoverIB). Check with Mellanox's support pages for additional firmware to change to Ethernet if available.
My simple solution was just to use Infiniband on my nodes for iSCSI and backend networks. I have a fairly clear how-to in these forums on how to do this if you are interested.
Kurti2k
16 Posts
Quote from Kurti2k on June 6, 2019, 12:30 pm
Hi you must change Port Type from your MLX Card
InfiniBand (1)
Ethernet (2)
AutoSense (3)default set is 3
set port type to 2 (edit /dev/mtXXXX_pciconfX)
mst status
mlxconfig -d /dev/mt4121_pciconf0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2
reboot
Best Regards
Hi you must change Port Type from your MLX Card
InfiniBand (1)
Ethernet (2)
AutoSense (3)
default set is 3
set port type to 2 (edit /dev/mtXXXX_pciconfX)
mst status
mlxconfig -d /dev/mt4121_pciconf0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2
reboot
Best Regards
Shiori
86 Posts
Quote from Shiori on June 13, 2019, 9:09 pmKurti2k,
That is the procedure for MLX cards, the question was to the use of a switch without any word on which network card was to be used. It is true however that most of the Mellanox Infiniband cards can be configured to connect to an Ethernet based switch or direct connected Ethernet card, the most notable exceptions are the IB series cards that do not support Ethernet at the hardware. Any ConnectX card supports both Ethernet and Infiniband.
Any network card with QSFP+ connections should work with an Infiniband switch, but if the switch does not recognize the much shorter MAC addresses used by Ethernet then it will not work. I had this issue with a Voltare switch that did not support Ethernet.
It also must be noted that Ethernet switches with QSFP+ connectors do not support Infiniband connections, again it has to do with the MAC address length.
Kurti2k,
That is the procedure for MLX cards, the question was to the use of a switch without any word on which network card was to be used. It is true however that most of the Mellanox Infiniband cards can be configured to connect to an Ethernet based switch or direct connected Ethernet card, the most notable exceptions are the IB series cards that do not support Ethernet at the hardware. Any ConnectX card supports both Ethernet and Infiniband.
Any network card with QSFP+ connections should work with an Infiniband switch, but if the switch does not recognize the much shorter MAC addresses used by Ethernet then it will not work. I had this issue with a Voltare switch that did not support Ethernet.
It also must be noted that Ethernet switches with QSFP+ connectors do not support Infiniband connections, again it has to do with the MAC address length.
Kurti2k
16 Posts
Quote from Kurti2k on June 19, 2019, 8:09 amok sry missunderstood
you mean IPoverIB ? you need a driver forit to see the device in interface for setting up the cluster
ok sry missunderstood
you mean IPoverIB ? you need a driver forit to see the device in interface for setting up the cluster
Shiori
86 Posts
Quote from Shiori on January 28, 2020, 11:42 amsorry, I didn't get notified you asked this question, so for clarity and to provide a fuller understanding:
No, IPoIB is a IB function to use a IB network as a IP network (we do this on our clusters). the Ethernet function is simply a pure Ethernet setup using a compatible IB card in Ethernet mode. The drivers will differ for the extended functions but are not directly needed as the standard Ethernet drivers found in most systems are workable.
If you are using IPoIB then you must load the IB driver set and the IPoIB module. In Linux just "modprobe mlx4" (change number for differing cards, 4 is Connectx2, 5 is connectx3) then "modprobe ipoib"
then setup your ip addresses as normal but instead of eth? it will be ib0 and ib1 (ports are numbered individually as if they are separate cards)
sorry, I didn't get notified you asked this question, so for clarity and to provide a fuller understanding:
No, IPoIB is a IB function to use a IB network as a IP network (we do this on our clusters). the Ethernet function is simply a pure Ethernet setup using a compatible IB card in Ethernet mode. The drivers will differ for the extended functions but are not directly needed as the standard Ethernet drivers found in most systems are workable.
If you are using IPoIB then you must load the IB driver set and the IPoIB module. In Linux just "modprobe mlx4" (change number for differing cards, 4 is Connectx2, 5 is connectx3) then "modprobe ipoib"
then setup your ip addresses as normal but instead of eth? it will be ib0 and ib1 (ports are numbered individually as if they are separate cards)
dh0253
15 Posts
Quote from dh0253 on November 14, 2022, 4:08 amHi Shiori,
We're interested in setting up PetaSAN with Mellanox InfiniBand switches SX6036 and HPE 544+QSFP (MT27520) adapters. Could you share your instructions on how to setup PetaSAN for IB only.
Thank you
Hi Shiori,
We're interested in setting up PetaSAN with Mellanox InfiniBand switches SX6036 and HPE 544+QSFP (MT27520) adapters. Could you share your instructions on how to setup PetaSAN for IB only.
Thank you