HP 544FLR-QSFP cards Infiniband/Ethernet card
wailer
75 Posts
September 13, 2019, 12:49 pmQuote from wailer on September 13, 2019, 12:49 pmHey guys,
We are planning a new ALL-FLASH cluster, anyone has experience using this type of card?. We'll be using a Nexus Switch with QSPF+, will the installer recognize these cards , or we'll have to make some "hacks" make them work?
Thanks!
Hey guys,
We are planning a new ALL-FLASH cluster, anyone has experience using this type of card?. We'll be using a Nexus Switch with QSPF+, will the installer recognize these cards , or we'll have to make some "hacks" make them work?
Thanks!
Shiori
86 Posts
September 18, 2019, 4:42 amQuote from Shiori on September 18, 2019, 4:42 amWe use these cards on out bl465cg8 blades. Dont have g9 blades as we use amd processors for our virtualization system.
First, infiniband? or being used as 40GBE? These cards must be in mezz slot1 and map port 1 to back slot 5 (5&6 for wide switch blades) and port 2 to back slot 7 (7&8 if a wide switch blade). You may have to run a few command line options to place the cards in the correct mode for your purpose. By default they are infiniband cards but will switch to ethernet automatically (most of the time) on boot up once connected to an ethernet switch.
If Infiniband, you must have a platinum enclosure and use the fdr infiniband blade switch (there are two, one managed and one not. Both work just the same and we opted to use the non managed). Make sure you run a subnet manager on a node that is not going to be down at all. It is possible to configure the subnet manager as an active failover. I highly suggest the use of dedicated storage only nodes and dedicated monitor/iscsi nodes.
If using as 40GBe cards, treat like a flex-LOM card. Just make sure you get the correct switch/pass-through blades. The port mapping stays the same as if in infiniband mode.
Either way you should lookup my tutorial on how to get the stack drivers installed in petasan. Not difficult, just different.
Not testest on 2.3.1 yet. That is next months work.
We use these cards on out bl465cg8 blades. Dont have g9 blades as we use amd processors for our virtualization system.
First, infiniband? or being used as 40GBE? These cards must be in mezz slot1 and map port 1 to back slot 5 (5&6 for wide switch blades) and port 2 to back slot 7 (7&8 if a wide switch blade). You may have to run a few command line options to place the cards in the correct mode for your purpose. By default they are infiniband cards but will switch to ethernet automatically (most of the time) on boot up once connected to an ethernet switch.
If Infiniband, you must have a platinum enclosure and use the fdr infiniband blade switch (there are two, one managed and one not. Both work just the same and we opted to use the non managed). Make sure you run a subnet manager on a node that is not going to be down at all. It is possible to configure the subnet manager as an active failover. I highly suggest the use of dedicated storage only nodes and dedicated monitor/iscsi nodes.
If using as 40GBe cards, treat like a flex-LOM card. Just make sure you get the correct switch/pass-through blades. The port mapping stays the same as if in infiniband mode.
Either way you should lookup my tutorial on how to get the stack drivers installed in petasan. Not difficult, just different.
Not testest on 2.3.1 yet. That is next months work.
wailer
75 Posts
September 18, 2019, 1:18 pmQuote from wailer on September 18, 2019, 1:18 pmHi Shiori,
Thanks for your reply , our plan is to use them as Ethernet cards, they will be connected to a Cisco Nexus port, they are really cheap, compared to other vendors nics...
So, if I get it correctly, it doesn't really matter if you use them as Ethernet or Infiniband, you still have to install drivers first, right?
I saw your tutorial same days ago, that's good to know.
Thanks!
Hi Shiori,
Thanks for your reply , our plan is to use them as Ethernet cards, they will be connected to a Cisco Nexus port, they are really cheap, compared to other vendors nics...
So, if I get it correctly, it doesn't really matter if you use them as Ethernet or Infiniband, you still have to install drivers first, right?
I saw your tutorial same days ago, that's good to know.
Thanks!
Shiori
86 Posts
November 3, 2019, 9:16 pmQuote from Shiori on November 3, 2019, 9:16 pmSorry for the long delay,
No, drivers are already in PetaSan if you plan to use as Ethernet. Follow my post, but make the changes of which modules you setup, eg instead of mlx4_ib, use mlx4_en (depending on the exact 544flx card you may have to use mlx5_en).
Do not setup any of the IB verbs or RDMA modules nor the ib_ipoib module. These are not needed.
Also, you have to make sure your set the mode of the card to Ethernet mode and not VPI/Infiniband. For this you do need the tools which do not compile on PetaSan. I have a debian HDD with the tools compiled on it, I boot with to set the cards first, then deploy them. since we are using them as Infiniband, I mostly just update the firmware to the same as the rest of our cards.
Sorry for the long delay,
No, drivers are already in PetaSan if you plan to use as Ethernet. Follow my post, but make the changes of which modules you setup, eg instead of mlx4_ib, use mlx4_en (depending on the exact 544flx card you may have to use mlx5_en).
Do not setup any of the IB verbs or RDMA modules nor the ib_ipoib module. These are not needed.
Also, you have to make sure your set the mode of the card to Ethernet mode and not VPI/Infiniband. For this you do need the tools which do not compile on PetaSan. I have a debian HDD with the tools compiled on it, I boot with to set the cards first, then deploy them. since we are using them as Infiniband, I mostly just update the firmware to the same as the rest of our cards.
HP 544FLR-QSFP cards Infiniband/Ethernet card
wailer
75 Posts
Quote from wailer on September 13, 2019, 12:49 pmHey guys,
We are planning a new ALL-FLASH cluster, anyone has experience using this type of card?. We'll be using a Nexus Switch with QSPF+, will the installer recognize these cards , or we'll have to make some "hacks" make them work?
Thanks!
Hey guys,
We are planning a new ALL-FLASH cluster, anyone has experience using this type of card?. We'll be using a Nexus Switch with QSPF+, will the installer recognize these cards , or we'll have to make some "hacks" make them work?
Thanks!
Shiori
86 Posts
Quote from Shiori on September 18, 2019, 4:42 amWe use these cards on out bl465cg8 blades. Dont have g9 blades as we use amd processors for our virtualization system.
First, infiniband? or being used as 40GBE? These cards must be in mezz slot1 and map port 1 to back slot 5 (5&6 for wide switch blades) and port 2 to back slot 7 (7&8 if a wide switch blade). You may have to run a few command line options to place the cards in the correct mode for your purpose. By default they are infiniband cards but will switch to ethernet automatically (most of the time) on boot up once connected to an ethernet switch.
If Infiniband, you must have a platinum enclosure and use the fdr infiniband blade switch (there are two, one managed and one not. Both work just the same and we opted to use the non managed). Make sure you run a subnet manager on a node that is not going to be down at all. It is possible to configure the subnet manager as an active failover. I highly suggest the use of dedicated storage only nodes and dedicated monitor/iscsi nodes.
If using as 40GBe cards, treat like a flex-LOM card. Just make sure you get the correct switch/pass-through blades. The port mapping stays the same as if in infiniband mode.
Either way you should lookup my tutorial on how to get the stack drivers installed in petasan. Not difficult, just different.
Not testest on 2.3.1 yet. That is next months work.
We use these cards on out bl465cg8 blades. Dont have g9 blades as we use amd processors for our virtualization system.
First, infiniband? or being used as 40GBE? These cards must be in mezz slot1 and map port 1 to back slot 5 (5&6 for wide switch blades) and port 2 to back slot 7 (7&8 if a wide switch blade). You may have to run a few command line options to place the cards in the correct mode for your purpose. By default they are infiniband cards but will switch to ethernet automatically (most of the time) on boot up once connected to an ethernet switch.
If Infiniband, you must have a platinum enclosure and use the fdr infiniband blade switch (there are two, one managed and one not. Both work just the same and we opted to use the non managed). Make sure you run a subnet manager on a node that is not going to be down at all. It is possible to configure the subnet manager as an active failover. I highly suggest the use of dedicated storage only nodes and dedicated monitor/iscsi nodes.
If using as 40GBe cards, treat like a flex-LOM card. Just make sure you get the correct switch/pass-through blades. The port mapping stays the same as if in infiniband mode.
Either way you should lookup my tutorial on how to get the stack drivers installed in petasan. Not difficult, just different.
Not testest on 2.3.1 yet. That is next months work.
wailer
75 Posts
Quote from wailer on September 18, 2019, 1:18 pmHi Shiori,
Thanks for your reply , our plan is to use them as Ethernet cards, they will be connected to a Cisco Nexus port, they are really cheap, compared to other vendors nics...
So, if I get it correctly, it doesn't really matter if you use them as Ethernet or Infiniband, you still have to install drivers first, right?
I saw your tutorial same days ago, that's good to know.
Thanks!
Hi Shiori,
Thanks for your reply , our plan is to use them as Ethernet cards, they will be connected to a Cisco Nexus port, they are really cheap, compared to other vendors nics...
So, if I get it correctly, it doesn't really matter if you use them as Ethernet or Infiniband, you still have to install drivers first, right?
I saw your tutorial same days ago, that's good to know.
Thanks!
Shiori
86 Posts
Quote from Shiori on November 3, 2019, 9:16 pmSorry for the long delay,
No, drivers are already in PetaSan if you plan to use as Ethernet. Follow my post, but make the changes of which modules you setup, eg instead of mlx4_ib, use mlx4_en (depending on the exact 544flx card you may have to use mlx5_en).
Do not setup any of the IB verbs or RDMA modules nor the ib_ipoib module. These are not needed.
Also, you have to make sure your set the mode of the card to Ethernet mode and not VPI/Infiniband. For this you do need the tools which do not compile on PetaSan. I have a debian HDD with the tools compiled on it, I boot with to set the cards first, then deploy them. since we are using them as Infiniband, I mostly just update the firmware to the same as the rest of our cards.
Sorry for the long delay,
No, drivers are already in PetaSan if you plan to use as Ethernet. Follow my post, but make the changes of which modules you setup, eg instead of mlx4_ib, use mlx4_en (depending on the exact 544flx card you may have to use mlx5_en).
Do not setup any of the IB verbs or RDMA modules nor the ib_ipoib module. These are not needed.
Also, you have to make sure your set the mode of the card to Ethernet mode and not VPI/Infiniband. For this you do need the tools which do not compile on PetaSan. I have a debian HDD with the tools compiled on it, I boot with to set the cards first, then deploy them. since we are using them as Infiniband, I mostly just update the firmware to the same as the rest of our cards.