Petasan with Infiniband switch
hoanglongvina
12 Posts
September 7, 2017, 3:43 amQuote from hoanglongvina on September 7, 2017, 3:43 amHi ADMIN,
I setup system petasan with switch infiniband but I can't connect petasan system with switch infiniband.
I install driver follow link: https://community.mellanox.com/docs/DOC-2377 , but step install packet mft-4.1.0-34.tgz is error :
root@node01:~/mft-4.5.0-31-x86_64-rpm# ./install.sh
-E- There are missing packages that are required for installation of MFT.
-I- You can install missing packages using: apt-get install linux-headers linux-headers-generic
please help me.
Thank Everybody.
Hi ADMIN,
I setup system petasan with switch infiniband but I can't connect petasan system with switch infiniband.
I install driver follow link: https://community.mellanox.com/docs/DOC-2377 , but step install packet mft-4.1.0-34.tgz is error :
root@node01:~/mft-4.5.0-31-x86_64-rpm# ./install.sh
-E- There are missing packages that are required for installation of MFT.
-I- You can install missing packages using: apt-get install linux-headers linux-headers-generic
please help me.
Thank Everybody.
Last edited on September 7, 2017, 8:52 am by hoanglongvina · #1
admin
2,930 Posts
September 7, 2017, 12:11 pmQuote from admin on September 7, 2017, 12:11 pmHi,
PetaSAN does not support infiniband.
but generally if you need the PetaSAN kernel headers to compile extra kernel modules:
https://drive.google.com/file/d/0B7VNYCjYBY2yUDhKLV9WOWU4a2M/view?usp=sharing
sorry cannot help more with infiniband.
Hi,
PetaSAN does not support infiniband.
but generally if you need the PetaSAN kernel headers to compile extra kernel modules:
https://drive.google.com/file/d/0B7VNYCjYBY2yUDhKLV9WOWU4a2M/view?usp=sharing
sorry cannot help more with infiniband.
Last edited on September 7, 2017, 12:20 pm by admin · #2
hoanglongvina
12 Posts
September 8, 2017, 3:57 amQuote from hoanglongvina on September 8, 2017, 3:57 amHi ADMIN,
Can You share with I document create LIO and network vituarl ip on petasan because I using switch Infiniband maybe I setup ceph core. please help me.
Thank ADMIN
Hi ADMIN,
Can You share with I document create LIO and network vituarl ip on petasan because I using switch Infiniband maybe I setup ceph core. please help me.
Thank ADMIN
hoanglongvina
12 Posts
September 12, 2017, 2:41 amQuote from hoanglongvina on September 12, 2017, 2:41 amHi ADMIN,
I'm change using switch quanta, it can connect with petasan. 🙂
Thank ADMIN.
Hi ADMIN,
I'm change using switch quanta, it can connect with petasan. 🙂
Thank ADMIN.
vitalinya
1 Post
December 15, 2018, 5:23 pmQuote from vitalinya on December 15, 2018, 5:23 pmHi admin,
Please can you also sharing with linux-headers-4.12.14-08-petasan.
Thanks.
Hi admin,
Please can you also sharing with linux-headers-4.12.14-08-petasan.
Thanks.
admin
2,930 Posts
December 15, 2018, 8:16 pmQuote from admin on December 15, 2018, 8:16 pmhttps://drive.google.com/drive/folders/1yKGmb47TpVQkkqAEV6Mn0euYhQNDqHik?usp=sharing
https://drive.google.com/drive/folders/1yKGmb47TpVQkkqAEV6Mn0euYhQNDqHik?usp=sharing
yangsm
31 Posts
December 30, 2018, 5:55 amQuote from yangsm on December 30, 2018, 5:55 amDoes petasan 2.2 support Infiniband now?
Does petasan 2.2 support Infiniband now?
admin
2,930 Posts
December 30, 2018, 1:56 pmQuote from admin on December 30, 2018, 1:56 pmNo, we know some users did install with infiniband, but we do not support it currently.
No, we know some users did install with infiniband, but we do not support it currently.
Shiori
86 Posts
February 12, 2019, 3:52 pmQuote from Shiori on February 12, 2019, 3:52 pmSearch the forum for infiniband and you will see my post on how to make infiniband work on PetaSAN. This process still works in PetaSAN 2.2
This must be done on each node and prior to joining the cluster. Also Upgrades wipe out the drivers and must be reinstalled during first boot before the node rejoins the cluster.
I found it easier to create a new cluster with the minimal amount of required storage then move the data over to the new cluster. Then destroy the old cluster and add the nodes to the new cluster. Not the best way but it is the safest.
The problem with infiniband is that the card you choose to use can have very different drivers, there is no set standard driver language for these cards like there is for ethernet cards. Eg the mellanox cx1 cards do not work with the 4.0 kernels due to the drivers but the cx2 cards do and the cx3 cards drivers do not work with 3.x kernels and older at all. The qlogic cards are another beast of driver/kernel compatibility issues. It is impossible for the dev team to be able to support this mess. But with more users of infiniband and the odd donation of equipment to you friendly dev teams, this may change. MS has already included many infiniband drivers into windows 2012 and newer, so it is just time to get linuxes up to that level.
Search the forum for infiniband and you will see my post on how to make infiniband work on PetaSAN. This process still works in PetaSAN 2.2
This must be done on each node and prior to joining the cluster. Also Upgrades wipe out the drivers and must be reinstalled during first boot before the node rejoins the cluster.
I found it easier to create a new cluster with the minimal amount of required storage then move the data over to the new cluster. Then destroy the old cluster and add the nodes to the new cluster. Not the best way but it is the safest.
The problem with infiniband is that the card you choose to use can have very different drivers, there is no set standard driver language for these cards like there is for ethernet cards. Eg the mellanox cx1 cards do not work with the 4.0 kernels due to the drivers but the cx2 cards do and the cx3 cards drivers do not work with 3.x kernels and older at all. The qlogic cards are another beast of driver/kernel compatibility issues. It is impossible for the dev team to be able to support this mess. But with more users of infiniband and the odd donation of equipment to you friendly dev teams, this may change. MS has already included many infiniband drivers into windows 2012 and newer, so it is just time to get linuxes up to that level.
Petasan with Infiniband switch
hoanglongvina
12 Posts
Quote from hoanglongvina on September 7, 2017, 3:43 amHi ADMIN,
I setup system petasan with switch infiniband but I can't connect petasan system with switch infiniband.
I install driver follow link: https://community.mellanox.com/docs/DOC-2377 , but step install packet mft-4.1.0-34.tgz is error :
root@node01:~/mft-4.5.0-31-x86_64-rpm# ./install.sh
-E- There are missing packages that are required for installation of MFT.
-I- You can install missing packages using: apt-get install linux-headers linux-headers-genericplease help me.
Thank Everybody.
Hi ADMIN,
I setup system petasan with switch infiniband but I can't connect petasan system with switch infiniband.
I install driver follow link: https://community.mellanox.com/docs/DOC-2377 , but step install packet mft-4.1.0-34.tgz is error :
root@node01:~/mft-4.5.0-31-x86_64-rpm# ./install.sh
-E- There are missing packages that are required for installation of MFT.
-I- You can install missing packages using: apt-get install linux-headers linux-headers-generic
please help me.
Thank Everybody.
admin
2,930 Posts
Quote from admin on September 7, 2017, 12:11 pmHi,
PetaSAN does not support infiniband.
but generally if you need the PetaSAN kernel headers to compile extra kernel modules:
https://drive.google.com/file/d/0B7VNYCjYBY2yUDhKLV9WOWU4a2M/view?usp=sharing
sorry cannot help more with infiniband.
Hi,
PetaSAN does not support infiniband.
but generally if you need the PetaSAN kernel headers to compile extra kernel modules:
https://drive.google.com/file/d/0B7VNYCjYBY2yUDhKLV9WOWU4a2M/view?usp=sharing
sorry cannot help more with infiniband.
hoanglongvina
12 Posts
Quote from hoanglongvina on September 8, 2017, 3:57 amHi ADMIN,
Can You share with I document create LIO and network vituarl ip on petasan because I using switch Infiniband maybe I setup ceph core. please help me.
Thank ADMIN
Hi ADMIN,
Can You share with I document create LIO and network vituarl ip on petasan because I using switch Infiniband maybe I setup ceph core. please help me.
Thank ADMIN
hoanglongvina
12 Posts
Quote from hoanglongvina on September 12, 2017, 2:41 amHi ADMIN,
I'm change using switch quanta, it can connect with petasan. 🙂
Thank ADMIN.
Hi ADMIN,
I'm change using switch quanta, it can connect with petasan. 🙂
Thank ADMIN.
vitalinya
1 Post
Quote from vitalinya on December 15, 2018, 5:23 pmHi admin,
Please can you also sharing with linux-headers-4.12.14-08-petasan.
Thanks.
Hi admin,
Please can you also sharing with linux-headers-4.12.14-08-petasan.
Thanks.
admin
2,930 Posts
Quote from admin on December 15, 2018, 8:16 pmhttps://drive.google.com/drive/folders/1yKGmb47TpVQkkqAEV6Mn0euYhQNDqHik?usp=sharing
https://drive.google.com/drive/folders/1yKGmb47TpVQkkqAEV6Mn0euYhQNDqHik?usp=sharing
yangsm
31 Posts
Quote from yangsm on December 30, 2018, 5:55 amDoes petasan 2.2 support Infiniband now?
Does petasan 2.2 support Infiniband now?
admin
2,930 Posts
Quote from admin on December 30, 2018, 1:56 pmNo, we know some users did install with infiniband, but we do not support it currently.
No, we know some users did install with infiniband, but we do not support it currently.
Shiori
86 Posts
Quote from Shiori on February 12, 2019, 3:52 pmSearch the forum for infiniband and you will see my post on how to make infiniband work on PetaSAN. This process still works in PetaSAN 2.2
This must be done on each node and prior to joining the cluster. Also Upgrades wipe out the drivers and must be reinstalled during first boot before the node rejoins the cluster.
I found it easier to create a new cluster with the minimal amount of required storage then move the data over to the new cluster. Then destroy the old cluster and add the nodes to the new cluster. Not the best way but it is the safest.
The problem with infiniband is that the card you choose to use can have very different drivers, there is no set standard driver language for these cards like there is for ethernet cards. Eg the mellanox cx1 cards do not work with the 4.0 kernels due to the drivers but the cx2 cards do and the cx3 cards drivers do not work with 3.x kernels and older at all. The qlogic cards are another beast of driver/kernel compatibility issues. It is impossible for the dev team to be able to support this mess. But with more users of infiniband and the odd donation of equipment to you friendly dev teams, this may change. MS has already included many infiniband drivers into windows 2012 and newer, so it is just time to get linuxes up to that level.
Search the forum for infiniband and you will see my post on how to make infiniband work on PetaSAN. This process still works in PetaSAN 2.2
This must be done on each node and prior to joining the cluster. Also Upgrades wipe out the drivers and must be reinstalled during first boot before the node rejoins the cluster.
I found it easier to create a new cluster with the minimal amount of required storage then move the data over to the new cluster. Then destroy the old cluster and add the nodes to the new cluster. Not the best way but it is the safest.
The problem with infiniband is that the card you choose to use can have very different drivers, there is no set standard driver language for these cards like there is for ethernet cards. Eg the mellanox cx1 cards do not work with the 4.0 kernels due to the drivers but the cx2 cards do and the cx3 cards drivers do not work with 3.x kernels and older at all. The qlogic cards are another beast of driver/kernel compatibility issues. It is impossible for the dev team to be able to support this mess. But with more users of infiniband and the odd donation of equipment to you friendly dev teams, this may change. MS has already included many infiniband drivers into windows 2012 and newer, so it is just time to get linuxes up to that level.