Physical node recovery
spiralmatrix
8 Posts
February 17, 2020, 2:34 pmQuote from spiralmatrix on February 17, 2020, 2:34 pmHi, I would like to better understand the position on recovery of a failed physical server acting as a storage node.
Scenario is as follows:
3 Physical servers, 2 HBA adapters, 24SSD per server, the usual multiple 10Gbps / 25Gbps Nic adapters for Management, frontend. backend etc.., 1 SSD with PetaSAN installed (no raid as only dual HBA adapters in the server). PetaSAN built with 3 replica's operational minimum 2.
each node is monitor, iSCSI, and storage.
SSD with PetaSAN OS fails on one node.
What is best / recommended approach to recovery?
As OSD's are not damaged but the node is down, is the expected approach, new SSD, install PetaSAN after removing the failed node in the management interface using one of the other 2 nodes, then re add all OSD's, PetaSAN resyncs and balances.
Or is there a better / recommended recovery method?
for example:
Add as a new node, then import OSD discs?
Add as a new node, reformat OSD discs from failed node and allow to resync and balance data (I am guessing not the best approach).
Additional question, when a node fails in order to avoid iSCSI path failure does the deployed iSCSI LUN need 3 paths per iSCSI subnet based on 3 physical servers, or does the IP per path add as a VIP per front end subnet?
I'm not 100% on the above as my understanding is the more nodes and discs you add to a cluster the potential speed increases, however not sure if this is front end / back end speed or both?
Sorry if the above questions have been answered somewhere already of if the questions seem odd, I just want to understand in a real world failure best approach to recovering a failed node without making a bad situation worse or ending up losing access across other nodes in the process.
Thanks in advance.
Hi, I would like to better understand the position on recovery of a failed physical server acting as a storage node.
Scenario is as follows:
3 Physical servers, 2 HBA adapters, 24SSD per server, the usual multiple 10Gbps / 25Gbps Nic adapters for Management, frontend. backend etc.., 1 SSD with PetaSAN installed (no raid as only dual HBA adapters in the server). PetaSAN built with 3 replica's operational minimum 2.
each node is monitor, iSCSI, and storage.
SSD with PetaSAN OS fails on one node.
What is best / recommended approach to recovery?
As OSD's are not damaged but the node is down, is the expected approach, new SSD, install PetaSAN after removing the failed node in the management interface using one of the other 2 nodes, then re add all OSD's, PetaSAN resyncs and balances.
Or is there a better / recommended recovery method?
for example:
Add as a new node, then import OSD discs?
Add as a new node, reformat OSD discs from failed node and allow to resync and balance data (I am guessing not the best approach).
Additional question, when a node fails in order to avoid iSCSI path failure does the deployed iSCSI LUN need 3 paths per iSCSI subnet based on 3 physical servers, or does the IP per path add as a VIP per front end subnet?
I'm not 100% on the above as my understanding is the more nodes and discs you add to a cluster the potential speed increases, however not sure if this is front end / back end speed or both?
Sorry if the above questions have been answered somewhere already of if the questions seem odd, I just want to understand in a real world failure best approach to recovering a failed node without making a bad situation worse or ending up losing access across other nodes in the process.
Thanks in advance.
admin
2,930 Posts
February 18, 2020, 5:59 pmQuote from admin on February 18, 2020, 5:59 pmFor nodes beyond 3, you can install PetaSAN on a new installation, plug the old OSDs in and things will work. You can plug the old OSDs after you install, or before, in the later case do not select the OSDs to be added as new OSDs during deployment (there is a message to warn you that new OSDs will be formatted), so do not select the old OSDs. You can also re-distribute the OSDs to any nodes. One thing to keep in mind is the data the cluster assigns to the OSD depends on the OSD id as well as the hostname, so if you place an OSD in a new host, there will be a lot of re balancing and shuffling of data (this is based on how crush distributes pgs)...so it is better to assign the new node to have the same hostname as the original node, although not a must. The IP address could be different if you wish, the OSDs will use the new ips.
For nodes 1-3, it is more strict, to replace a node it must have the same hostname and PetaSAN will give it the same ip addresses (apart from management network which you assign via installer), the ips cannot change as it needs to be fixed by Ceph monitors, Consul server, Gluster servers ...etc.
For nodes beyond 3, you can install PetaSAN on a new installation, plug the old OSDs in and things will work. You can plug the old OSDs after you install, or before, in the later case do not select the OSDs to be added as new OSDs during deployment (there is a message to warn you that new OSDs will be formatted), so do not select the old OSDs. You can also re-distribute the OSDs to any nodes. One thing to keep in mind is the data the cluster assigns to the OSD depends on the OSD id as well as the hostname, so if you place an OSD in a new host, there will be a lot of re balancing and shuffling of data (this is based on how crush distributes pgs)...so it is better to assign the new node to have the same hostname as the original node, although not a must. The IP address could be different if you wish, the OSDs will use the new ips.
For nodes 1-3, it is more strict, to replace a node it must have the same hostname and PetaSAN will give it the same ip addresses (apart from management network which you assign via installer), the ips cannot change as it needs to be fixed by Ceph monitors, Consul server, Gluster servers ...etc.
spiralmatrix
8 Posts
February 20, 2020, 10:52 amQuote from spiralmatrix on February 20, 2020, 10:52 amPerfect, thank you for explaining.
Perfect, thank you for explaining.
Physical node recovery
spiralmatrix
8 Posts
Quote from spiralmatrix on February 17, 2020, 2:34 pmHi, I would like to better understand the position on recovery of a failed physical server acting as a storage node.
Scenario is as follows:
3 Physical servers, 2 HBA adapters, 24SSD per server, the usual multiple 10Gbps / 25Gbps Nic adapters for Management, frontend. backend etc.., 1 SSD with PetaSAN installed (no raid as only dual HBA adapters in the server). PetaSAN built with 3 replica's operational minimum 2.
each node is monitor, iSCSI, and storage.
SSD with PetaSAN OS fails on one node.
What is best / recommended approach to recovery?
As OSD's are not damaged but the node is down, is the expected approach, new SSD, install PetaSAN after removing the failed node in the management interface using one of the other 2 nodes, then re add all OSD's, PetaSAN resyncs and balances.
Or is there a better / recommended recovery method?
for example:
Add as a new node, then import OSD discs?
Add as a new node, reformat OSD discs from failed node and allow to resync and balance data (I am guessing not the best approach).
Additional question, when a node fails in order to avoid iSCSI path failure does the deployed iSCSI LUN need 3 paths per iSCSI subnet based on 3 physical servers, or does the IP per path add as a VIP per front end subnet?
I'm not 100% on the above as my understanding is the more nodes and discs you add to a cluster the potential speed increases, however not sure if this is front end / back end speed or both?
Sorry if the above questions have been answered somewhere already of if the questions seem odd, I just want to understand in a real world failure best approach to recovering a failed node without making a bad situation worse or ending up losing access across other nodes in the process.
Thanks in advance.
Hi, I would like to better understand the position on recovery of a failed physical server acting as a storage node.
Scenario is as follows:
3 Physical servers, 2 HBA adapters, 24SSD per server, the usual multiple 10Gbps / 25Gbps Nic adapters for Management, frontend. backend etc.., 1 SSD with PetaSAN installed (no raid as only dual HBA adapters in the server). PetaSAN built with 3 replica's operational minimum 2.
each node is monitor, iSCSI, and storage.
SSD with PetaSAN OS fails on one node.
What is best / recommended approach to recovery?
As OSD's are not damaged but the node is down, is the expected approach, new SSD, install PetaSAN after removing the failed node in the management interface using one of the other 2 nodes, then re add all OSD's, PetaSAN resyncs and balances.
Or is there a better / recommended recovery method?
for example:
Add as a new node, then import OSD discs?
Add as a new node, reformat OSD discs from failed node and allow to resync and balance data (I am guessing not the best approach).
Additional question, when a node fails in order to avoid iSCSI path failure does the deployed iSCSI LUN need 3 paths per iSCSI subnet based on 3 physical servers, or does the IP per path add as a VIP per front end subnet?
I'm not 100% on the above as my understanding is the more nodes and discs you add to a cluster the potential speed increases, however not sure if this is front end / back end speed or both?
Sorry if the above questions have been answered somewhere already of if the questions seem odd, I just want to understand in a real world failure best approach to recovering a failed node without making a bad situation worse or ending up losing access across other nodes in the process.
Thanks in advance.
admin
2,930 Posts
Quote from admin on February 18, 2020, 5:59 pmFor nodes beyond 3, you can install PetaSAN on a new installation, plug the old OSDs in and things will work. You can plug the old OSDs after you install, or before, in the later case do not select the OSDs to be added as new OSDs during deployment (there is a message to warn you that new OSDs will be formatted), so do not select the old OSDs. You can also re-distribute the OSDs to any nodes. One thing to keep in mind is the data the cluster assigns to the OSD depends on the OSD id as well as the hostname, so if you place an OSD in a new host, there will be a lot of re balancing and shuffling of data (this is based on how crush distributes pgs)...so it is better to assign the new node to have the same hostname as the original node, although not a must. The IP address could be different if you wish, the OSDs will use the new ips.
For nodes 1-3, it is more strict, to replace a node it must have the same hostname and PetaSAN will give it the same ip addresses (apart from management network which you assign via installer), the ips cannot change as it needs to be fixed by Ceph monitors, Consul server, Gluster servers ...etc.
For nodes beyond 3, you can install PetaSAN on a new installation, plug the old OSDs in and things will work. You can plug the old OSDs after you install, or before, in the later case do not select the OSDs to be added as new OSDs during deployment (there is a message to warn you that new OSDs will be formatted), so do not select the old OSDs. You can also re-distribute the OSDs to any nodes. One thing to keep in mind is the data the cluster assigns to the OSD depends on the OSD id as well as the hostname, so if you place an OSD in a new host, there will be a lot of re balancing and shuffling of data (this is based on how crush distributes pgs)...so it is better to assign the new node to have the same hostname as the original node, although not a must. The IP address could be different if you wish, the OSDs will use the new ips.
For nodes 1-3, it is more strict, to replace a node it must have the same hostname and PetaSAN will give it the same ip addresses (apart from management network which you assign via installer), the ips cannot change as it needs to be fixed by Ceph monitors, Consul server, Gluster servers ...etc.
spiralmatrix
8 Posts
Quote from spiralmatrix on February 20, 2020, 10:52 amPerfect, thank you for explaining.
Perfect, thank you for explaining.