Path Assignment & Hard Drives
storageman
10 Posts
May 22, 2021, 6:33 pmQuote from storageman on May 22, 2021, 6:33 pmI have a cluster with 4 nodes, 3 nodes have identical hard drives with same combination on SAS & SSD drives. Few days back we added 4th node, I have confusion that is we need to add same hard drive like other 3 nodes or we can add different type/size hdd's. Any complication?
Few more things for clarification:-
1. We are planning to add NVMe drives on all nodes but how to segregate NVMe drives to create separate NVMe pools like we are doing for SAS & SSD by adding rules.
2. Due to some unforeseen circumstances , Our 3rd node goes down for few hours so now all the path assigned to only node 1 & node 2, Node 3 & 4 are empty. How to reassign all path for balance cross all nodes. Best way to resolve without downtime. Any complication or some extra cautious. Please guide.
3. Identical HDDs required across all the cluster?
Thanks & Regards,
I have a cluster with 4 nodes, 3 nodes have identical hard drives with same combination on SAS & SSD drives. Few days back we added 4th node, I have confusion that is we need to add same hard drive like other 3 nodes or we can add different type/size hdd's. Any complication?
Few more things for clarification:-
1. We are planning to add NVMe drives on all nodes but how to segregate NVMe drives to create separate NVMe pools like we are doing for SAS & SSD by adding rules.
2. Due to some unforeseen circumstances , Our 3rd node goes down for few hours so now all the path assigned to only node 1 & node 2, Node 3 & 4 are empty. How to reassign all path for balance cross all nodes. Best way to resolve without downtime. Any complication or some extra cautious. Please guide.
3. Identical HDDs required across all the cluster?
Thanks & Regards,
Abhishek Kumar Singh
18 Posts
May 28, 2021, 7:13 amQuote from Abhishek Kumar Singh on May 28, 2021, 7:13 amAny update on this team? I have concern about identical hard disks across cluster?
Any update on this team? I have concern about identical hard disks across cluster?
Last edited on May 28, 2021, 7:15 am by Abhishek Kumar Singh · #2
admin
2,930 Posts
May 29, 2021, 3:42 pmQuote from admin on May 29, 2021, 3:42 pmThe system will work if you use different disk models and disk sizes.
However the slowest disk will become the bottleneck of the entire pool, even a disk with larger size will get more traffic and hence will also become a bottleneck. so for better performance try to use disks that are as identical as possible in terms of size and performance characteristics, in many cases try to stick with same vendor and same product line.
The system will work if you use different disk models and disk sizes.
However the slowest disk will become the bottleneck of the entire pool, even a disk with larger size will get more traffic and hence will also become a bottleneck. so for better performance try to use disks that are as identical as possible in terms of size and performance characteristics, in many cases try to stick with same vendor and same product line.
storageman
10 Posts
May 30, 2021, 6:33 amQuote from storageman on May 30, 2021, 6:33 amThank you PetaSAN team, you guys are really awesome.
Still our disk path are on only node 1 & node 2, We need to re-balance the cluster with equal amount of path across the cluster without downtime or any complication.
and
How we can segregate NVMe disks like normal SSD or SAS. if it is not possible surly we can save few bucks.
Thank you PetaSAN team, you guys are really awesome.
Still our disk path are on only node 1 & node 2, We need to re-balance the cluster with equal amount of path across the cluster without downtime or any complication.
and
How we can segregate NVMe disks like normal SSD or SAS. if it is not possible surly we can save few bucks.
Last edited on May 30, 2021, 7:08 am by storageman · #4
admin
2,930 Posts
June 2, 2021, 3:02 pmQuote from admin on June 2, 2021, 3:02 pmYou can use the path assignment page to dynamically move paths around, it has an auto feature to balance dynamically
NVME are detected as SSD, you can set a device custom class via
ceph osd crush set-device-class NEWCLASS osd.X osd.Y
then you will need to create a crush rule to place data on that new class. Note currently with Ceph, pure NVMEs will yield same performance as SSDs due to the distributed latency.
You can use the path assignment page to dynamically move paths around, it has an auto feature to balance dynamically
NVME are detected as SSD, you can set a device custom class via
ceph osd crush set-device-class NEWCLASS osd.X osd.Y
then you will need to create a crush rule to place data on that new class. Note currently with Ceph, pure NVMEs will yield same performance as SSDs due to the distributed latency.
Shiori
86 Posts
December 17, 2021, 2:29 pmQuote from Shiori on December 17, 2021, 2:29 pmjust to add some more:
We add OSDs as needed and have found that a drive that is twice as large as the others will see up to four times the load. This is a Ceph thing and can be tuned but then you are basically telling Ceph that your 6tb drive is the same size as your 3tb drives. This is not a bad thing though as it is useful when upgrading a live cluster to larger drives and allowing them to finish replication before rebalancing the cluster.
By keeping to the same size and speed of drives, you make performance more predictable and easier to calculate in scale. if your 2tb 7200rpm drives give you 100iops each and you have 24 drives across four nodes then you should see 2400iops. But now you add 10 more drives but they are 5200rpm and only give 75iops each, they do not add in like you would expect but actually pull the first 24 drives down to a figure thats closer to 90iops each while still only giving 75iops. As you can see it gets complicated very quickly to figure out why your cluster suddenly slower than expected. Again there are ways to mitigate this with Ceph commands but then you loose the auto balancing that makes Ceph so efficient.
Path assignments are linked to but separate of the underlying storage system. Just create your new cluster of NVMe drives and create a few rules to make them do as needed, it really no different from setting up SSDs as a second cluster on the nodes.
just to add some more:
We add OSDs as needed and have found that a drive that is twice as large as the others will see up to four times the load. This is a Ceph thing and can be tuned but then you are basically telling Ceph that your 6tb drive is the same size as your 3tb drives. This is not a bad thing though as it is useful when upgrading a live cluster to larger drives and allowing them to finish replication before rebalancing the cluster.
By keeping to the same size and speed of drives, you make performance more predictable and easier to calculate in scale. if your 2tb 7200rpm drives give you 100iops each and you have 24 drives across four nodes then you should see 2400iops. But now you add 10 more drives but they are 5200rpm and only give 75iops each, they do not add in like you would expect but actually pull the first 24 drives down to a figure thats closer to 90iops each while still only giving 75iops. As you can see it gets complicated very quickly to figure out why your cluster suddenly slower than expected. Again there are ways to mitigate this with Ceph commands but then you loose the auto balancing that makes Ceph so efficient.
Path assignments are linked to but separate of the underlying storage system. Just create your new cluster of NVMe drives and create a few rules to make them do as needed, it really no different from setting up SSDs as a second cluster on the nodes.
Path Assignment & Hard Drives
storageman
10 Posts
Quote from storageman on May 22, 2021, 6:33 pmI have a cluster with 4 nodes, 3 nodes have identical hard drives with same combination on SAS & SSD drives. Few days back we added 4th node, I have confusion that is we need to add same hard drive like other 3 nodes or we can add different type/size hdd's. Any complication?
Few more things for clarification:-
1. We are planning to add NVMe drives on all nodes but how to segregate NVMe drives to create separate NVMe pools like we are doing for SAS & SSD by adding rules.
2. Due to some unforeseen circumstances , Our 3rd node goes down for few hours so now all the path assigned to only node 1 & node 2, Node 3 & 4 are empty. How to reassign all path for balance cross all nodes. Best way to resolve without downtime. Any complication or some extra cautious. Please guide.
3. Identical HDDs required across all the cluster?
Thanks & Regards,
I have a cluster with 4 nodes, 3 nodes have identical hard drives with same combination on SAS & SSD drives. Few days back we added 4th node, I have confusion that is we need to add same hard drive like other 3 nodes or we can add different type/size hdd's. Any complication?
Few more things for clarification:-
1. We are planning to add NVMe drives on all nodes but how to segregate NVMe drives to create separate NVMe pools like we are doing for SAS & SSD by adding rules.
2. Due to some unforeseen circumstances , Our 3rd node goes down for few hours so now all the path assigned to only node 1 & node 2, Node 3 & 4 are empty. How to reassign all path for balance cross all nodes. Best way to resolve without downtime. Any complication or some extra cautious. Please guide.
3. Identical HDDs required across all the cluster?
Thanks & Regards,
Abhishek Kumar Singh
18 Posts
Quote from Abhishek Kumar Singh on May 28, 2021, 7:13 amAny update on this team? I have concern about identical hard disks across cluster?
Any update on this team? I have concern about identical hard disks across cluster?
admin
2,930 Posts
Quote from admin on May 29, 2021, 3:42 pmThe system will work if you use different disk models and disk sizes.
However the slowest disk will become the bottleneck of the entire pool, even a disk with larger size will get more traffic and hence will also become a bottleneck. so for better performance try to use disks that are as identical as possible in terms of size and performance characteristics, in many cases try to stick with same vendor and same product line.
The system will work if you use different disk models and disk sizes.
However the slowest disk will become the bottleneck of the entire pool, even a disk with larger size will get more traffic and hence will also become a bottleneck. so for better performance try to use disks that are as identical as possible in terms of size and performance characteristics, in many cases try to stick with same vendor and same product line.
storageman
10 Posts
Quote from storageman on May 30, 2021, 6:33 amThank you PetaSAN team, you guys are really awesome.
Still our disk path are on only node 1 & node 2, We need to re-balance the cluster with equal amount of path across the cluster without downtime or any complication.
and
How we can segregate NVMe disks like normal SSD or SAS. if it is not possible surly we can save few bucks.
Thank you PetaSAN team, you guys are really awesome.
Still our disk path are on only node 1 & node 2, We need to re-balance the cluster with equal amount of path across the cluster without downtime or any complication.
and
How we can segregate NVMe disks like normal SSD or SAS. if it is not possible surly we can save few bucks.
admin
2,930 Posts
Quote from admin on June 2, 2021, 3:02 pmYou can use the path assignment page to dynamically move paths around, it has an auto feature to balance dynamically
NVME are detected as SSD, you can set a device custom class via
ceph osd crush set-device-class NEWCLASS osd.X osd.Y
then you will need to create a crush rule to place data on that new class. Note currently with Ceph, pure NVMEs will yield same performance as SSDs due to the distributed latency.
You can use the path assignment page to dynamically move paths around, it has an auto feature to balance dynamically
NVME are detected as SSD, you can set a device custom class via
ceph osd crush set-device-class NEWCLASS osd.X osd.Y
then you will need to create a crush rule to place data on that new class. Note currently with Ceph, pure NVMEs will yield same performance as SSDs due to the distributed latency.
Shiori
86 Posts
Quote from Shiori on December 17, 2021, 2:29 pmjust to add some more:
We add OSDs as needed and have found that a drive that is twice as large as the others will see up to four times the load. This is a Ceph thing and can be tuned but then you are basically telling Ceph that your 6tb drive is the same size as your 3tb drives. This is not a bad thing though as it is useful when upgrading a live cluster to larger drives and allowing them to finish replication before rebalancing the cluster.
By keeping to the same size and speed of drives, you make performance more predictable and easier to calculate in scale. if your 2tb 7200rpm drives give you 100iops each and you have 24 drives across four nodes then you should see 2400iops. But now you add 10 more drives but they are 5200rpm and only give 75iops each, they do not add in like you would expect but actually pull the first 24 drives down to a figure thats closer to 90iops each while still only giving 75iops. As you can see it gets complicated very quickly to figure out why your cluster suddenly slower than expected. Again there are ways to mitigate this with Ceph commands but then you loose the auto balancing that makes Ceph so efficient.
Path assignments are linked to but separate of the underlying storage system. Just create your new cluster of NVMe drives and create a few rules to make them do as needed, it really no different from setting up SSDs as a second cluster on the nodes.
just to add some more:
We add OSDs as needed and have found that a drive that is twice as large as the others will see up to four times the load. This is a Ceph thing and can be tuned but then you are basically telling Ceph that your 6tb drive is the same size as your 3tb drives. This is not a bad thing though as it is useful when upgrading a live cluster to larger drives and allowing them to finish replication before rebalancing the cluster.
By keeping to the same size and speed of drives, you make performance more predictable and easier to calculate in scale. if your 2tb 7200rpm drives give you 100iops each and you have 24 drives across four nodes then you should see 2400iops. But now you add 10 more drives but they are 5200rpm and only give 75iops each, they do not add in like you would expect but actually pull the first 24 drives down to a figure thats closer to 90iops each while still only giving 75iops. As you can see it gets complicated very quickly to figure out why your cluster suddenly slower than expected. Again there are ways to mitigate this with Ceph commands but then you loose the auto balancing that makes Ceph so efficient.
Path assignments are linked to but separate of the underlying storage system. Just create your new cluster of NVMe drives and create a few rules to make them do as needed, it really no different from setting up SSDs as a second cluster on the nodes.