SSD storage for a small cluster
blokmech
3 Posts
January 19, 2023, 2:44 pmQuote from blokmech on January 19, 2023, 2:44 pmHi all,
I tested (virtual) PetaSAN and I'm impressed. I would like to build a 3 node cluster for a small Hyper-v cluster for 6 vm's. The reason to do this setup will be high availability.
I ordered already 3 refurbished R630 with 10gb nics, and now I have to choose what (SSD) disks I will put into it. I now only need 8TB of storage, but the hardware will have enough bays left for expansion.
I want to use Samsung PM1643a disks, but I'm not sure in what configuration. Should I use 4 x 1,92TB for the OSD's, or will 2 x 3.84 TB also be sufficient? Wil 2 x 3.84 TB disk already fully saturate my 10GB nics?
Please advise.
Regards,
Nico.
Hi all,
I tested (virtual) PetaSAN and I'm impressed. I would like to build a 3 node cluster for a small Hyper-v cluster for 6 vm's. The reason to do this setup will be high availability.
I ordered already 3 refurbished R630 with 10gb nics, and now I have to choose what (SSD) disks I will put into it. I now only need 8TB of storage, but the hardware will have enough bays left for expansion.
I want to use Samsung PM1643a disks, but I'm not sure in what configuration. Should I use 4 x 1,92TB for the OSD's, or will 2 x 3.84 TB also be sufficient? Wil 2 x 3.84 TB disk already fully saturate my 10GB nics?
Please advise.
Regards,
Nico.
admin
2,930 Posts
January 19, 2023, 8:32 pmQuote from admin on January 19, 2023, 8:32 pmOne important factor is what block size or average block size is your workload. A 4MB block size could saturate a 10 GB network with 2 or so disk, a 4KB workload will not saturate it even with 20 disks. Backup load tend to use large block sizes, virtualisation and database tend to use small block size (more iops/latency demanding).
That may not give a yes/no answer to your question, but the advantage of having many OSD/disks is you will have more concurrency, using more cpu cores which will lead to higher iops and thus better throughput performance when using small block size.
One important factor is what block size or average block size is your workload. A 4MB block size could saturate a 10 GB network with 2 or so disk, a 4KB workload will not saturate it even with 20 disks. Backup load tend to use large block sizes, virtualisation and database tend to use small block size (more iops/latency demanding).
That may not give a yes/no answer to your question, but the advantage of having many OSD/disks is you will have more concurrency, using more cpu cores which will lead to higher iops and thus better throughput performance when using small block size.
Last edited on January 19, 2023, 8:34 pm by admin · #2
blokmech
3 Posts
January 19, 2023, 10:39 pmQuote from blokmech on January 19, 2023, 10:39 pmThanks for the input. I understand.
At the moment the vms's are running on the local storage of the hyper-v server. This is a Raid 5 local volume. The current disks are SATA SSD's with the following specs:
Speed: 520 MBps (read) / 480 MBps (write)
Random read (4K): 58 500 IOPS
Random write (4K): 29 000 IOPS
The performance data for the new 3.84 TB disks are:
Speed: 2100 MBps (read) / 2000 MBps (write)
Random read (4K): 450 000 IOPS
Random write (4K): 58 000 IOPS
Do you expect that with the PetaSAN cluster and the disk layout with 2 3.84 TB OSD's per Node will perform comparable with my current setup with local storage?
My servers will have cores enough to handle more disks, but 6 bigger drives are much cheaper than 12 smaller ones (because of the 3 nodes).
Thanks for the input. I understand.
At the moment the vms's are running on the local storage of the hyper-v server. This is a Raid 5 local volume. The current disks are SATA SSD's with the following specs:
Speed: 520 MBps (read) / 480 MBps (write)
Random read (4K): 58 500 IOPS
Random write (4K): 29 000 IOPS
The performance data for the new 3.84 TB disks are:
Speed: 2100 MBps (read) / 2000 MBps (write)
Random read (4K): 450 000 IOPS
Random write (4K): 58 000 IOPS
Do you expect that with the PetaSAN cluster and the disk layout with 2 3.84 TB OSD's per Node will perform comparable with my current setup with local storage?
My servers will have cores enough to handle more disks, but 6 bigger drives are much cheaper than 12 smaller ones (because of the 3 nodes).
Last edited on January 20, 2023, 7:29 am by blokmech · #3
admin
2,930 Posts
January 20, 2023, 9:37 amQuote from admin on January 20, 2023, 9:37 amThe throughput MB/s is not an issue, but the iops will be lower specially for writes. Latency for a local setup is much lower than going across the network for replication in a networked SAN in addition a distributed storage system will incur additional latency due to the software services layer (OSDs) involved.
The more OSDs you add per node, the higher the iops you'll get until your cpu cores max out , so the more cores and cpu power the have the better. It is quite depended on the hardware, if you can setup a 3 nodes node cluster with 1 or few disks and perform benchmark from PetaSAN UI, it will tell you the speed as well as resource % busy so you can know how your cpu and disks are doing and if you can add more disks or are your cores maxed.
The throughput MB/s is not an issue, but the iops will be lower specially for writes. Latency for a local setup is much lower than going across the network for replication in a networked SAN in addition a distributed storage system will incur additional latency due to the software services layer (OSDs) involved.
The more OSDs you add per node, the higher the iops you'll get until your cpu cores max out , so the more cores and cpu power the have the better. It is quite depended on the hardware, if you can setup a 3 nodes node cluster with 1 or few disks and perform benchmark from PetaSAN UI, it will tell you the speed as well as resource % busy so you can know how your cpu and disks are doing and if you can add more disks or are your cores maxed.
blokmech
3 Posts
January 20, 2023, 11:06 amQuote from blokmech on January 20, 2023, 11:06 amOkay. Thx for the response again. I will go for the smaller drives.
Okay. Thx for the response again. I will go for the smaller drives.
SSD storage for a small cluster
blokmech
3 Posts
Quote from blokmech on January 19, 2023, 2:44 pmHi all,
I tested (virtual) PetaSAN and I'm impressed. I would like to build a 3 node cluster for a small Hyper-v cluster for 6 vm's. The reason to do this setup will be high availability.
I ordered already 3 refurbished R630 with 10gb nics, and now I have to choose what (SSD) disks I will put into it. I now only need 8TB of storage, but the hardware will have enough bays left for expansion.
I want to use Samsung PM1643a disks, but I'm not sure in what configuration. Should I use 4 x 1,92TB for the OSD's, or will 2 x 3.84 TB also be sufficient? Wil 2 x 3.84 TB disk already fully saturate my 10GB nics?
Please advise.
Regards,
Nico.
Hi all,
I tested (virtual) PetaSAN and I'm impressed. I would like to build a 3 node cluster for a small Hyper-v cluster for 6 vm's. The reason to do this setup will be high availability.
I ordered already 3 refurbished R630 with 10gb nics, and now I have to choose what (SSD) disks I will put into it. I now only need 8TB of storage, but the hardware will have enough bays left for expansion.
I want to use Samsung PM1643a disks, but I'm not sure in what configuration. Should I use 4 x 1,92TB for the OSD's, or will 2 x 3.84 TB also be sufficient? Wil 2 x 3.84 TB disk already fully saturate my 10GB nics?
Please advise.
Regards,
Nico.
admin
2,930 Posts
Quote from admin on January 19, 2023, 8:32 pmOne important factor is what block size or average block size is your workload. A 4MB block size could saturate a 10 GB network with 2 or so disk, a 4KB workload will not saturate it even with 20 disks. Backup load tend to use large block sizes, virtualisation and database tend to use small block size (more iops/latency demanding).
That may not give a yes/no answer to your question, but the advantage of having many OSD/disks is you will have more concurrency, using more cpu cores which will lead to higher iops and thus better throughput performance when using small block size.
One important factor is what block size or average block size is your workload. A 4MB block size could saturate a 10 GB network with 2 or so disk, a 4KB workload will not saturate it even with 20 disks. Backup load tend to use large block sizes, virtualisation and database tend to use small block size (more iops/latency demanding).
That may not give a yes/no answer to your question, but the advantage of having many OSD/disks is you will have more concurrency, using more cpu cores which will lead to higher iops and thus better throughput performance when using small block size.
blokmech
3 Posts
Quote from blokmech on January 19, 2023, 10:39 pmThanks for the input. I understand.
At the moment the vms's are running on the local storage of the hyper-v server. This is a Raid 5 local volume. The current disks are SATA SSD's with the following specs:
Speed: 520 MBps (read) / 480 MBps (write)
Random read (4K): 58 500 IOPS
Random write (4K): 29 000 IOPSThe performance data for the new 3.84 TB disks are:
Speed: 2100 MBps (read) / 2000 MBps (write)
Random read (4K): 450 000 IOPS
Random write (4K): 58 000 IOPSDo you expect that with the PetaSAN cluster and the disk layout with 2 3.84 TB OSD's per Node will perform comparable with my current setup with local storage?
My servers will have cores enough to handle more disks, but 6 bigger drives are much cheaper than 12 smaller ones (because of the 3 nodes).
Thanks for the input. I understand.
At the moment the vms's are running on the local storage of the hyper-v server. This is a Raid 5 local volume. The current disks are SATA SSD's with the following specs:
Speed: 520 MBps (read) / 480 MBps (write)
Random read (4K): 58 500 IOPS
Random write (4K): 29 000 IOPS
The performance data for the new 3.84 TB disks are:
Speed: 2100 MBps (read) / 2000 MBps (write)
Random read (4K): 450 000 IOPS
Random write (4K): 58 000 IOPS
Do you expect that with the PetaSAN cluster and the disk layout with 2 3.84 TB OSD's per Node will perform comparable with my current setup with local storage?
My servers will have cores enough to handle more disks, but 6 bigger drives are much cheaper than 12 smaller ones (because of the 3 nodes).
admin
2,930 Posts
Quote from admin on January 20, 2023, 9:37 amThe throughput MB/s is not an issue, but the iops will be lower specially for writes. Latency for a local setup is much lower than going across the network for replication in a networked SAN in addition a distributed storage system will incur additional latency due to the software services layer (OSDs) involved.
The more OSDs you add per node, the higher the iops you'll get until your cpu cores max out , so the more cores and cpu power the have the better. It is quite depended on the hardware, if you can setup a 3 nodes node cluster with 1 or few disks and perform benchmark from PetaSAN UI, it will tell you the speed as well as resource % busy so you can know how your cpu and disks are doing and if you can add more disks or are your cores maxed.
The throughput MB/s is not an issue, but the iops will be lower specially for writes. Latency for a local setup is much lower than going across the network for replication in a networked SAN in addition a distributed storage system will incur additional latency due to the software services layer (OSDs) involved.
The more OSDs you add per node, the higher the iops you'll get until your cpu cores max out , so the more cores and cpu power the have the better. It is quite depended on the hardware, if you can setup a 3 nodes node cluster with 1 or few disks and perform benchmark from PetaSAN UI, it will tell you the speed as well as resource % busy so you can know how your cpu and disks are doing and if you can add more disks or are your cores maxed.
blokmech
3 Posts
Quote from blokmech on January 20, 2023, 11:06 amOkay. Thx for the response again. I will go for the smaller drives.
Okay. Thx for the response again. I will go for the smaller drives.