choice of physical disk number per node
tb
9 Posts
November 3, 2017, 6:47 amQuote from tb on November 3, 2017, 6:47 amHi.
What do you recommend ?
It is better to have a large number of disks per node and a low node number, or a large number of nodes with little disk per node?
Indeed, my client wants to optimize his electricity consumption. Less knot, less power consumption.
We will go on a cluster of 3 nodes, 11 disks per node. 2 x 1Gbit/ s network cards per node and 2 x 12Gbit / s cards per node, for a total of 4 network cards per node.
Management being confused with iSCSI 1 (card 1 Gbits/s)
iSCSI 2 (card 1 Gbits/s)
Backend 1 (card 12 Gbits/s)
Backend 2 (Card 12 Gbits/s)
The disks are identical and have a capacity of 2 TB. (With 1 small internal scsi disk for the system), 16 GB of RAM (2U server, Dell), per node.
What do you think ?
Thanks.
Hi.
What do you recommend ?
It is better to have a large number of disks per node and a low node number, or a large number of nodes with little disk per node?
Indeed, my client wants to optimize his electricity consumption. Less knot, less power consumption.
We will go on a cluster of 3 nodes, 11 disks per node. 2 x 1Gbit/ s network cards per node and 2 x 12Gbit / s cards per node, for a total of 4 network cards per node.
Management being confused with iSCSI 1 (card 1 Gbits/s)
iSCSI 2 (card 1 Gbits/s)
Backend 1 (card 12 Gbits/s)
Backend 2 (Card 12 Gbits/s)
The disks are identical and have a capacity of 2 TB. (With 1 small internal scsi disk for the system), 16 GB of RAM (2U server, Dell), per node.
What do you think ?
Thanks.
Last edited on November 3, 2017, 6:50 am by tb · #1
admin
2,930 Posts
November 3, 2017, 9:17 amQuote from admin on November 3, 2017, 9:17 amRAM is a lot below recommendation. Increase to 32 GB if possible,
Putting the iSCSI subnets on 1 Gb/s is limiting for both throughput MB/s and IOPS (due to latency). If you can use 10 Gb/s or maybe share them with the backend nics will give better performance.
Re number or nodes / disks: Theoretically the more cpu/disk/nics resources you can give (and afford) to Ceph the better, it will give you better performance since it scales out. But practically to get the best performance for your money you should tune your cluster so that no 1 resources is utilized more that others which means some other resources you paid money for are not utilized.
We help you with this in PetaSAN via the Cluster Benchmark page: in this page you can benchmark to top speed of your cluster and see what resources need tuning:
If your cpu % is low and disk % busy is high: your disks are the bottleneck and you are not making use of your cpu/net hardware...need to add more disks per node
If your cpu % is high and disk % busy is low: your disks are under utilized due to cpu limitations: more some disks to new node or add cpu socket if supported.
We also show you RAM and NIC utilizations which also can be the bottleneck
I addition to the benchmark page you can view such load stats and more from the dashboard charts under production load.
RAM is a lot below recommendation. Increase to 32 GB if possible,
Putting the iSCSI subnets on 1 Gb/s is limiting for both throughput MB/s and IOPS (due to latency). If you can use 10 Gb/s or maybe share them with the backend nics will give better performance.
Re number or nodes / disks: Theoretically the more cpu/disk/nics resources you can give (and afford) to Ceph the better, it will give you better performance since it scales out. But practically to get the best performance for your money you should tune your cluster so that no 1 resources is utilized more that others which means some other resources you paid money for are not utilized.
We help you with this in PetaSAN via the Cluster Benchmark page: in this page you can benchmark to top speed of your cluster and see what resources need tuning:
If your cpu % is low and disk % busy is high: your disks are the bottleneck and you are not making use of your cpu/net hardware...need to add more disks per node
If your cpu % is high and disk % busy is low: your disks are under utilized due to cpu limitations: more some disks to new node or add cpu socket if supported.
We also show you RAM and NIC utilizations which also can be the bottleneck
I addition to the benchmark page you can view such load stats and more from the dashboard charts under production load.
Last edited on November 3, 2017, 9:23 am by admin · #2
tb
9 Posts
November 6, 2017, 3:37 pmQuote from tb on November 6, 2017, 3:37 pmOk. I am understand. Last version of system and network :
3 HP DL160 G6 4 LFF with 4 Caddy Hotplug + 1x120Go SSD for system per node + 2 Network Cards PciE 10Gbps per node + 2 NetWork Cards 1 Gbps per node
32 Go RAM per node DDR3 Ecc Reg and 2 processors Intel Xeon L5520 per node.
12 HDD Hitachi 2 To SATA 7,2K
2 HP A5820AF-24XG Ports 10Gbps stackable
2 Cisco 3750E-48TD-S Stackable with 2 moduls : 2 x 10Gpbs (46 ports 1 Gbps)
It's good now ?
Ok. I am understand. Last version of system and network :
3 HP DL160 G6 4 LFF with 4 Caddy Hotplug + 1x120Go SSD for system per node + 2 Network Cards PciE 10Gbps per node + 2 NetWork Cards 1 Gbps per node
32 Go RAM per node DDR3 Ecc Reg and 2 processors Intel Xeon L5520 per node.
12 HDD Hitachi 2 To SATA 7,2K
2 HP A5820AF-24XG Ports 10Gbps stackable
2 Cisco 3750E-48TD-S Stackable with 2 moduls : 2 x 10Gpbs (46 ports 1 Gbps)
It's good now ?
Last edited on November 6, 2017, 3:43 pm by tb · #3
tb
9 Posts
November 6, 2017, 5:50 pmQuote from tb on November 6, 2017, 5:50 pm
Last edited on November 6, 2017, 5:55 pm by tb · #4
admin
2,930 Posts
November 7, 2017, 10:24 amQuote from admin on November 7, 2017, 10:24 amHard to give an accurate hardware recommendationm there are so many variations /costs. The best thing is to benchmark it and try to tune it as per above.
The G6 is a bit old. If you use 12 spinning disks in total (4 per node), you may find your write iops will not exceed 200 iops for small block size writes (4k), but generally for throughput read/write and read iops should be fine.
Hard to give an accurate hardware recommendationm there are so many variations /costs. The best thing is to benchmark it and try to tune it as per above.
The G6 is a bit old. If you use 12 spinning disks in total (4 per node), you may find your write iops will not exceed 200 iops for small block size writes (4k), but generally for throughput read/write and read iops should be fine.
Last edited on November 7, 2017, 10:25 am by admin · #5
tb
9 Posts
choice of physical disk number per node
tb
9 Posts
Quote from tb on November 3, 2017, 6:47 amHi.
What do you recommend ?
It is better to have a large number of disks per node and a low node number, or a large number of nodes with little disk per node?
Indeed, my client wants to optimize his electricity consumption. Less knot, less power consumption.
We will go on a cluster of 3 nodes, 11 disks per node. 2 x 1Gbit/ s network cards per node and 2 x 12Gbit / s cards per node, for a total of 4 network cards per node.
Management being confused with iSCSI 1 (card 1 Gbits/s)
iSCSI 2 (card 1 Gbits/s)
Backend 1 (card 12 Gbits/s)
Backend 2 (Card 12 Gbits/s)The disks are identical and have a capacity of 2 TB. (With 1 small internal scsi disk for the system), 16 GB of RAM (2U server, Dell), per node.
What do you think ?
Thanks.
Hi.
What do you recommend ?
It is better to have a large number of disks per node and a low node number, or a large number of nodes with little disk per node?
Indeed, my client wants to optimize his electricity consumption. Less knot, less power consumption.
We will go on a cluster of 3 nodes, 11 disks per node. 2 x 1Gbit/ s network cards per node and 2 x 12Gbit / s cards per node, for a total of 4 network cards per node.
Management being confused with iSCSI 1 (card 1 Gbits/s)
iSCSI 2 (card 1 Gbits/s)
Backend 1 (card 12 Gbits/s)
Backend 2 (Card 12 Gbits/s)
The disks are identical and have a capacity of 2 TB. (With 1 small internal scsi disk for the system), 16 GB of RAM (2U server, Dell), per node.
What do you think ?
Thanks.
admin
2,930 Posts
Quote from admin on November 3, 2017, 9:17 amRAM is a lot below recommendation. Increase to 32 GB if possible,
Putting the iSCSI subnets on 1 Gb/s is limiting for both throughput MB/s and IOPS (due to latency). If you can use 10 Gb/s or maybe share them with the backend nics will give better performance.
Re number or nodes / disks: Theoretically the more cpu/disk/nics resources you can give (and afford) to Ceph the better, it will give you better performance since it scales out. But practically to get the best performance for your money you should tune your cluster so that no 1 resources is utilized more that others which means some other resources you paid money for are not utilized.
We help you with this in PetaSAN via the Cluster Benchmark page: in this page you can benchmark to top speed of your cluster and see what resources need tuning:
If your cpu % is low and disk % busy is high: your disks are the bottleneck and you are not making use of your cpu/net hardware...need to add more disks per node
If your cpu % is high and disk % busy is low: your disks are under utilized due to cpu limitations: more some disks to new node or add cpu socket if supported.
We also show you RAM and NIC utilizations which also can be the bottleneck
I addition to the benchmark page you can view such load stats and more from the dashboard charts under production load.
RAM is a lot below recommendation. Increase to 32 GB if possible,
Putting the iSCSI subnets on 1 Gb/s is limiting for both throughput MB/s and IOPS (due to latency). If you can use 10 Gb/s or maybe share them with the backend nics will give better performance.
Re number or nodes / disks: Theoretically the more cpu/disk/nics resources you can give (and afford) to Ceph the better, it will give you better performance since it scales out. But practically to get the best performance for your money you should tune your cluster so that no 1 resources is utilized more that others which means some other resources you paid money for are not utilized.
We help you with this in PetaSAN via the Cluster Benchmark page: in this page you can benchmark to top speed of your cluster and see what resources need tuning:
If your cpu % is low and disk % busy is high: your disks are the bottleneck and you are not making use of your cpu/net hardware...need to add more disks per node
If your cpu % is high and disk % busy is low: your disks are under utilized due to cpu limitations: more some disks to new node or add cpu socket if supported.
We also show you RAM and NIC utilizations which also can be the bottleneck
I addition to the benchmark page you can view such load stats and more from the dashboard charts under production load.
tb
9 Posts
Quote from tb on November 6, 2017, 3:37 pmOk. I am understand. Last version of system and network :
3 HP DL160 G6 4 LFF with 4 Caddy Hotplug + 1x120Go SSD for system per node + 2 Network Cards PciE 10Gbps per node + 2 NetWork Cards 1 Gbps per node
32 Go RAM per node DDR3 Ecc Reg and 2 processors Intel Xeon L5520 per node.12 HDD Hitachi 2 To SATA 7,2K
2 HP A5820AF-24XG Ports 10Gbps stackable
2 Cisco 3750E-48TD-S Stackable with 2 moduls : 2 x 10Gpbs (46 ports 1 Gbps)
It's good now ?
Ok. I am understand. Last version of system and network :
3 HP DL160 G6 4 LFF with 4 Caddy Hotplug + 1x120Go SSD for system per node + 2 Network Cards PciE 10Gbps per node + 2 NetWork Cards 1 Gbps per node
32 Go RAM per node DDR3 Ecc Reg and 2 processors Intel Xeon L5520 per node.
12 HDD Hitachi 2 To SATA 7,2K
2 HP A5820AF-24XG Ports 10Gbps stackable
2 Cisco 3750E-48TD-S Stackable with 2 moduls : 2 x 10Gpbs (46 ports 1 Gbps)
It's good now ?
tb
9 Posts
Quote from tb on November 6, 2017, 5:50 pm
admin
2,930 Posts
Quote from admin on November 7, 2017, 10:24 amHard to give an accurate hardware recommendationm there are so many variations /costs. The best thing is to benchmark it and try to tune it as per above.
The G6 is a bit old. If you use 12 spinning disks in total (4 per node), you may find your write iops will not exceed 200 iops for small block size writes (4k), but generally for throughput read/write and read iops should be fine.
Hard to give an accurate hardware recommendationm there are so many variations /costs. The best thing is to benchmark it and try to tune it as per above.
The G6 is a bit old. If you use 12 spinning disks in total (4 per node), you may find your write iops will not exceed 200 iops for small block size writes (4k), but generally for throughput read/write and read iops should be fine.
tb
9 Posts