Could you please review my hardware architecture plan?
fernoliv
6 Posts
June 3, 2019, 3:28 amQuote from fernoliv on June 3, 2019, 3:28 amHi admin,
I'm building a new environment using that hardware below and I need some tips if possible:
8x 4U Supermicro Storage Expander 3.5" 45 Bay JBOD CSE-PTJBOD-CB2
360x HGST (Hitachi) 4TB SAS 3.5" drives 7200RPM as OSDs (45 disks per chassis)
12x Intel SSD S3710 400GB as WAL+DB (these drives will be installed on R720XD using its internal RAID controller and each will be presented to CEPH as a RAID0 device (12 drives per R720XD node)
8x R720XD with 256GB of memory DDR3 12800R, 2x Intel processors E5-2680V2 10-core and 2x Intel SSD for OS in RAID1
1x dual-port 40Gb Intel NIC in LACP (each port on a different Arista switch) for CEPH network and replication using different VLANs for each traffic (80Gb dynamicly divided by the two different traffics as needed) per node
1x dual-port 10Gb Intel NIC for iSCSI traffic (each port on a different Arista switch) for iSCSI 1 and 2 using different VLANs per node
I started this project with 2x HBA cards LSI SAS9207-8E per R720XD server, where the first one is controlling 24 front SAS drives and the second one the other 21 drives in the back of the Supermicro Expander chassis.
I was reading other topics in the forum and you are recommending to use RAID controllers with write-back and BBU to increase the mechanical disks performance twice. But CEPH developers always recommended to use HBA to pass-through all devices directly to CEPH take about.
If I follow your recommendation to use a RAID card, I'm thinking in replace the 2x external HBAs per node with 2x LSI Megaraid 9286CV-8E 6GB/S 1G cache with capacitor and create a lot of RAID0 devices to present to CEPH. I'm using 8088 external cables to interconnect the Supermicro chassis with the Dell R720XD servers.
In this way, how PetaSAN will have access to SMART information from the disks? Will it really improves the I/O in hybrid environments like that?
I have also some old Dell R610 nodes with 96GB of RAM memory and Intel X5690 processors that I could use to be dedicated monitor or even iSCSI nodes. What you recommend for me? Is worth use old servers as dedicated monitors or iSCSI or still being better run the monitor and iSCSI services inside of the 8 nodes?
In the end, it will be a big cluster with 360 OSDs of 4TB each one, 4 OSDs per Intel SSD WAL+DB with around 90GB of journal for each OSD (you recommend 60GB per OSD as minimum size), with enough RAM memory for CEPH (4GB per OSD that means 180GB and more 76GB for OS and other PetaSAN services (Consul, graphs, Webgui, etc).
Sorry for this long topic and several questions, but it is very important for me and my small business.
Have a great week guys!
Hi admin,
I'm building a new environment using that hardware below and I need some tips if possible:
8x 4U Supermicro Storage Expander 3.5" 45 Bay JBOD CSE-PTJBOD-CB2
360x HGST (Hitachi) 4TB SAS 3.5" drives 7200RPM as OSDs (45 disks per chassis)
12x Intel SSD S3710 400GB as WAL+DB (these drives will be installed on R720XD using its internal RAID controller and each will be presented to CEPH as a RAID0 device (12 drives per R720XD node)
8x R720XD with 256GB of memory DDR3 12800R, 2x Intel processors E5-2680V2 10-core and 2x Intel SSD for OS in RAID1
1x dual-port 40Gb Intel NIC in LACP (each port on a different Arista switch) for CEPH network and replication using different VLANs for each traffic (80Gb dynamicly divided by the two different traffics as needed) per node
1x dual-port 10Gb Intel NIC for iSCSI traffic (each port on a different Arista switch) for iSCSI 1 and 2 using different VLANs per node
I started this project with 2x HBA cards LSI SAS9207-8E per R720XD server, where the first one is controlling 24 front SAS drives and the second one the other 21 drives in the back of the Supermicro Expander chassis.
I was reading other topics in the forum and you are recommending to use RAID controllers with write-back and BBU to increase the mechanical disks performance twice. But CEPH developers always recommended to use HBA to pass-through all devices directly to CEPH take about.
If I follow your recommendation to use a RAID card, I'm thinking in replace the 2x external HBAs per node with 2x LSI Megaraid 9286CV-8E 6GB/S 1G cache with capacitor and create a lot of RAID0 devices to present to CEPH. I'm using 8088 external cables to interconnect the Supermicro chassis with the Dell R720XD servers.
In this way, how PetaSAN will have access to SMART information from the disks? Will it really improves the I/O in hybrid environments like that?
I have also some old Dell R610 nodes with 96GB of RAM memory and Intel X5690 processors that I could use to be dedicated monitor or even iSCSI nodes. What you recommend for me? Is worth use old servers as dedicated monitors or iSCSI or still being better run the monitor and iSCSI services inside of the 8 nodes?
In the end, it will be a big cluster with 360 OSDs of 4TB each one, 4 OSDs per Intel SSD WAL+DB with around 90GB of journal for each OSD (you recommend 60GB per OSD as minimum size), with enough RAM memory for CEPH (4GB per OSD that means 180GB and more 76GB for OS and other PetaSAN services (Consul, graphs, Webgui, etc).
Sorry for this long topic and several questions, but it is very important for me and my small business.
Have a great week guys!
Last edited on June 3, 2019, 3:34 am by fernoliv · #1
admin
2,930 Posts
June 3, 2019, 4:32 pmQuote from admin on June 3, 2019, 4:32 pmYes it is our experience write-back cache gives a boost to spinning disks. If you can, build a small 3 node cluster yourself and have a couple of OSDs to test your hardware and see the impact of the cache.
Some controllers can pass SMART info accurately, others do not, so we just report back what the controller tells us.
For using old nodes, for monitors it is OK but iSCSI does require cpu resources, you can add/join to your cluster nodes that only act as iSCSI and test if they are good enough, if not you can easily delete/remove them from the cluster using the ui since they contain no storage.
Yes it is our experience write-back cache gives a boost to spinning disks. If you can, build a small 3 node cluster yourself and have a couple of OSDs to test your hardware and see the impact of the cache.
Some controllers can pass SMART info accurately, others do not, so we just report back what the controller tells us.
For using old nodes, for monitors it is OK but iSCSI does require cpu resources, you can add/join to your cluster nodes that only act as iSCSI and test if they are good enough, if not you can easily delete/remove them from the cluster using the ui since they contain no storage.
fernoliv
6 Posts
June 3, 2019, 11:26 pmQuote from fernoliv on June 3, 2019, 11:26 pmAwesome. Thanks a lot for your advice. I'll try it soon so.
Awesome. Thanks a lot for your advice. I'll try it soon so.
Could you please review my hardware architecture plan?
fernoliv
6 Posts
Quote from fernoliv on June 3, 2019, 3:28 amHi admin,
I'm building a new environment using that hardware below and I need some tips if possible:
8x 4U Supermicro Storage Expander 3.5" 45 Bay JBOD CSE-PTJBOD-CB2
360x HGST (Hitachi) 4TB SAS 3.5" drives 7200RPM as OSDs (45 disks per chassis)
12x Intel SSD S3710 400GB as WAL+DB (these drives will be installed on R720XD using its internal RAID controller and each will be presented to CEPH as a RAID0 device (12 drives per R720XD node)
8x R720XD with 256GB of memory DDR3 12800R, 2x Intel processors E5-2680V2 10-core and 2x Intel SSD for OS in RAID1
1x dual-port 40Gb Intel NIC in LACP (each port on a different Arista switch) for CEPH network and replication using different VLANs for each traffic (80Gb dynamicly divided by the two different traffics as needed) per node
1x dual-port 10Gb Intel NIC for iSCSI traffic (each port on a different Arista switch) for iSCSI 1 and 2 using different VLANs per node
I started this project with 2x HBA cards LSI SAS9207-8E per R720XD server, where the first one is controlling 24 front SAS drives and the second one the other 21 drives in the back of the Supermicro Expander chassis.
I was reading other topics in the forum and you are recommending to use RAID controllers with write-back and BBU to increase the mechanical disks performance twice. But CEPH developers always recommended to use HBA to pass-through all devices directly to CEPH take about.
If I follow your recommendation to use a RAID card, I'm thinking in replace the 2x external HBAs per node with 2x LSI Megaraid 9286CV-8E 6GB/S 1G cache with capacitor and create a lot of RAID0 devices to present to CEPH. I'm using 8088 external cables to interconnect the Supermicro chassis with the Dell R720XD servers.
In this way, how PetaSAN will have access to SMART information from the disks? Will it really improves the I/O in hybrid environments like that?
I have also some old Dell R610 nodes with 96GB of RAM memory and Intel X5690 processors that I could use to be dedicated monitor or even iSCSI nodes. What you recommend for me? Is worth use old servers as dedicated monitors or iSCSI or still being better run the monitor and iSCSI services inside of the 8 nodes?
In the end, it will be a big cluster with 360 OSDs of 4TB each one, 4 OSDs per Intel SSD WAL+DB with around 90GB of journal for each OSD (you recommend 60GB per OSD as minimum size), with enough RAM memory for CEPH (4GB per OSD that means 180GB and more 76GB for OS and other PetaSAN services (Consul, graphs, Webgui, etc).
Sorry for this long topic and several questions, but it is very important for me and my small business.
Have a great week guys!
Hi admin,
I'm building a new environment using that hardware below and I need some tips if possible:
8x 4U Supermicro Storage Expander 3.5" 45 Bay JBOD CSE-PTJBOD-CB2
360x HGST (Hitachi) 4TB SAS 3.5" drives 7200RPM as OSDs (45 disks per chassis)
12x Intel SSD S3710 400GB as WAL+DB (these drives will be installed on R720XD using its internal RAID controller and each will be presented to CEPH as a RAID0 device (12 drives per R720XD node)
8x R720XD with 256GB of memory DDR3 12800R, 2x Intel processors E5-2680V2 10-core and 2x Intel SSD for OS in RAID1
1x dual-port 40Gb Intel NIC in LACP (each port on a different Arista switch) for CEPH network and replication using different VLANs for each traffic (80Gb dynamicly divided by the two different traffics as needed) per node
1x dual-port 10Gb Intel NIC for iSCSI traffic (each port on a different Arista switch) for iSCSI 1 and 2 using different VLANs per node
I started this project with 2x HBA cards LSI SAS9207-8E per R720XD server, where the first one is controlling 24 front SAS drives and the second one the other 21 drives in the back of the Supermicro Expander chassis.
I was reading other topics in the forum and you are recommending to use RAID controllers with write-back and BBU to increase the mechanical disks performance twice. But CEPH developers always recommended to use HBA to pass-through all devices directly to CEPH take about.
If I follow your recommendation to use a RAID card, I'm thinking in replace the 2x external HBAs per node with 2x LSI Megaraid 9286CV-8E 6GB/S 1G cache with capacitor and create a lot of RAID0 devices to present to CEPH. I'm using 8088 external cables to interconnect the Supermicro chassis with the Dell R720XD servers.
In this way, how PetaSAN will have access to SMART information from the disks? Will it really improves the I/O in hybrid environments like that?
I have also some old Dell R610 nodes with 96GB of RAM memory and Intel X5690 processors that I could use to be dedicated monitor or even iSCSI nodes. What you recommend for me? Is worth use old servers as dedicated monitors or iSCSI or still being better run the monitor and iSCSI services inside of the 8 nodes?
In the end, it will be a big cluster with 360 OSDs of 4TB each one, 4 OSDs per Intel SSD WAL+DB with around 90GB of journal for each OSD (you recommend 60GB per OSD as minimum size), with enough RAM memory for CEPH (4GB per OSD that means 180GB and more 76GB for OS and other PetaSAN services (Consul, graphs, Webgui, etc).
Sorry for this long topic and several questions, but it is very important for me and my small business.
Have a great week guys!
admin
2,930 Posts
Quote from admin on June 3, 2019, 4:32 pmYes it is our experience write-back cache gives a boost to spinning disks. If you can, build a small 3 node cluster yourself and have a couple of OSDs to test your hardware and see the impact of the cache.
Some controllers can pass SMART info accurately, others do not, so we just report back what the controller tells us.
For using old nodes, for monitors it is OK but iSCSI does require cpu resources, you can add/join to your cluster nodes that only act as iSCSI and test if they are good enough, if not you can easily delete/remove them from the cluster using the ui since they contain no storage.
Yes it is our experience write-back cache gives a boost to spinning disks. If you can, build a small 3 node cluster yourself and have a couple of OSDs to test your hardware and see the impact of the cache.
Some controllers can pass SMART info accurately, others do not, so we just report back what the controller tells us.
For using old nodes, for monitors it is OK but iSCSI does require cpu resources, you can add/join to your cluster nodes that only act as iSCSI and test if they are good enough, if not you can easily delete/remove them from the cluster using the ui since they contain no storage.
fernoliv
6 Posts
Quote from fernoliv on June 3, 2019, 11:26 pmAwesome. Thanks a lot for your advice. I'll try it soon so.
Awesome. Thanks a lot for your advice. I'll try it soon so.