Storage and Journal and SSDs
MannIT
1 Post
December 5, 2018, 10:20 pmQuote from MannIT on December 5, 2018, 10:20 pmHi,
thank you for you great work.
I will build an Cluster with 4 Nodes an 12OSD per node.
Boot 120GB SSD, 12x 4TB per node.
And 1 Node with 25x SSDs
I read many about OSD, Journal an so on.
OSD is clear and need no question.
Journal what ist the best option for it?
Faster is better? NVE,SSD or is a disk enough.
Which Size do I need? 1TB or 512Gb? 1 Journal DISK per node or 2?
What is when I use one and it lost du failure? Problem or never mind ? Replace and it works again
I read that is not good do mix SSD and HDD in one node.
So i think I build 1 Server with 25xSSDs, do petsan use it automatic to speed up?
For security it is better to have 2 nodes with 25xSSDs?
Greetings Mario
Hi,
thank you for you great work.
I will build an Cluster with 4 Nodes an 12OSD per node.
Boot 120GB SSD, 12x 4TB per node.
And 1 Node with 25x SSDs
I read many about OSD, Journal an so on.
OSD is clear and need no question.
Journal what ist the best option for it?
Faster is better? NVE,SSD or is a disk enough.
Which Size do I need? 1TB or 512Gb? 1 Journal DISK per node or 2?
What is when I use one and it lost du failure? Problem or never mind ? Replace and it works again
I read that is not good do mix SSD and HDD in one node.
So i think I build 1 Server with 25xSSDs, do petsan use it automatic to speed up?
For security it is better to have 2 nodes with 25xSSDs?
Greetings Mario
admin
2,930 Posts
December 6, 2018, 6:51 amQuote from admin on December 6, 2018, 6:51 amSome general advise:
Unless you are going for a high end system, using ssds without external nvme journals is good. Get enterprise rather than consumer ssds, Samsung PM/SM863a Intel S4500/S4600 series are good ssds with Ceph as they offer good performance and lifespan. Many low end ssds will give poor performance when doing sync writes, you can get a difference of more than 10x if using a good disk, you can use the PetaSAN node console screen (blue screen) to do raw disk performance and check the sync writes speed.
The next step is if you have the budget is to use nvme as journals, with a ratio of 1:4 sata ssds. Some good nvmes are Intel P3600/P3700/P4500/P4600 series Samsung PM963.
Try to use nodes that have the same number of OSDs and have the OSDs of the same type and size, else you will get more load on specific nodes and/or disks which will create a performance bottleneck.
The more OSDs per host you have the better performance, untill your cpu cores saturate so it does not matter to add more. So it is better to have 16 x 4TB hdds rather than 8 x 8 TB hdds. With ssds, this applies to a limit since the cpu will saturate sooner ( typically 12 ssds or earlier, but depends on cores and disks).
If you have a mix of hdds and ssds, you can use ssds as journals to hdds and/or create 2 storage pools: 1 slow for hdds ( use for backups etc ) and a faster ssd pool. Do not mix hdds and ssds as osds in the same pool.
Some general advise:
Unless you are going for a high end system, using ssds without external nvme journals is good. Get enterprise rather than consumer ssds, Samsung PM/SM863a Intel S4500/S4600 series are good ssds with Ceph as they offer good performance and lifespan. Many low end ssds will give poor performance when doing sync writes, you can get a difference of more than 10x if using a good disk, you can use the PetaSAN node console screen (blue screen) to do raw disk performance and check the sync writes speed.
The next step is if you have the budget is to use nvme as journals, with a ratio of 1:4 sata ssds. Some good nvmes are Intel P3600/P3700/P4500/P4600 series Samsung PM963.
Try to use nodes that have the same number of OSDs and have the OSDs of the same type and size, else you will get more load on specific nodes and/or disks which will create a performance bottleneck.
The more OSDs per host you have the better performance, untill your cpu cores saturate so it does not matter to add more. So it is better to have 16 x 4TB hdds rather than 8 x 8 TB hdds. With ssds, this applies to a limit since the cpu will saturate sooner ( typically 12 ssds or earlier, but depends on cores and disks).
If you have a mix of hdds and ssds, you can use ssds as journals to hdds and/or create 2 storage pools: 1 slow for hdds ( use for backups etc ) and a faster ssd pool. Do not mix hdds and ssds as osds in the same pool.
Storage and Journal and SSDs
MannIT
1 Post
Quote from MannIT on December 5, 2018, 10:20 pmHi,
thank you for you great work.
I will build an Cluster with 4 Nodes an 12OSD per node.
Boot 120GB SSD, 12x 4TB per node.
And 1 Node with 25x SSDsI read many about OSD, Journal an so on.
OSD is clear and need no question.
Journal what ist the best option for it?
Faster is better? NVE,SSD or is a disk enough.
Which Size do I need? 1TB or 512Gb? 1 Journal DISK per node or 2?
What is when I use one and it lost du failure? Problem or never mind ? Replace and it works againI read that is not good do mix SSD and HDD in one node.
So i think I build 1 Server with 25xSSDs, do petsan use it automatic to speed up?
For security it is better to have 2 nodes with 25xSSDs?Greetings Mario
Hi,
thank you for you great work.
I will build an Cluster with 4 Nodes an 12OSD per node.
Boot 120GB SSD, 12x 4TB per node.
And 1 Node with 25x SSDs
I read many about OSD, Journal an so on.
OSD is clear and need no question.
Journal what ist the best option for it?
Faster is better? NVE,SSD or is a disk enough.
Which Size do I need? 1TB or 512Gb? 1 Journal DISK per node or 2?
What is when I use one and it lost du failure? Problem or never mind ? Replace and it works again
I read that is not good do mix SSD and HDD in one node.
So i think I build 1 Server with 25xSSDs, do petsan use it automatic to speed up?
For security it is better to have 2 nodes with 25xSSDs?
Greetings Mario
admin
2,930 Posts
Quote from admin on December 6, 2018, 6:51 amSome general advise:
Unless you are going for a high end system, using ssds without external nvme journals is good. Get enterprise rather than consumer ssds, Samsung PM/SM863a Intel S4500/S4600 series are good ssds with Ceph as they offer good performance and lifespan. Many low end ssds will give poor performance when doing sync writes, you can get a difference of more than 10x if using a good disk, you can use the PetaSAN node console screen (blue screen) to do raw disk performance and check the sync writes speed.
The next step is if you have the budget is to use nvme as journals, with a ratio of 1:4 sata ssds. Some good nvmes are Intel P3600/P3700/P4500/P4600 series Samsung PM963.
Try to use nodes that have the same number of OSDs and have the OSDs of the same type and size, else you will get more load on specific nodes and/or disks which will create a performance bottleneck.
The more OSDs per host you have the better performance, untill your cpu cores saturate so it does not matter to add more. So it is better to have 16 x 4TB hdds rather than 8 x 8 TB hdds. With ssds, this applies to a limit since the cpu will saturate sooner ( typically 12 ssds or earlier, but depends on cores and disks).
If you have a mix of hdds and ssds, you can use ssds as journals to hdds and/or create 2 storage pools: 1 slow for hdds ( use for backups etc ) and a faster ssd pool. Do not mix hdds and ssds as osds in the same pool.
Some general advise:
Unless you are going for a high end system, using ssds without external nvme journals is good. Get enterprise rather than consumer ssds, Samsung PM/SM863a Intel S4500/S4600 series are good ssds with Ceph as they offer good performance and lifespan. Many low end ssds will give poor performance when doing sync writes, you can get a difference of more than 10x if using a good disk, you can use the PetaSAN node console screen (blue screen) to do raw disk performance and check the sync writes speed.
The next step is if you have the budget is to use nvme as journals, with a ratio of 1:4 sata ssds. Some good nvmes are Intel P3600/P3700/P4500/P4600 series Samsung PM963.
Try to use nodes that have the same number of OSDs and have the OSDs of the same type and size, else you will get more load on specific nodes and/or disks which will create a performance bottleneck.
The more OSDs per host you have the better performance, untill your cpu cores saturate so it does not matter to add more. So it is better to have 16 x 4TB hdds rather than 8 x 8 TB hdds. With ssds, this applies to a limit since the cpu will saturate sooner ( typically 12 ssds or earlier, but depends on cores and disks).
If you have a mix of hdds and ssds, you can use ssds as journals to hdds and/or create 2 storage pools: 1 slow for hdds ( use for backups etc ) and a faster ssd pool. Do not mix hdds and ssds as osds in the same pool.