Running monitors under VMs
wailer
75 Posts
October 15, 2017, 1:54 amQuote from wailer on October 15, 2017, 1:54 amHi,
We are planning a production deployment , our plan is to start with this :
3 monitors using VM's (HA)
3 Physical OSD Nodes with 25 x 2'5 HDD slots and HP Gen8 Hardware
The backend network will be 10GbE
The iSCSI network will have 4x1Gbit Links
We have some doubts about it:
Is it a good idea to use monitors under VM's ? Will this affect performance ?
We have been reading about SSD and non-SSD disks ratio.. I see many people suggesting 1 SSD for journal per 6 non SSD disks at max.
Would it be feasable to run 1 SSD and 24 non SSD disks per node?
Thanks!
Hi,
We are planning a production deployment , our plan is to start with this :
3 monitors using VM's (HA)
3 Physical OSD Nodes with 25 x 2'5 HDD slots and HP Gen8 Hardware
The backend network will be 10GbE
The iSCSI network will have 4x1Gbit Links
We have some doubts about it:
Is it a good idea to use monitors under VM's ? Will this affect performance ?
We have been reading about SSD and non-SSD disks ratio.. I see many people suggesting 1 SSD for journal per 6 non SSD disks at max.
Would it be feasable to run 1 SSD and 24 non SSD disks per node?
Thanks!
admin
2,930 Posts
October 15, 2017, 11:38 amQuote from admin on October 15, 2017, 11:38 amIt should be fine to run the monitors as vms if they will only act as such ie no storage and no iscsi services.
They only get functioning during changes to the cluster: osd failures/additions/recovery..since they need to "publish" a cluster state for each change such where objects/pgs are stored and their status. So their performance will impact during these situations, but for a 25 osd cluster, the size of disls/pgs to track is relatively small.
For ssd/hdd mix: it is a good idea if you will use hdds but currently mixing ssd/hdd is not supported in PetaSAN ui, it is one of the features in 1.5. But in PetaSAN you can use pure ceph cli commands so you do it manually using ceph-disk command.
I would recommend 4 hdds per ssd, above this you will get worse performance. You can have several small capacity ssds instead of 1 larger capacity, you can use 128 or 64 G.
It should be fine to run the monitors as vms if they will only act as such ie no storage and no iscsi services.
They only get functioning during changes to the cluster: osd failures/additions/recovery..since they need to "publish" a cluster state for each change such where objects/pgs are stored and their status. So their performance will impact during these situations, but for a 25 osd cluster, the size of disls/pgs to track is relatively small.
For ssd/hdd mix: it is a good idea if you will use hdds but currently mixing ssd/hdd is not supported in PetaSAN ui, it is one of the features in 1.5. But in PetaSAN you can use pure ceph cli commands so you do it manually using ceph-disk command.
I would recommend 4 hdds per ssd, above this you will get worse performance. You can have several small capacity ssds instead of 1 larger capacity, you can use 128 or 64 G.
wailer
75 Posts
October 15, 2017, 11:00 pmQuote from wailer on October 15, 2017, 11:00 pmHi , Thanks for your inputs
So we would be talking about 3 storage Nodes x 20 HDD's each... that's 60 OSD's to monitor. with 5 x SSD on each node.
Is there any ETA for 1.5 version ?
Also , about bluestore release... would that SSD HDD ratio change? with the improvements it's supposed to bring ?
Thanks again!
Hi , Thanks for your inputs
So we would be talking about 3 storage Nodes x 20 HDD's each... that's 60 OSD's to monitor. with 5 x SSD on each node.
Is there any ETA for 1.5 version ?
Also , about bluestore release... would that SSD HDD ratio change? with the improvements it's supposed to bring ?
Thanks again!
Last edited on October 15, 2017, 11:32 pm by wailer · #3
admin
2,930 Posts
October 16, 2017, 10:37 amQuote from admin on October 16, 2017, 10:37 am5 SSDs will be a good ratio, each SSD will serve 4 HDD, the journal partition is 10 or 20 G max so 128 GB will be fine. It is better to have some free space in SSD as it will make them live longer.
We are working hard on 1.5 but i cannot say a release date yet.
Bluestore will also benefit from SSDs, but the impact of SSD journals on existing Filestore is more significant. The ratio 1:4 is the same. Bluestore does use journal/wal for small blocks but bypasses the journal for larger writes avoiding the 2 phase write/commit approach. Bluestore uses a rocksdb database to keep track of where objects are stored and provide a transactions to replace double write. Both wal and rocksdb will benefit from being stored on SSD, wal having the larger impact. Wal does not require much space, 1 G is more than enough since it is used for small sized blocks. How much rocksdb SSD space required is still debatable (still needs more performance tests) but a 1/100 of the HDD block device for storage is what seems recommended now.
5 SSDs will be a good ratio, each SSD will serve 4 HDD, the journal partition is 10 or 20 G max so 128 GB will be fine. It is better to have some free space in SSD as it will make them live longer.
We are working hard on 1.5 but i cannot say a release date yet.
Bluestore will also benefit from SSDs, but the impact of SSD journals on existing Filestore is more significant. The ratio 1:4 is the same. Bluestore does use journal/wal for small blocks but bypasses the journal for larger writes avoiding the 2 phase write/commit approach. Bluestore uses a rocksdb database to keep track of where objects are stored and provide a transactions to replace double write. Both wal and rocksdb will benefit from being stored on SSD, wal having the larger impact. Wal does not require much space, 1 G is more than enough since it is used for small sized blocks. How much rocksdb SSD space required is still debatable (still needs more performance tests) but a 1/100 of the HDD block device for storage is what seems recommended now.
Last edited on October 16, 2017, 10:37 am by admin · #4
Running monitors under VMs
wailer
75 Posts
Quote from wailer on October 15, 2017, 1:54 amHi,
We are planning a production deployment , our plan is to start with this :
3 monitors using VM's (HA)
3 Physical OSD Nodes with 25 x 2'5 HDD slots and HP Gen8 Hardware
The backend network will be 10GbE
The iSCSI network will have 4x1Gbit Links
We have some doubts about it:
Is it a good idea to use monitors under VM's ? Will this affect performance ?
We have been reading about SSD and non-SSD disks ratio.. I see many people suggesting 1 SSD for journal per 6 non SSD disks at max.
Would it be feasable to run 1 SSD and 24 non SSD disks per node?
Thanks!
Hi,
We are planning a production deployment , our plan is to start with this :
3 monitors using VM's (HA)
3 Physical OSD Nodes with 25 x 2'5 HDD slots and HP Gen8 Hardware
The backend network will be 10GbE
The iSCSI network will have 4x1Gbit Links
We have some doubts about it:
Is it a good idea to use monitors under VM's ? Will this affect performance ?
We have been reading about SSD and non-SSD disks ratio.. I see many people suggesting 1 SSD for journal per 6 non SSD disks at max.
Would it be feasable to run 1 SSD and 24 non SSD disks per node?
Thanks!
admin
2,930 Posts
Quote from admin on October 15, 2017, 11:38 amIt should be fine to run the monitors as vms if they will only act as such ie no storage and no iscsi services.
They only get functioning during changes to the cluster: osd failures/additions/recovery..since they need to "publish" a cluster state for each change such where objects/pgs are stored and their status. So their performance will impact during these situations, but for a 25 osd cluster, the size of disls/pgs to track is relatively small.
For ssd/hdd mix: it is a good idea if you will use hdds but currently mixing ssd/hdd is not supported in PetaSAN ui, it is one of the features in 1.5. But in PetaSAN you can use pure ceph cli commands so you do it manually using ceph-disk command.
I would recommend 4 hdds per ssd, above this you will get worse performance. You can have several small capacity ssds instead of 1 larger capacity, you can use 128 or 64 G.
It should be fine to run the monitors as vms if they will only act as such ie no storage and no iscsi services.
They only get functioning during changes to the cluster: osd failures/additions/recovery..since they need to "publish" a cluster state for each change such where objects/pgs are stored and their status. So their performance will impact during these situations, but for a 25 osd cluster, the size of disls/pgs to track is relatively small.
For ssd/hdd mix: it is a good idea if you will use hdds but currently mixing ssd/hdd is not supported in PetaSAN ui, it is one of the features in 1.5. But in PetaSAN you can use pure ceph cli commands so you do it manually using ceph-disk command.
I would recommend 4 hdds per ssd, above this you will get worse performance. You can have several small capacity ssds instead of 1 larger capacity, you can use 128 or 64 G.
wailer
75 Posts
Quote from wailer on October 15, 2017, 11:00 pmHi , Thanks for your inputs
So we would be talking about 3 storage Nodes x 20 HDD's each... that's 60 OSD's to monitor. with 5 x SSD on each node.
Is there any ETA for 1.5 version ?
Also , about bluestore release... would that SSD HDD ratio change? with the improvements it's supposed to bring ?
Thanks again!
Hi , Thanks for your inputs
So we would be talking about 3 storage Nodes x 20 HDD's each... that's 60 OSD's to monitor. with 5 x SSD on each node.
Is there any ETA for 1.5 version ?
Also , about bluestore release... would that SSD HDD ratio change? with the improvements it's supposed to bring ?
Thanks again!
admin
2,930 Posts
Quote from admin on October 16, 2017, 10:37 am5 SSDs will be a good ratio, each SSD will serve 4 HDD, the journal partition is 10 or 20 G max so 128 GB will be fine. It is better to have some free space in SSD as it will make them live longer.
We are working hard on 1.5 but i cannot say a release date yet.
Bluestore will also benefit from SSDs, but the impact of SSD journals on existing Filestore is more significant. The ratio 1:4 is the same. Bluestore does use journal/wal for small blocks but bypasses the journal for larger writes avoiding the 2 phase write/commit approach. Bluestore uses a rocksdb database to keep track of where objects are stored and provide a transactions to replace double write. Both wal and rocksdb will benefit from being stored on SSD, wal having the larger impact. Wal does not require much space, 1 G is more than enough since it is used for small sized blocks. How much rocksdb SSD space required is still debatable (still needs more performance tests) but a 1/100 of the HDD block device for storage is what seems recommended now.
5 SSDs will be a good ratio, each SSD will serve 4 HDD, the journal partition is 10 or 20 G max so 128 GB will be fine. It is better to have some free space in SSD as it will make them live longer.
We are working hard on 1.5 but i cannot say a release date yet.
Bluestore will also benefit from SSDs, but the impact of SSD journals on existing Filestore is more significant. The ratio 1:4 is the same. Bluestore does use journal/wal for small blocks but bypasses the journal for larger writes avoiding the 2 phase write/commit approach. Bluestore uses a rocksdb database to keep track of where objects are stored and provide a transactions to replace double write. Both wal and rocksdb will benefit from being stored on SSD, wal having the larger impact. Wal does not require much space, 1 G is more than enough since it is used for small sized blocks. How much rocksdb SSD space required is still debatable (still needs more performance tests) but a 1/100 of the HDD block device for storage is what seems recommended now.