Sizing Metadata SSD
Ingo Hellmer
19 Posts
November 15, 2023, 7:01 amQuote from Ingo Hellmer on November 15, 2023, 7:01 amI am using PetaSAN as S3 Object Storage for Veeam Backup. Here it is important to have metadata on SSD. At the moment I am using 600GB (instead of 60GB default). I have read, that this size can only be adjusted in the next step to 6000 GB. Is this still true or is it posibble to use a size of 1.5 TB for example? This is important as the size of a HDD growths an on a 16TB Harddisk Veeam can create Millions of Blocks (1MB or 4MB sized).
I am using PetaSAN as S3 Object Storage for Veeam Backup. Here it is important to have metadata on SSD. At the moment I am using 600GB (instead of 60GB default). I have read, that this size can only be adjusted in the next step to 6000 GB. Is this still true or is it posibble to use a size of 1.5 TB for example? This is important as the size of a HDD growths an on a 16TB Harddisk Veeam can create Millions of Blocks (1MB or 4MB sized).
admin
2,930 Posts
November 15, 2023, 10:37 amQuote from admin on November 15, 2023, 10:37 amThe size goes up in levels using a factor of 10x. The rocksdb setting has layers at size 30 GB, 300 GB, 3000GB but it is recommended to allow additional space used during db compaction which occur periodically, to be safe the recommendation is to use 2x the layer size so 60,600, 6000 GBs. Technically to go over 600, you need at least 3000 + additional buffer for compaction.
First you need to consider if you really do need that much, test a couple of your most loaded OSDs with
ceph daemon osd.XX perf dump | grep db
and see what is actual bytes used
Typically you would worry about this if you are storing very small objects, like 4k so you have a much higher object count stored.
The size goes up in levels using a factor of 10x. The rocksdb setting has layers at size 30 GB, 300 GB, 3000GB but it is recommended to allow additional space used during db compaction which occur periodically, to be safe the recommendation is to use 2x the layer size so 60,600, 6000 GBs. Technically to go over 600, you need at least 3000 + additional buffer for compaction.
First you need to consider if you really do need that much, test a couple of your most loaded OSDs with
ceph daemon osd.XX perf dump | grep db
and see what is actual bytes used
Typically you would worry about this if you are storing very small objects, like 4k so you have a much higher object count stored.
Ingo Hellmer
19 Posts
November 16, 2023, 10:25 amQuote from Ingo Hellmer on November 16, 2023, 10:25 amThank you for the response.
On the first cluster with 12x 8TB HDD and 12x 960GB SSD I see this (nearly the same on all OSDs)
ceph daemon osd.0 perf dump | grep db
"db_total_bytes": 644245086208,
"db_used_bytes": 129098571776,
"max_bytes_db": 195634913280,
"rocksdb": {
"rocksdb_write_wal_time": {
"rocksdb_write_memtable_time": {
"rocksdb_write_delay_time": {
"rocksdb_write_pre_and_post_time": {
On the first cluster with 12x 12TB HDD and 12x 960GB SSD I see this (nearly the same on all OSDs)
ceph daemon osd.2 perf dump | grep db
"db_total_bytes": 644245086208,
"db_used_bytes": 180371841024,
"max_bytes_db": 181953093632,
"rocksdb": {
"rocksdb_write_wal_time": {
"rocksdb_write_memtable_time": {
"rocksdb_write_delay_time": {
"rocksdb_write_pre_and_post_time": {
So I am under 300GB on each system and I expect that this will be sufficient for 16TB HDD.
Both clusters are used as S3 Archive from Veeam Backup & Replication with a Backup Block Size of 4MB.
Thank you for the response.
On the first cluster with 12x 8TB HDD and 12x 960GB SSD I see this (nearly the same on all OSDs)
ceph daemon osd.0 perf dump | grep db
"db_total_bytes": 644245086208,
"db_used_bytes": 129098571776,
"max_bytes_db": 195634913280,
"rocksdb": {
"rocksdb_write_wal_time": {
"rocksdb_write_memtable_time": {
"rocksdb_write_delay_time": {
"rocksdb_write_pre_and_post_time": {
On the first cluster with 12x 12TB HDD and 12x 960GB SSD I see this (nearly the same on all OSDs)
ceph daemon osd.2 perf dump | grep db
"db_total_bytes": 644245086208,
"db_used_bytes": 180371841024,
"max_bytes_db": 181953093632,
"rocksdb": {
"rocksdb_write_wal_time": {
"rocksdb_write_memtable_time": {
"rocksdb_write_delay_time": {
"rocksdb_write_pre_and_post_time": {
So I am under 300GB on each system and I expect that this will be sufficient for 16TB HDD.
Both clusters are used as S3 Archive from Veeam Backup & Replication with a Backup Block Size of 4MB.
Sizing Metadata SSD
Ingo Hellmer
19 Posts
Quote from Ingo Hellmer on November 15, 2023, 7:01 amI am using PetaSAN as S3 Object Storage for Veeam Backup. Here it is important to have metadata on SSD. At the moment I am using 600GB (instead of 60GB default). I have read, that this size can only be adjusted in the next step to 6000 GB. Is this still true or is it posibble to use a size of 1.5 TB for example? This is important as the size of a HDD growths an on a 16TB Harddisk Veeam can create Millions of Blocks (1MB or 4MB sized).
I am using PetaSAN as S3 Object Storage for Veeam Backup. Here it is important to have metadata on SSD. At the moment I am using 600GB (instead of 60GB default). I have read, that this size can only be adjusted in the next step to 6000 GB. Is this still true or is it posibble to use a size of 1.5 TB for example? This is important as the size of a HDD growths an on a 16TB Harddisk Veeam can create Millions of Blocks (1MB or 4MB sized).
admin
2,930 Posts
Quote from admin on November 15, 2023, 10:37 amThe size goes up in levels using a factor of 10x. The rocksdb setting has layers at size 30 GB, 300 GB, 3000GB but it is recommended to allow additional space used during db compaction which occur periodically, to be safe the recommendation is to use 2x the layer size so 60,600, 6000 GBs. Technically to go over 600, you need at least 3000 + additional buffer for compaction.
First you need to consider if you really do need that much, test a couple of your most loaded OSDs with
ceph daemon osd.XX perf dump | grep db
and see what is actual bytes used
Typically you would worry about this if you are storing very small objects, like 4k so you have a much higher object count stored.
The size goes up in levels using a factor of 10x. The rocksdb setting has layers at size 30 GB, 300 GB, 3000GB but it is recommended to allow additional space used during db compaction which occur periodically, to be safe the recommendation is to use 2x the layer size so 60,600, 6000 GBs. Technically to go over 600, you need at least 3000 + additional buffer for compaction.
First you need to consider if you really do need that much, test a couple of your most loaded OSDs with
ceph daemon osd.XX perf dump | grep db
and see what is actual bytes used
Typically you would worry about this if you are storing very small objects, like 4k so you have a much higher object count stored.
Ingo Hellmer
19 Posts
Quote from Ingo Hellmer on November 16, 2023, 10:25 amThank you for the response.
On the first cluster with 12x 8TB HDD and 12x 960GB SSD I see this (nearly the same on all OSDs)
ceph daemon osd.0 perf dump | grep db
"db_total_bytes": 644245086208,
"db_used_bytes": 129098571776,
"max_bytes_db": 195634913280,
"rocksdb": {
"rocksdb_write_wal_time": {
"rocksdb_write_memtable_time": {
"rocksdb_write_delay_time": {
"rocksdb_write_pre_and_post_time": {
On the first cluster with 12x 12TB HDD and 12x 960GB SSD I see this (nearly the same on all OSDs)
ceph daemon osd.2 perf dump | grep db
"db_total_bytes": 644245086208,
"db_used_bytes": 180371841024,
"max_bytes_db": 181953093632,
"rocksdb": {
"rocksdb_write_wal_time": {
"rocksdb_write_memtable_time": {
"rocksdb_write_delay_time": {
"rocksdb_write_pre_and_post_time": {So I am under 300GB on each system and I expect that this will be sufficient for 16TB HDD.
Both clusters are used as S3 Archive from Veeam Backup & Replication with a Backup Block Size of 4MB.
Thank you for the response.
On the first cluster with 12x 8TB HDD and 12x 960GB SSD I see this (nearly the same on all OSDs)
ceph daemon osd.0 perf dump | grep db
"db_total_bytes": 644245086208,
"db_used_bytes": 129098571776,
"max_bytes_db": 195634913280,
"rocksdb": {
"rocksdb_write_wal_time": {
"rocksdb_write_memtable_time": {
"rocksdb_write_delay_time": {
"rocksdb_write_pre_and_post_time": {
On the first cluster with 12x 12TB HDD and 12x 960GB SSD I see this (nearly the same on all OSDs)
ceph daemon osd.2 perf dump | grep db
"db_total_bytes": 644245086208,
"db_used_bytes": 180371841024,
"max_bytes_db": 181953093632,
"rocksdb": {
"rocksdb_write_wal_time": {
"rocksdb_write_memtable_time": {
"rocksdb_write_delay_time": {
"rocksdb_write_pre_and_post_time": {
So I am under 300GB on each system and I expect that this will be sufficient for 16TB HDD.
Both clusters are used as S3 Archive from Veeam Backup & Replication with a Backup Block Size of 4MB.