Journal not showing on OSD's
admin
2,930 Posts
January 31, 2021, 3:35 pmQuote from admin on January 31, 2021, 3:35 pmIf adding OSD without cache works, one possibility of cache failure is memory, you need 2.5% of ram for your cache partition size. Does this happen on some nodes only ?
If you add an OSD to a node and causes other OSDs to go down on/off, make sure the backfill speed is not high, you can set it to slow. if the hdds are busy the re-balance traffic can be too high so better lower the speed.
If adding OSD without cache works, one possibility of cache failure is memory, you need 2.5% of ram for your cache partition size. Does this happen on some nodes only ?
If you add an OSD to a node and causes other OSDs to go down on/off, make sure the backfill speed is not high, you can set it to slow. if the hdds are busy the re-balance traffic can be too high so better lower the speed.
neiltorda
98 Posts
January 31, 2021, 4:03 pmQuote from neiltorda on January 31, 2021, 4:03 pmWhat should by memory setting be set to?
These boxes are from 45drives, they are they hybrid models with 30 10T disks and currently 21 1T ssd.
When we first installed petasan, the boxes only had 64G of ram, so we only configured 16 OSD's per system and also lowered the RAM settings so we didn't run out of memory.
We have since installed more RAM (each system now has 128G)...
What should we set the memory to? It looks like currently the osd_memory_target is set to 1073741824
Thanks for the backfill tip, I lowered that from medium to slow to see if that stops those issues.
What should by memory setting be set to?
These boxes are from 45drives, they are they hybrid models with 30 10T disks and currently 21 1T ssd.
When we first installed petasan, the boxes only had 64G of ram, so we only configured 16 OSD's per system and also lowered the RAM settings so we didn't run out of memory.
We have since installed more RAM (each system now has 128G)...
What should we set the memory to? It looks like currently the osd_memory_target is set to 1073741824
Thanks for the backfill tip, I lowered that from medium to slow to see if that stops those issues.
admin
2,930 Posts
January 31, 2021, 4:48 pmQuote from admin on January 31, 2021, 4:48 pmYou should make sure you have 2.5% of cache partition size being used for ram. this is just for the write cache itself. the cache itself is persistent on disk but the ram is used internally during cache operation. Assuming you partition the SSD cache disk to 4 partitions, ie 250 GB per cache partition, then each cached OSD will require 6 GB of ram for cache operation.
The osd_memory_target is used as a volatile read cache by the OSD, the default is 4 GB per OSD rather than 1 GB. The more the better.
You should make sure you have 2.5% of cache partition size being used for ram. this is just for the write cache itself. the cache itself is persistent on disk but the ram is used internally during cache operation. Assuming you partition the SSD cache disk to 4 partitions, ie 250 GB per cache partition, then each cached OSD will require 6 GB of ram for cache operation.
The osd_memory_target is used as a volatile read cache by the OSD, the default is 4 GB per OSD rather than 1 GB. The more the better.
neiltorda
98 Posts
January 31, 2021, 5:24 pmQuote from neiltorda on January 31, 2021, 5:24 pmso, yes, most of the cache's are set to 4 per ssd… is there a way to change that to a higher number since I don't have any more ram to add to these systems?
so, yes, most of the cache's are set to 4 per ssd… is there a way to change that to a higher number since I don't have any more ram to add to these systems?
Journal not showing on OSD's
admin
2,930 Posts
Quote from admin on January 31, 2021, 3:35 pmIf adding OSD without cache works, one possibility of cache failure is memory, you need 2.5% of ram for your cache partition size. Does this happen on some nodes only ?
If you add an OSD to a node and causes other OSDs to go down on/off, make sure the backfill speed is not high, you can set it to slow. if the hdds are busy the re-balance traffic can be too high so better lower the speed.
If adding OSD without cache works, one possibility of cache failure is memory, you need 2.5% of ram for your cache partition size. Does this happen on some nodes only ?
If you add an OSD to a node and causes other OSDs to go down on/off, make sure the backfill speed is not high, you can set it to slow. if the hdds are busy the re-balance traffic can be too high so better lower the speed.
neiltorda
98 Posts
Quote from neiltorda on January 31, 2021, 4:03 pmWhat should by memory setting be set to?
These boxes are from 45drives, they are they hybrid models with 30 10T disks and currently 21 1T ssd.
When we first installed petasan, the boxes only had 64G of ram, so we only configured 16 OSD's per system and also lowered the RAM settings so we didn't run out of memory.
We have since installed more RAM (each system now has 128G)...
What should we set the memory to? It looks like currently the osd_memory_target is set to 1073741824
Thanks for the backfill tip, I lowered that from medium to slow to see if that stops those issues.
What should by memory setting be set to?
These boxes are from 45drives, they are they hybrid models with 30 10T disks and currently 21 1T ssd.
When we first installed petasan, the boxes only had 64G of ram, so we only configured 16 OSD's per system and also lowered the RAM settings so we didn't run out of memory.
We have since installed more RAM (each system now has 128G)...
What should we set the memory to? It looks like currently the osd_memory_target is set to 1073741824
Thanks for the backfill tip, I lowered that from medium to slow to see if that stops those issues.
admin
2,930 Posts
Quote from admin on January 31, 2021, 4:48 pmYou should make sure you have 2.5% of cache partition size being used for ram. this is just for the write cache itself. the cache itself is persistent on disk but the ram is used internally during cache operation. Assuming you partition the SSD cache disk to 4 partitions, ie 250 GB per cache partition, then each cached OSD will require 6 GB of ram for cache operation.
The osd_memory_target is used as a volatile read cache by the OSD, the default is 4 GB per OSD rather than 1 GB. The more the better.
You should make sure you have 2.5% of cache partition size being used for ram. this is just for the write cache itself. the cache itself is persistent on disk but the ram is used internally during cache operation. Assuming you partition the SSD cache disk to 4 partitions, ie 250 GB per cache partition, then each cached OSD will require 6 GB of ram for cache operation.
The osd_memory_target is used as a volatile read cache by the OSD, the default is 4 GB per OSD rather than 1 GB. The more the better.
neiltorda
98 Posts
Quote from neiltorda on January 31, 2021, 5:24 pmso, yes, most of the cache's are set to 4 per ssd… is there a way to change that to a higher number since I don't have any more ram to add to these systems?
so, yes, most of the cache's are set to 4 per ssd… is there a way to change that to a higher number since I don't have any more ram to add to these systems?