Testing PetaSAN - Question on Cache
RobertH
27 Posts
September 10, 2020, 12:44 pmQuote from RobertH on September 10, 2020, 12:44 pmDoing testing on PetaSAN to see if it will work for us but have a question regarding the caching
Testing using an NVME drive as a cache for spinning 15k disks. Test configuration currently has 1x 256GB NVME servicing 8x spinning (I know recommendation is 1:4 but testing budget didn't allow) disks which allows for roughly 30GB cache partitions for each of the spinning drives. PetaSAN cluster consists of 6 nodes with same configuration using 4x 10gb NICs in each node. Cluster would be servicing 16 iSCSI Hyper-V hosts that each have 2x 10gb NICs.
Writes are landing on the cache drive as expected but not seeing activity on the spinning disks afterwards to indicate that the data is being moved from the cache back out to the spinning disks. After doing a load test on the system from the Hyper-V hosts the data is landing on the cache but doesn't seem to be getting flushed out to the spinning disks, then when doing read testing all of the reads are coming back from the NVME.
Initial impressions would be that the cache would handle any writes coming in since its faster, then flush out to the spinning disks in the background as resources allowed, and ultimately data would live on the spinning disks with reads only coming from the cache drive for data that hasn't been flushed to the spinning disks.
Documentation seems to be great in how to get up and running and a quick overview of all of the settings in the GUI, but doesn't cover any of the settings really in-depth or what can be done to tune anything
Questions:
With the caching how long is the data expected to stay on the cache disk before its moved out to the spinning disk?
Is the caching tune-able in any way?
If so is there any way to tune the caching so that data doesn't live on the cache drive for very long (ie 15-20 minutes tops)?
Doing testing on PetaSAN to see if it will work for us but have a question regarding the caching
Testing using an NVME drive as a cache for spinning 15k disks. Test configuration currently has 1x 256GB NVME servicing 8x spinning (I know recommendation is 1:4 but testing budget didn't allow) disks which allows for roughly 30GB cache partitions for each of the spinning drives. PetaSAN cluster consists of 6 nodes with same configuration using 4x 10gb NICs in each node. Cluster would be servicing 16 iSCSI Hyper-V hosts that each have 2x 10gb NICs.
Writes are landing on the cache drive as expected but not seeing activity on the spinning disks afterwards to indicate that the data is being moved from the cache back out to the spinning disks. After doing a load test on the system from the Hyper-V hosts the data is landing on the cache but doesn't seem to be getting flushed out to the spinning disks, then when doing read testing all of the reads are coming back from the NVME.
Initial impressions would be that the cache would handle any writes coming in since its faster, then flush out to the spinning disks in the background as resources allowed, and ultimately data would live on the spinning disks with reads only coming from the cache drive for data that hasn't been flushed to the spinning disks.
Documentation seems to be great in how to get up and running and a quick overview of all of the settings in the GUI, but doesn't cover any of the settings really in-depth or what can be done to tune anything
Questions:
With the caching how long is the data expected to stay on the cache disk before its moved out to the spinning disk?
Is the caching tune-able in any way?
If so is there any way to tune the caching so that data doesn't live on the cache drive for very long (ie 15-20 minutes tops)?
Last edited on September 10, 2020, 12:45 pm by RobertH · #1
admin
2,930 Posts
September 10, 2020, 2:00 pmQuote from admin on September 10, 2020, 2:00 pmIt is based on kernel dm-writecache. i works using a high and low watermarks, set at 45 and 50%. All writes land in cache, once cache fills to the high watermark flushing starts and stops when reaching the low watermark. Flushing uses LRU (least recently used) blocks and tries concatenate blocks to reduce hdd seeks.
it is possible to change the watermarks and other settings via dmsetup but we do not recommend it.
It is based on kernel dm-writecache. i works using a high and low watermarks, set at 45 and 50%. All writes land in cache, once cache fills to the high watermark flushing starts and stops when reaching the low watermark. Flushing uses LRU (least recently used) blocks and tries concatenate blocks to reduce hdd seeks.
it is possible to change the watermarks and other settings via dmsetup but we do not recommend it.
Testing PetaSAN - Question on Cache
RobertH
27 Posts
Quote from RobertH on September 10, 2020, 12:44 pmDoing testing on PetaSAN to see if it will work for us but have a question regarding the caching
Testing using an NVME drive as a cache for spinning 15k disks. Test configuration currently has 1x 256GB NVME servicing 8x spinning (I know recommendation is 1:4 but testing budget didn't allow) disks which allows for roughly 30GB cache partitions for each of the spinning drives. PetaSAN cluster consists of 6 nodes with same configuration using 4x 10gb NICs in each node. Cluster would be servicing 16 iSCSI Hyper-V hosts that each have 2x 10gb NICs.
Writes are landing on the cache drive as expected but not seeing activity on the spinning disks afterwards to indicate that the data is being moved from the cache back out to the spinning disks. After doing a load test on the system from the Hyper-V hosts the data is landing on the cache but doesn't seem to be getting flushed out to the spinning disks, then when doing read testing all of the reads are coming back from the NVME.
Initial impressions would be that the cache would handle any writes coming in since its faster, then flush out to the spinning disks in the background as resources allowed, and ultimately data would live on the spinning disks with reads only coming from the cache drive for data that hasn't been flushed to the spinning disks.
Documentation seems to be great in how to get up and running and a quick overview of all of the settings in the GUI, but doesn't cover any of the settings really in-depth or what can be done to tune anything
Questions:
With the caching how long is the data expected to stay on the cache disk before its moved out to the spinning disk?
Is the caching tune-able in any way?
If so is there any way to tune the caching so that data doesn't live on the cache drive for very long (ie 15-20 minutes tops)?
Doing testing on PetaSAN to see if it will work for us but have a question regarding the caching
Testing using an NVME drive as a cache for spinning 15k disks. Test configuration currently has 1x 256GB NVME servicing 8x spinning (I know recommendation is 1:4 but testing budget didn't allow) disks which allows for roughly 30GB cache partitions for each of the spinning drives. PetaSAN cluster consists of 6 nodes with same configuration using 4x 10gb NICs in each node. Cluster would be servicing 16 iSCSI Hyper-V hosts that each have 2x 10gb NICs.
Writes are landing on the cache drive as expected but not seeing activity on the spinning disks afterwards to indicate that the data is being moved from the cache back out to the spinning disks. After doing a load test on the system from the Hyper-V hosts the data is landing on the cache but doesn't seem to be getting flushed out to the spinning disks, then when doing read testing all of the reads are coming back from the NVME.
Initial impressions would be that the cache would handle any writes coming in since its faster, then flush out to the spinning disks in the background as resources allowed, and ultimately data would live on the spinning disks with reads only coming from the cache drive for data that hasn't been flushed to the spinning disks.
Documentation seems to be great in how to get up and running and a quick overview of all of the settings in the GUI, but doesn't cover any of the settings really in-depth or what can be done to tune anything
Questions:
With the caching how long is the data expected to stay on the cache disk before its moved out to the spinning disk?
Is the caching tune-able in any way?
If so is there any way to tune the caching so that data doesn't live on the cache drive for very long (ie 15-20 minutes tops)?
admin
2,930 Posts
Quote from admin on September 10, 2020, 2:00 pmIt is based on kernel dm-writecache. i works using a high and low watermarks, set at 45 and 50%. All writes land in cache, once cache fills to the high watermark flushing starts and stops when reaching the low watermark. Flushing uses LRU (least recently used) blocks and tries concatenate blocks to reduce hdd seeks.
it is possible to change the watermarks and other settings via dmsetup but we do not recommend it.
It is based on kernel dm-writecache. i works using a high and low watermarks, set at 45 and 50%. All writes land in cache, once cache fills to the high watermark flushing starts and stops when reaching the low watermark. Flushing uses LRU (least recently used) blocks and tries concatenate blocks to reduce hdd seeks.
it is possible to change the watermarks and other settings via dmsetup but we do not recommend it.