BBWC Guidance
BILDr
16 Posts
March 6, 2020, 12:05 pmQuote from BILDr on March 6, 2020, 12:05 pmI currently have HP P420i in HBA mode with 1GB BBWC. Would there be any significant performance benefit in switching the 1GB for 2GB memory?
The storage is 12 x 2TB SAS (x 3 nodes)
I currently have HP P420i in HBA mode with 1GB BBWC. Would there be any significant performance benefit in switching the 1GB for 2GB memory?
The storage is 12 x 2TB SAS (x 3 nodes)
ubx_cloud_steve
7 Posts
March 7, 2020, 5:47 amQuote from ubx_cloud_steve on March 7, 2020, 5:47 amI’m nearly certain that when you use the card in HBA mode the cache is completely bypassed and the drive is presented to the OS without any further abstraction.
if you want to improve small writes you can switch to raid mode on your controller and create a RAID0 for each OSD disk. That will allow each OSD to take advantage of the fast memory cache on the raid controller.
I’m nearly certain that when you use the card in HBA mode the cache is completely bypassed and the drive is presented to the OS without any further abstraction.
if you want to improve small writes you can switch to raid mode on your controller and create a RAID0 for each OSD disk. That will allow each OSD to take advantage of the fast memory cache on the raid controller.
therm
121 Posts
March 7, 2020, 7:45 amQuote from therm on March 7, 2020, 7:45 amThere has been a long discussion whether having raid underneath osds or not (most do recommend to not have raid). What I really would recommend you is to test your setup! We used P421 with RAID-0 and BBU and discovered that this one did not disable the disks own write cache(HGST). As a result everytime when I pulled power cables I lost OSDs!(you need to have Write-IO at this time) For our setup we switched to h221 sas hba to simple attach the disks via sas. Also we use filestore which will be better for short time write Load as they are fully journaled on our nvme devices, while bluestore will bypass at a certain io size.(Commit is faster on Filestore, while apply will be faster on bluestore because it bypasses some IO) This has impacts for other load types but that is what worked for our load characteristic.
There has been a long discussion whether having raid underneath osds or not (most do recommend to not have raid). What I really would recommend you is to test your setup! We used P421 with RAID-0 and BBU and discovered that this one did not disable the disks own write cache(HGST). As a result everytime when I pulled power cables I lost OSDs!(you need to have Write-IO at this time) For our setup we switched to h221 sas hba to simple attach the disks via sas. Also we use filestore which will be better for short time write Load as they are fully journaled on our nvme devices, while bluestore will bypass at a certain io size.(Commit is faster on Filestore, while apply will be faster on bluestore because it bypasses some IO) This has impacts for other load types but that is what worked for our load characteristic.
Last edited on March 7, 2020, 7:49 am by therm · #3
BILDr
16 Posts
March 7, 2020, 10:16 amQuote from BILDr on March 7, 2020, 10:16 amThanks for the feedback guys. I am stuck with what I have unfortunately.
The question now is would I be better with RAID-0 and onboard cache or straight HBA with no cache. The workload will be VM's. There's no NVMe journals either at this stage although the plan was to add some as workload and storage demands increase.
I am starting to thing it may be better with RAID-0 and standard install options to get me going. What are your thoughts?
It's always a juggling act between perfection and a starting point. I am thinking just get going, learn lessons, and deploy an optimised solution in further nodes.
Thanks for the feedback guys. I am stuck with what I have unfortunately.
The question now is would I be better with RAID-0 and onboard cache or straight HBA with no cache. The workload will be VM's. There's no NVMe journals either at this stage although the plan was to add some as workload and storage demands increase.
I am starting to thing it may be better with RAID-0 and standard install options to get me going. What are your thoughts?
It's always a juggling act between perfection and a starting point. I am thinking just get going, learn lessons, and deploy an optimised solution in further nodes.
ubx_cloud_steve
7 Posts
March 10, 2020, 6:16 amQuote from ubx_cloud_steve on March 10, 2020, 6:16 amRaid-0 all the way, run a benchmark before and after if you are curious.
we saw 2x-3x improvements with small writes when using raid-0 and caching, huge difference!
Raid-0 all the way, run a benchmark before and after if you are curious.
we saw 2x-3x improvements with small writes when using raid-0 and caching, huge difference!
ubx_cloud_steve
7 Posts
March 18, 2020, 6:12 amQuote from ubx_cloud_steve on March 18, 2020, 6:12 amHow did this work out for you?
How did this work out for you?
BILDr
16 Posts
March 18, 2020, 8:15 amQuote from BILDr on March 18, 2020, 8:15 amI ended up buying some H420 HBA's.
On consideration I decided the reliability in the event of losing an OSD with an HBA outweighed the speed penalty. I can always increase overall performance by adding M2 journals, additional storage nodes, offloading iSCSI to separate nodes or adding OSD's.
Those things are likely to happen anyway as the SAN matures. With HP kit replacing a failed disk should be fairly painless.
I ended up buying some H420 HBA's.
On consideration I decided the reliability in the event of losing an OSD with an HBA outweighed the speed penalty. I can always increase overall performance by adding M2 journals, additional storage nodes, offloading iSCSI to separate nodes or adding OSD's.
Those things are likely to happen anyway as the SAN matures. With HP kit replacing a failed disk should be fairly painless.
BBWC Guidance
BILDr
16 Posts
Quote from BILDr on March 6, 2020, 12:05 pmI currently have HP P420i in HBA mode with 1GB BBWC. Would there be any significant performance benefit in switching the 1GB for 2GB memory?
The storage is 12 x 2TB SAS (x 3 nodes)
I currently have HP P420i in HBA mode with 1GB BBWC. Would there be any significant performance benefit in switching the 1GB for 2GB memory?
The storage is 12 x 2TB SAS (x 3 nodes)
ubx_cloud_steve
7 Posts
Quote from ubx_cloud_steve on March 7, 2020, 5:47 amI’m nearly certain that when you use the card in HBA mode the cache is completely bypassed and the drive is presented to the OS without any further abstraction.
if you want to improve small writes you can switch to raid mode on your controller and create a RAID0 for each OSD disk. That will allow each OSD to take advantage of the fast memory cache on the raid controller.
I’m nearly certain that when you use the card in HBA mode the cache is completely bypassed and the drive is presented to the OS without any further abstraction.
if you want to improve small writes you can switch to raid mode on your controller and create a RAID0 for each OSD disk. That will allow each OSD to take advantage of the fast memory cache on the raid controller.
therm
121 Posts
Quote from therm on March 7, 2020, 7:45 amThere has been a long discussion whether having raid underneath osds or not (most do recommend to not have raid). What I really would recommend you is to test your setup! We used P421 with RAID-0 and BBU and discovered that this one did not disable the disks own write cache(HGST). As a result everytime when I pulled power cables I lost OSDs!(you need to have Write-IO at this time) For our setup we switched to h221 sas hba to simple attach the disks via sas. Also we use filestore which will be better for short time write Load as they are fully journaled on our nvme devices, while bluestore will bypass at a certain io size.(Commit is faster on Filestore, while apply will be faster on bluestore because it bypasses some IO) This has impacts for other load types but that is what worked for our load characteristic.
There has been a long discussion whether having raid underneath osds or not (most do recommend to not have raid). What I really would recommend you is to test your setup! We used P421 with RAID-0 and BBU and discovered that this one did not disable the disks own write cache(HGST). As a result everytime when I pulled power cables I lost OSDs!(you need to have Write-IO at this time) For our setup we switched to h221 sas hba to simple attach the disks via sas. Also we use filestore which will be better for short time write Load as they are fully journaled on our nvme devices, while bluestore will bypass at a certain io size.(Commit is faster on Filestore, while apply will be faster on bluestore because it bypasses some IO) This has impacts for other load types but that is what worked for our load characteristic.
BILDr
16 Posts
Quote from BILDr on March 7, 2020, 10:16 amThanks for the feedback guys. I am stuck with what I have unfortunately.
The question now is would I be better with RAID-0 and onboard cache or straight HBA with no cache. The workload will be VM's. There's no NVMe journals either at this stage although the plan was to add some as workload and storage demands increase.
I am starting to thing it may be better with RAID-0 and standard install options to get me going. What are your thoughts?
It's always a juggling act between perfection and a starting point. I am thinking just get going, learn lessons, and deploy an optimised solution in further nodes.
Thanks for the feedback guys. I am stuck with what I have unfortunately.
The question now is would I be better with RAID-0 and onboard cache or straight HBA with no cache. The workload will be VM's. There's no NVMe journals either at this stage although the plan was to add some as workload and storage demands increase.
I am starting to thing it may be better with RAID-0 and standard install options to get me going. What are your thoughts?
It's always a juggling act between perfection and a starting point. I am thinking just get going, learn lessons, and deploy an optimised solution in further nodes.
ubx_cloud_steve
7 Posts
Quote from ubx_cloud_steve on March 10, 2020, 6:16 amRaid-0 all the way, run a benchmark before and after if you are curious.
we saw 2x-3x improvements with small writes when using raid-0 and caching, huge difference!
Raid-0 all the way, run a benchmark before and after if you are curious.
we saw 2x-3x improvements with small writes when using raid-0 and caching, huge difference!
ubx_cloud_steve
7 Posts
Quote from ubx_cloud_steve on March 18, 2020, 6:12 amHow did this work out for you?
How did this work out for you?
BILDr
16 Posts
Quote from BILDr on March 18, 2020, 8:15 amI ended up buying some H420 HBA's.
On consideration I decided the reliability in the event of losing an OSD with an HBA outweighed the speed penalty. I can always increase overall performance by adding M2 journals, additional storage nodes, offloading iSCSI to separate nodes or adding OSD's.
Those things are likely to happen anyway as the SAN matures. With HP kit replacing a failed disk should be fairly painless.
I ended up buying some H420 HBA's.
On consideration I decided the reliability in the event of losing an OSD with an HBA outweighed the speed penalty. I can always increase overall performance by adding M2 journals, additional storage nodes, offloading iSCSI to separate nodes or adding OSD's.
Those things are likely to happen anyway as the SAN matures. With HP kit replacing a failed disk should be fairly painless.