CSVFS volumes slow while fast when not assigned as cluster volume
sschulze1984
3 Posts
November 15, 2022, 12:38 pmQuote from sschulze1984 on November 15, 2022, 12:38 pmHello everybody,
any ideas on why iSCSI LUN is quite fast when assigned as "standard" iSCSI directly to a Win Srv 2019 node while getting incredebily slow when assigned as CSVFS?
Setup:
- 5 nodes (3 mgm, 2 stor) each with 3x10T HDD and 1x1T SSD as cache
- 2x40G network between nodes
- 4x10G network between nodes and Windows server(s)
Benchmarks:
- IO write performance when iSCSI lun is "directly" attached as NTFS to a node: ~900 MByte/s
- IO write performance when iSCSI lun is used as CSVFS: ~50 MByte/s
Can't see any bottlenecks at CPU nor RAM nor network.
Any help or suggestions are strongly appreciated.
Hello everybody,
any ideas on why iSCSI LUN is quite fast when assigned as "standard" iSCSI directly to a Win Srv 2019 node while getting incredebily slow when assigned as CSVFS?
Setup:
- 5 nodes (3 mgm, 2 stor) each with 3x10T HDD and 1x1T SSD as cache
- 2x40G network between nodes
- 4x10G network between nodes and Windows server(s)
Benchmarks:
- IO write performance when iSCSI lun is "directly" attached as NTFS to a node: ~900 MByte/s
- IO write performance when iSCSI lun is used as CSVFS: ~50 MByte/s
Can't see any bottlenecks at CPU nor RAM nor network.
Any help or suggestions are strongly appreciated.
admin
2,930 Posts
November 15, 2022, 7:06 pmQuote from admin on November 15, 2022, 7:06 pmHave looked at our Hyper-v and/or SFOS and MPIO guides in setting up CSV ?
Are you using CSV on top of NTFS ?
How you test this ? Do you have CrystalDiskMark ? if so can you show irs results ?
Have looked at our Hyper-v and/or SFOS and MPIO guides in setting up CSV ?
Are you using CSV on top of NTFS ?
How you test this ? Do you have CrystalDiskMark ? if so can you show irs results ?
sschulze1984
3 Posts
November 15, 2022, 7:16 pmQuote from sschulze1984 on November 15, 2022, 7:16 pmHi,
1.) CrystalDiskMark (when running to "just" attached as iSCSI lun formatted with NTFS, not attached to cluster):
- Sequental 4M Q8T10 Mix 1105.47
- Sequental 4M Q1T10 Mix 1059,48
- Random 64K Q32T10 Mix 1122.59
- Random 64K Q1T10 Mix 419,90
2) Yes, we do - the iSCSI luns are attached to Hyper-V Failover Cluster; as we do for our "primary" PetaSAN cluster
3) Did several tests:
- Running CrystalDiskMark as described above
- Copying file(s) - bunch of small ones as well as few huge ones (>5G) - from node local storage to the new one as "one thread", copying files "multithreaded" (robocopy MT-option)
- Copying file(s) as above from "primary" PetaSAN cluster to the new one
- It makes no difference whether the tests are run on the node, which is currently in charge of the LUN or on another node - the performance difference is negligible (55 <-> 50 MB/s)
Hi,
1.) CrystalDiskMark (when running to "just" attached as iSCSI lun formatted with NTFS, not attached to cluster):
- Sequental 4M Q8T10 Mix 1105.47
- Sequental 4M Q1T10 Mix 1059,48
- Random 64K Q32T10 Mix 1122.59
- Random 64K Q1T10 Mix 419,90
2) Yes, we do - the iSCSI luns are attached to Hyper-V Failover Cluster; as we do for our "primary" PetaSAN cluster
3) Did several tests:
- Running CrystalDiskMark as described above
- Copying file(s) - bunch of small ones as well as few huge ones (>5G) - from node local storage to the new one as "one thread", copying files "multithreaded" (robocopy MT-option)
- Copying file(s) as above from "primary" PetaSAN cluster to the new one
- It makes no difference whether the tests are run on the node, which is currently in charge of the LUN or on another node - the performance difference is negligible (55 <-> 50 MB/s)
admin
2,930 Posts
November 15, 2022, 7:26 pmQuote from admin on November 15, 2022, 7:26 pmCan you show the CrystalDiskmark for the CSV case
Can you show the CrystalDiskmark for the CSV case
admin
2,930 Posts
November 15, 2022, 7:27 pmQuote from admin on November 15, 2022, 7:27 pmAlso if you can show 4k Q1T1 test for both cases
Also if you can show 4k Q1T1 test for both cases
Last edited on November 15, 2022, 7:28 pm by admin · #5
sschulze1984
3 Posts
November 16, 2022, 2:11 pmQuote from sschulze1984 on November 16, 2022, 2:11 pmNevermind - sorry for the (unnecessary) efforts ... it wasn't PetaSANs fault but the way CSVFS works (= long story short).
Nevermind - sorry for the (unnecessary) efforts ... it wasn't PetaSANs fault but the way CSVFS works (= long story short).
CSVFS volumes slow while fast when not assigned as cluster volume
sschulze1984
3 Posts
Quote from sschulze1984 on November 15, 2022, 12:38 pmHello everybody,
any ideas on why iSCSI LUN is quite fast when assigned as "standard" iSCSI directly to a Win Srv 2019 node while getting incredebily slow when assigned as CSVFS?
Setup:
- 5 nodes (3 mgm, 2 stor) each with 3x10T HDD and 1x1T SSD as cache
- 2x40G network between nodes
- 4x10G network between nodes and Windows server(s)
Benchmarks:
- IO write performance when iSCSI lun is "directly" attached as NTFS to a node: ~900 MByte/s
- IO write performance when iSCSI lun is used as CSVFS: ~50 MByte/s
Can't see any bottlenecks at CPU nor RAM nor network.
Any help or suggestions are strongly appreciated.
Hello everybody,
any ideas on why iSCSI LUN is quite fast when assigned as "standard" iSCSI directly to a Win Srv 2019 node while getting incredebily slow when assigned as CSVFS?
Setup:
- 5 nodes (3 mgm, 2 stor) each with 3x10T HDD and 1x1T SSD as cache
- 2x40G network between nodes
- 4x10G network between nodes and Windows server(s)
Benchmarks:
- IO write performance when iSCSI lun is "directly" attached as NTFS to a node: ~900 MByte/s
- IO write performance when iSCSI lun is used as CSVFS: ~50 MByte/s
Can't see any bottlenecks at CPU nor RAM nor network.
Any help or suggestions are strongly appreciated.
admin
2,930 Posts
Quote from admin on November 15, 2022, 7:06 pmHave looked at our Hyper-v and/or SFOS and MPIO guides in setting up CSV ?
Are you using CSV on top of NTFS ?
How you test this ? Do you have CrystalDiskMark ? if so can you show irs results ?
Have looked at our Hyper-v and/or SFOS and MPIO guides in setting up CSV ?
Are you using CSV on top of NTFS ?
How you test this ? Do you have CrystalDiskMark ? if so can you show irs results ?
sschulze1984
3 Posts
Quote from sschulze1984 on November 15, 2022, 7:16 pmHi,
1.) CrystalDiskMark (when running to "just" attached as iSCSI lun formatted with NTFS, not attached to cluster):
- Sequental 4M Q8T10 Mix 1105.47
- Sequental 4M Q1T10 Mix 1059,48
- Random 64K Q32T10 Mix 1122.59
- Random 64K Q1T10 Mix 419,90
2) Yes, we do - the iSCSI luns are attached to Hyper-V Failover Cluster; as we do for our "primary" PetaSAN cluster
3) Did several tests:
- Running CrystalDiskMark as described above
- Copying file(s) - bunch of small ones as well as few huge ones (>5G) - from node local storage to the new one as "one thread", copying files "multithreaded" (robocopy MT-option)
- Copying file(s) as above from "primary" PetaSAN cluster to the new one
- It makes no difference whether the tests are run on the node, which is currently in charge of the LUN or on another node - the performance difference is negligible (55 <-> 50 MB/s)
Hi,
1.) CrystalDiskMark (when running to "just" attached as iSCSI lun formatted with NTFS, not attached to cluster):
- Sequental 4M Q8T10 Mix 1105.47
- Sequental 4M Q1T10 Mix 1059,48
- Random 64K Q32T10 Mix 1122.59
- Random 64K Q1T10 Mix 419,90
2) Yes, we do - the iSCSI luns are attached to Hyper-V Failover Cluster; as we do for our "primary" PetaSAN cluster
3) Did several tests:
- Running CrystalDiskMark as described above
- Copying file(s) - bunch of small ones as well as few huge ones (>5G) - from node local storage to the new one as "one thread", copying files "multithreaded" (robocopy MT-option)
- Copying file(s) as above from "primary" PetaSAN cluster to the new one
- It makes no difference whether the tests are run on the node, which is currently in charge of the LUN or on another node - the performance difference is negligible (55 <-> 50 MB/s)
admin
2,930 Posts
Quote from admin on November 15, 2022, 7:26 pmCan you show the CrystalDiskmark for the CSV case
Can you show the CrystalDiskmark for the CSV case
admin
2,930 Posts
Quote from admin on November 15, 2022, 7:27 pmAlso if you can show 4k Q1T1 test for both cases
Also if you can show 4k Q1T1 test for both cases
sschulze1984
3 Posts
Quote from sschulze1984 on November 16, 2022, 2:11 pmNevermind - sorry for the (unnecessary) efforts ... it wasn't PetaSANs fault but the way CSVFS works (= long story short).
Nevermind - sorry for the (unnecessary) efforts ... it wasn't PetaSANs fault but the way CSVFS works (= long story short).