iSCSI multi-client access issue, multi-client disk access fails when try validate initial configuration for a hyper-v cluster.
Pages: 1 2
JG
26 Posts
July 27, 2020, 1:16 pmQuote from JG on July 27, 2020, 1:16 pmHello all,
We are trying to configure a two node Hyper-V cluster and we are having the exact same error described here [1]
Implementation details:
Hardware (3 Nodes)
Dell R630 + (2) Xeon E5-2640 v3 + 128 GB RAM + (2) HDD 15k 320 GB RAID 1 + (2) SSD Enterprise (2 TB) for OSDs + (2) 1 Gb / s copper Ethernet ports + ( 4) SFP + 10Gb / s fiber
There are also 2 Dell S4112F-ON switches (linked through 1 QSFP 40 Gb/s port for VLT)
Two bond interfaces were created with the SFP + ports for the public network and for the PetaSAN iSCSI network. Jumbo frames have been enabled.
PetaSAN:
Version: 2.5
Linux Kernel: 4.12.14-28-petasan
All nodes are enabled as iSCSI Targets
In PetaSAN there are two iSCSI disks. LUN 00003 1 GB for "quorum" (3 paths) and LUN 00004 (250GB) (3 paths) "VM hdds"
* I know that the storage disk for VMs is small but if I create a multiple TB iSCSI disk in PetaSAN, formatting It takes forever.
HyperV-Nodes
(2) Nodes running Windows Server 2019 Datacenter
Windows Version: 1809
WindowsBuildLabEx: 17763.1.amd64fre.rs5_release.180914-1434
Issue:
Initial validation for cluster creation fails returning the following error:
Validate Disk Failover
Description: Validate that a disk can fail over successfully with data intact.
Start: 7/27/2020 4:35:14 AM.
Node HVTEST01.mydomain.local holds the SCSI PR on Test Disk 1 and brought the disk online, but failed in its attempt to write file data to partition table entry 1. The disk structure is corrupted and unreadable.
Node HVTEST01.mydomain.local holds the SCSI PR on Test Disk 1 and brought the disk online, but failed in its attempt to write file data to partition table entry 1. The disk structure is corrupted and unreadable.
Stop: 7/27/2020 4:35:18 AM.
In PetaSAN nodes the output of the dmesg command shows the following:
[Jul27 07:34] PR register with aptpl unset. Treating as aptpl=1
[ +0.008770] PR register with aptpl unset. Treating as aptpl=1
[ +0.034657] PR register with aptpl unset. Treating as aptpl=1
[ +0.002504] PR register with aptpl unset. Treating as aptpl=1
[ +0.173418] PR register with aptpl unset. Treating as aptpl=1
[ +0.006792] PR register with aptpl unset. Treating as aptpl=1
[ +0.013864] PR register with aptpl unset. Treating as aptpl=1
[ +0.009985] PR register with aptpl unset. Treating as aptpl=1
[ +0.002107] SPC-3 PR: Attempted RESERVE from iqn.1991-05.com.microsoft:hvtest01.mydomain.local,i,0x3430303030313337,iqn.2016-05.com.petasan:00004,t,0x2 while reservation already held by iqn.1991-05.com.microsoft:hvtest02.mydomain.local,i,0x3430303030313337,iqn.2016-05.com.petasan:00004,t,0x2, returning RESERVATION_CONFLICT
[ +0.033549] SPC-3 PR: Attempted RESERVE from iqn.1991-05.com.microsoft:hvtest01.mydomain.local,i,0x3430303030313337,iqn.2016-05.com.petasan:00003,t,0x3 while reservation already held by iqn.1991-05.com.microsoft:hvtest02.mydomain.local,i,0x3430303030313337,iqn.2016-05.com.petasan:00003,t,0x3, returning RESERVATION_CONFLICT
[ +0.021239] PR register with aptpl unset. Treating as aptpl=1
[ +0.005935] PR register with aptpl unset. Treating as aptpl=1
[ +0.005848] SPC-3 PR: Attempted RESERVE from iqn.1991-05.com.microsoft:hvtest02.mydomain.local,i,0x3430303030313337,iqn.2016-05.com.petasan:00004,t,0x2 while reservation already held by iqn.1991-05.com.microsoft:hvtest01.mydomain.local,i,0x3430303030313337,iqn.2016-05.com.petasan:00004,t,0x2, returning RESERVATION_CONFLICT
[ +0.027333] SPC-3 PR: Attempted RESERVE from iqn.1991-05.com.microsoft:hvtest02.mydomain.local,i,0x3430303030313337,iqn.2016-05.com.petasan:00003,t,0x3 while reservation already held by iqn.1991-05.com.microsoft:hvtest01.mydomain.local,i,0x3430303030313337,iqn.2016-05.com.petasan:00003,t,0x2, returning RESERVATION_CONFLICT
[ +0.027573] PR register with aptpl unset. Treating as aptpl=1
[ +0.006129] PR register with aptpl unset. Treating as aptpl=1
...
[ +6.024224] PR register with aptpl unset. Treating as aptpl=1
[ +0.006669] PR register with aptpl unset. Treating as aptpl=1
[ +0.001338] PR register with aptpl unset. Treating as aptpl=1
[ +0.003035] SPC-3 PR: Attempted RESERVE from iqn.1991-05.com.microsoft:hvtest02.rainbowdesign.net,i,0x3430303030313337,iqn.2016-05.com.petasan:00003,t,0x1 while reservation already held by iqn.1991-05.com.microsoft:hvtest01.rainbowdesign.net,i,0x3430303030313337,iqn.2016-05.com.petasan:00003,t,0x2, returning RESERVATION_CONFLICT
[ +6.024011] PR register with aptpl unset. Treating as aptpl=1
[ +0.009249] PR register with aptpl unset. Treating as aptpl=1
[ +0.001279] PR register with aptpl unset. Treating as aptpl=1
[ +6.027354] PR register with aptpl unset. Treating as aptpl=1
[ +0.007597] PR register with aptpl unset. Treating as aptpl=1
[ +0.009238] PR register with aptpl unset. Treating as aptpl=1
[ +2.207343] PR register with aptpl unset. Treating as aptpl=1
[ +0.007402] PR register with aptpl unset. Treating as aptpl=1
[ +0.000881] PR register with aptpl unset. Treating as aptpl=1
[ +1.103159] PR register with aptpl unset. Treating as aptpl=1
[ +0.007697] PR register with aptpl unset. Treating as aptpl=1
[ +1.500708] Unknown VPD Code: 0x89
[ +0.009900] Unknown VPD Code: 0x89
[ +0.266820] Unknown VPD Code: 0x89
[ +0.011765] Unknown VPD Code: 0x89
[ +0.343959] Unknown VPD Code: 0x89
[ +0.011601] Unknown VPD Code: 0x89
...
As mentioned here [1] we have run the powershell command Clear-ClusterDiskReservation, with no results.
We have decreased the number of Paths on iSCSI disks from 3 to 2, with no results.
Please let me know if I can provide any further info.
Any help will be highly appreciated.
Thank you.
Hello all,
We are trying to configure a two node Hyper-V cluster and we are having the exact same error described here [1]
Implementation details:
Hardware (3 Nodes)
Dell R630 + (2) Xeon E5-2640 v3 + 128 GB RAM + (2) HDD 15k 320 GB RAID 1 + (2) SSD Enterprise (2 TB) for OSDs + (2) 1 Gb / s copper Ethernet ports + ( 4) SFP + 10Gb / s fiber
There are also 2 Dell S4112F-ON switches (linked through 1 QSFP 40 Gb/s port for VLT)
Two bond interfaces were created with the SFP + ports for the public network and for the PetaSAN iSCSI network. Jumbo frames have been enabled.
PetaSAN:
Version: 2.5
Linux Kernel: 4.12.14-28-petasan
All nodes are enabled as iSCSI Targets
In PetaSAN there are two iSCSI disks. LUN 00003 1 GB for "quorum" (3 paths) and LUN 00004 (250GB) (3 paths) "VM hdds"
* I know that the storage disk for VMs is small but if I create a multiple TB iSCSI disk in PetaSAN, formatting It takes forever.
HyperV-Nodes
(2) Nodes running Windows Server 2019 Datacenter
Windows Version: 1809
WindowsBuildLabEx: 17763.1.amd64fre.rs5_release.180914-1434
Issue:
Initial validation for cluster creation fails returning the following error:
Validate Disk Failover
Description: Validate that a disk can fail over successfully with data intact.
Start: 7/27/2020 4:35:14 AM.
Node HVTEST01.mydomain.local holds the SCSI PR on Test Disk 1 and brought the disk online, but failed in its attempt to write file data to partition table entry 1. The disk structure is corrupted and unreadable.
Node HVTEST01.mydomain.local holds the SCSI PR on Test Disk 1 and brought the disk online, but failed in its attempt to write file data to partition table entry 1. The disk structure is corrupted and unreadable.
Stop: 7/27/2020 4:35:18 AM.
In PetaSAN nodes the output of the dmesg command shows the following:
[Jul27 07:34] PR register with aptpl unset. Treating as aptpl=1
[ +0.008770] PR register with aptpl unset. Treating as aptpl=1
[ +0.034657] PR register with aptpl unset. Treating as aptpl=1
[ +0.002504] PR register with aptpl unset. Treating as aptpl=1
[ +0.173418] PR register with aptpl unset. Treating as aptpl=1
[ +0.006792] PR register with aptpl unset. Treating as aptpl=1
[ +0.013864] PR register with aptpl unset. Treating as aptpl=1
[ +0.009985] PR register with aptpl unset. Treating as aptpl=1
[ +0.002107] SPC-3 PR: Attempted RESERVE from iqn.1991-05.com.microsoft:hvtest01.mydomain.local,i,0x3430303030313337,iqn.2016-05.com.petasan:00004,t,0x2 while reservation already held by iqn.1991-05.com.microsoft:hvtest02.mydomain.local,i,0x3430303030313337,iqn.2016-05.com.petasan:00004,t,0x2, returning RESERVATION_CONFLICT
[ +0.033549] SPC-3 PR: Attempted RESERVE from iqn.1991-05.com.microsoft:hvtest01.mydomain.local,i,0x3430303030313337,iqn.2016-05.com.petasan:00003,t,0x3 while reservation already held by iqn.1991-05.com.microsoft:hvtest02.mydomain.local,i,0x3430303030313337,iqn.2016-05.com.petasan:00003,t,0x3, returning RESERVATION_CONFLICT
[ +0.021239] PR register with aptpl unset. Treating as aptpl=1
[ +0.005935] PR register with aptpl unset. Treating as aptpl=1
[ +0.005848] SPC-3 PR: Attempted RESERVE from iqn.1991-05.com.microsoft:hvtest02.mydomain.local,i,0x3430303030313337,iqn.2016-05.com.petasan:00004,t,0x2 while reservation already held by iqn.1991-05.com.microsoft:hvtest01.mydomain.local,i,0x3430303030313337,iqn.2016-05.com.petasan:00004,t,0x2, returning RESERVATION_CONFLICT
[ +0.027333] SPC-3 PR: Attempted RESERVE from iqn.1991-05.com.microsoft:hvtest02.mydomain.local,i,0x3430303030313337,iqn.2016-05.com.petasan:00003,t,0x3 while reservation already held by iqn.1991-05.com.microsoft:hvtest01.mydomain.local,i,0x3430303030313337,iqn.2016-05.com.petasan:00003,t,0x2, returning RESERVATION_CONFLICT
[ +0.027573] PR register with aptpl unset. Treating as aptpl=1
[ +0.006129] PR register with aptpl unset. Treating as aptpl=1
...
[ +6.024224] PR register with aptpl unset. Treating as aptpl=1
[ +0.006669] PR register with aptpl unset. Treating as aptpl=1
[ +0.001338] PR register with aptpl unset. Treating as aptpl=1
[ +0.003035] SPC-3 PR: Attempted RESERVE from iqn.1991-05.com.microsoft:hvtest02.rainbowdesign.net,i,0x3430303030313337,iqn.2016-05.com.petasan:00003,t,0x1 while reservation already held by iqn.1991-05.com.microsoft:hvtest01.rainbowdesign.net,i,0x3430303030313337,iqn.2016-05.com.petasan:00003,t,0x2, returning RESERVATION_CONFLICT
[ +6.024011] PR register with aptpl unset. Treating as aptpl=1
[ +0.009249] PR register with aptpl unset. Treating as aptpl=1
[ +0.001279] PR register with aptpl unset. Treating as aptpl=1
[ +6.027354] PR register with aptpl unset. Treating as aptpl=1
[ +0.007597] PR register with aptpl unset. Treating as aptpl=1
[ +0.009238] PR register with aptpl unset. Treating as aptpl=1
[ +2.207343] PR register with aptpl unset. Treating as aptpl=1
[ +0.007402] PR register with aptpl unset. Treating as aptpl=1
[ +0.000881] PR register with aptpl unset. Treating as aptpl=1
[ +1.103159] PR register with aptpl unset. Treating as aptpl=1
[ +0.007697] PR register with aptpl unset. Treating as aptpl=1
[ +1.500708] Unknown VPD Code: 0x89
[ +0.009900] Unknown VPD Code: 0x89
[ +0.266820] Unknown VPD Code: 0x89
[ +0.011765] Unknown VPD Code: 0x89
[ +0.343959] Unknown VPD Code: 0x89
[ +0.011601] Unknown VPD Code: 0x89
...
As mentioned here [1] we have run the powershell command Clear-ClusterDiskReservation, with no results.
We have decreased the number of Paths on iSCSI disks from 3 to 2, with no results.
Please let me know if I can provide any further info.
Any help will be highly appreciated.
Thank you.
[1] https://www.petasan.org/forums/?view=thread&id=302
Last edited on July 27, 2020, 1:20 pm by JG · #1
admin
2,930 Posts
July 27, 2020, 1:40 pmQuote from admin on July 27, 2020, 1:40 pmThe Windows cluster validation test is something we always do, so it should be working.
I do not know, but the 1G quorum disk maybe too small. Can you try like 64G and see. Also if you say things are very slow then there could be connection issues.
The Windows cluster validation test is something we always do, so it should be working.
I do not know, but the 1G quorum disk maybe too small. Can you try like 64G and see. Also if you say things are very slow then there could be connection issues.
JG
26 Posts
July 27, 2020, 3:20 pmQuote from JG on July 27, 2020, 3:20 pmThanks for your answer.
Increasing the size of the quorum disk fixed the problem. This post could be considered closed.
I wonder if I should create a different thread about the slowness when formatting large iSCSI drives (I wonder what the time it takes for the OS to format the iSCSI drive depends on since there are no physical devices as such but the Formatting the iSCSI disk in PetaSAN happens at the "logical" level) and how long should it take for the OS to actually format an iSCSI drive.
Interestingly, CPU usage and network traffic on both PetaSAN nodes and Hyper-V servers is quite low when formatting large iSCSI drives.
The connectivity tests carried out do not show any problem, the Hyper-V nodes reach the maximum possible transfer in their 10 Gb / s interfaces with the PetaSAN servers as can be seen in the output of iperf3 below, from one of the Hyper-V nodes to one of PetaSAN nodes.
[ID] Interval Transfer Bandwidth
[5] 0.00-10.01 sec 0.00 Bytes 0.00 bits / sec sender
[5] 0.00-10.01 sec 3.08 GBytes 2.65 Gbits / sec receiver
[7] 0.00-10.01 sec 0.00 Bytes 0.00 bits / sec sender
[7] 0.00-10.01 sec 2.66 GBytes 2.29 Gbits / sec receiver
[9] 0.00-10.01 sec 0.00 Bytes 0.00 bits / sec sender
[9] 0.00-10.01 sec 3.01 GBytes 2.59 Gbits / sec receiver
[11] 0.00-10.01 sec 0.00 Bytes 0.00 bits / sec sender
[11] 0.00-10.01 sec 2.72 GBytes 2.33 Gbits / sec receiver
[SUM] 0.00-10.01 sec 0.00 Bytes 0.00 bits / sec sender
[SUM] 0.00-10.01 sec 11.5 GBytes 9.85 Gbits / sec receiver
Thanks for your answer.
Increasing the size of the quorum disk fixed the problem. This post could be considered closed.
I wonder if I should create a different thread about the slowness when formatting large iSCSI drives (I wonder what the time it takes for the OS to format the iSCSI drive depends on since there are no physical devices as such but the Formatting the iSCSI disk in PetaSAN happens at the "logical" level) and how long should it take for the OS to actually format an iSCSI drive.
Interestingly, CPU usage and network traffic on both PetaSAN nodes and Hyper-V servers is quite low when formatting large iSCSI drives.
The connectivity tests carried out do not show any problem, the Hyper-V nodes reach the maximum possible transfer in their 10 Gb / s interfaces with the PetaSAN servers as can be seen in the output of iperf3 below, from one of the Hyper-V nodes to one of PetaSAN nodes.
[ID] Interval Transfer Bandwidth
[5] 0.00-10.01 sec 0.00 Bytes 0.00 bits / sec sender
[5] 0.00-10.01 sec 3.08 GBytes 2.65 Gbits / sec receiver
[7] 0.00-10.01 sec 0.00 Bytes 0.00 bits / sec sender
[7] 0.00-10.01 sec 2.66 GBytes 2.29 Gbits / sec receiver
[9] 0.00-10.01 sec 0.00 Bytes 0.00 bits / sec sender
[9] 0.00-10.01 sec 3.01 GBytes 2.59 Gbits / sec receiver
[11] 0.00-10.01 sec 0.00 Bytes 0.00 bits / sec sender
[11] 0.00-10.01 sec 2.72 GBytes 2.33 Gbits / sec receiver
[SUM] 0.00-10.01 sec 0.00 Bytes 0.00 bits / sec sender
[SUM] 0.00-10.01 sec 11.5 GBytes 9.85 Gbits / sec receiver
admin
2,930 Posts
July 27, 2020, 3:52 pmQuote from admin on July 27, 2020, 3:52 pmit could be latency related rather than bandwidth.
Can you run the cluster benchmark from ui, choosing 1 client and 1 thread.
it could be latency related rather than bandwidth.
Can you run the cluster benchmark from ui, choosing 1 client and 1 thread.
JG
26 Posts
July 27, 2020, 4:12 pmQuote from JG on July 27, 2020, 4:12 pmThank you for answer. See benchmark test results below. Thank you.
Cluster IOPS
Write
Read
775
3325
Write Resource Load:
Node
Memory Util%
CPU Util%
Network Util%
Disks Util%
Actions
Avg
Max
Avg
Max
Journals
OSDs
Avg
Max
Avg
Max
ceph-node2
5
2
7
0
0
0
0
8
10
ceph-node3
6
2
10
0
0
0
0
9
11
Read Resource Load:
Node
Memory Util%
CPU Util%
Network Util%
Disks Util%
Actions
Avg
Max
Avg
Max
Journals
OSDs
Avg
Max
Avg
Max
ceph-node2
6
0
3
0
0
0
0
1
1
ceph-node3
6
0
3
0
0
0
0
1
1
Thank you for answer. See benchmark test results below. Thank you.
Cluster IOPS
Write
Read
775
3325
Write Resource Load:
Node
Memory Util%
CPU Util%
Network Util%
Disks Util%
Actions
Avg
Max
Avg
Max
Journals
OSDs
Avg
Max
Avg
Max
ceph-node2
5
2
7
0
0
0
0
8
10
ceph-node3
6
2
10
0
0
0
0
9
11
Read Resource Load:
Node
Memory Util%
CPU Util%
Network Util%
Disks Util%
Actions
Avg
Max
Avg
Max
Journals
OSDs
Avg
Max
Avg
Max
ceph-node2
6
0
3
0
0
0
0
1
1
ceph-node3
6
0
3
0
0
0
0
1
1
admin
2,930 Posts
July 27, 2020, 6:00 pmQuote from admin on July 27, 2020, 6:00 pmThe latency numbers are good: 0.3 ms read, 1.3 ms write. These are at the ceph rados level,
can you test at the Windows iSCSI client using Microsoft diskspd:
read iops
diskspd -b4k -d30 -o4 -t32 -r -Su -w0 -c10G Z:\iotest.dat
write iops
diskspd -b4k -d30 -o4 -t32 -r -Su -w100 -c10G Z:\iotest.dat
read latency
diskspd -b4k -d30 -o1 -t1 -r -Su -w0 -c10G Z:\iotest.dat
write latency
diskspd -b4k -d30 -o1 -t1 -r -Su -w100 -c10G Z:\iotest.dat
The latency numbers are good: 0.3 ms read, 1.3 ms write. These are at the ceph rados level,
can you test at the Windows iSCSI client using Microsoft diskspd:
read iops
diskspd -b4k -d30 -o4 -t32 -r -Su -w0 -c10G Z:\iotest.dat
write iops
diskspd -b4k -d30 -o4 -t32 -r -Su -w100 -c10G Z:\iotest.dat
read latency
diskspd -b4k -d30 -o1 -t1 -r -Su -w0 -c10G Z:\iotest.dat
write latency
diskspd -b4k -d30 -o1 -t1 -r -Su -w100 -c10G Z:\iotest.dat
Last edited on July 27, 2020, 6:01 pm by admin · #6
JG
26 Posts
July 27, 2020, 6:56 pmQuote from JG on July 27, 2020, 6:56 pmThis is output from the diskspd.exe utility. Thanks in advance for your time and all the help!
read iops
Command Line: diskspd.exe -b4k -d30 -o4 -t32 -r -Su -w0 -c10G C:\ClusterStorage\Volume1\iotest.dat
Input parameters:
timespan: 1
-------------
duration: 30s
warm up time: 5s
cool down time: 0s
random seed: 0
path: 'C:\ClusterStorage\Volume1\iotest.dat'
think time: 0ms
burst size: 0
software cache disabled
using hardware write cache, writethrough off
performing read test
block size: 4096
using random I/O (alignment: 4096)
number of outstanding I/O operations: 4
thread stride size: 0
threads per file: 32
using I/O Completion Ports
IO priority: normal
System information:
computer name: HVTEST01
start time: 2020/07/27 18:38:46 UTC
Results for timespan 1:
*******************************************************************************
actual test time: 30.00s
thread count: 32
proc count: 8
CPU | Usage | User | Kernel | Idle
-------------------------------------------
0| 79.11%| 3.33%| 75.78%| 20.89%
1| 58.18%| 3.33%| 54.84%| 41.82%
2| 88.07%| 1.51%| 86.56%| 11.93%
3| 59.79%| 4.06%| 55.73%| 40.21%
4| 96.46%| 1.25%| 95.21%| 3.54%
5| 59.79%| 2.97%| 56.82%| 40.21%
6| 55.57%| 3.65%| 51.93%| 44.43%
7| 54.01%| 3.65%| 50.36%| 45.99%
-------------------------------------------
avg.| 68.87%| 2.97%| 65.90%| 31.13%
Total IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 467173376 | 114056 | 14.85 | 3801.76 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
1 | 517079040 | 126240 | 16.44 | 4207.88 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
2 | 424247296 | 103576 | 13.49 | 3452.43 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
3 | 518680576 | 126631 | 16.49 | 4220.91 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
4 | 321974272 | 78607 | 10.23 | 2620.16 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
5 | 522252288 | 127503 | 16.60 | 4249.98 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
6 | 523501568 | 127808 | 16.64 | 4260.15 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
7 | 526270464 | 128484 | 16.73 | 4282.68 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
8 | 466718720 | 113945 | 14.84 | 3798.06 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
9 | 516403200 | 126075 | 16.42 | 4202.38 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
10 | 423788544 | 103464 | 13.47 | 3448.70 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
11 | 519172096 | 126751 | 16.50 | 4224.91 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
12 | 324067328 | 79118 | 10.30 | 2637.19 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
13 | 522407936 | 127541 | 16.61 | 4251.25 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
14 | 523976704 | 127924 | 16.66 | 4264.01 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
15 | 526610432 | 128567 | 16.74 | 4285.44 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
16 | 466575360 | 113910 | 14.83 | 3796.89 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
17 | 516907008 | 126198 | 16.43 | 4206.48 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
18 | 424329216 | 103596 | 13.49 | 3453.10 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
19 | 519110656 | 126736 | 16.50 | 4224.41 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
20 | 321810432 | 78567 | 10.23 | 2618.83 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
21 | 521105408 | 127223 | 16.57 | 4240.65 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
22 | 522428416 | 127546 | 16.61 | 4251.41 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
23 | 526249984 | 128479 | 16.73 | 4282.51 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
24 | 466419712 | 113872 | 14.83 | 3795.63 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
25 | 516202496 | 126026 | 16.41 | 4200.75 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
26 | 423526400 | 103400 | 13.46 | 3446.57 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
27 | 519311360 | 126785 | 16.51 | 4226.05 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
28 | 322256896 | 78676 | 10.24 | 2622.46 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
29 | 523022336 | 127691 | 16.63 | 4256.25 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
30 | 524419072 | 128032 | 16.67 | 4267.61 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
31 | 526954496 | 128651 | 16.75 | 4288.24 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 15284953088 | 3731678 | 485.88 | 124385.72
Read IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 467173376 | 114056 | 14.85 | 3801.76 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
1 | 517079040 | 126240 | 16.44 | 4207.88 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
2 | 424247296 | 103576 | 13.49 | 3452.43 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
3 | 518680576 | 126631 | 16.49 | 4220.91 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
4 | 321974272 | 78607 | 10.23 | 2620.16 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
5 | 522252288 | 127503 | 16.60 | 4249.98 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
6 | 523501568 | 127808 | 16.64 | 4260.15 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
7 | 526270464 | 128484 | 16.73 | 4282.68 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
8 | 466718720 | 113945 | 14.84 | 3798.06 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
9 | 516403200 | 126075 | 16.42 | 4202.38 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
10 | 423788544 | 103464 | 13.47 | 3448.70 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
11 | 519172096 | 126751 | 16.50 | 4224.91 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
12 | 324067328 | 79118 | 10.30 | 2637.19 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
13 | 522407936 | 127541 | 16.61 | 4251.25 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
14 | 523976704 | 127924 | 16.66 | 4264.01 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
15 | 526610432 | 128567 | 16.74 | 4285.44 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
16 | 466575360 | 113910 | 14.83 | 3796.89 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
17 | 516907008 | 126198 | 16.43 | 4206.48 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
18 | 424329216 | 103596 | 13.49 | 3453.10 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
19 | 519110656 | 126736 | 16.50 | 4224.41 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
20 | 321810432 | 78567 | 10.23 | 2618.83 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
21 | 521105408 | 127223 | 16.57 | 4240.65 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
22 | 522428416 | 127546 | 16.61 | 4251.41 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
23 | 526249984 | 128479 | 16.73 | 4282.51 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
24 | 466419712 | 113872 | 14.83 | 3795.63 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
25 | 516202496 | 126026 | 16.41 | 4200.75 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
26 | 423526400 | 103400 | 13.46 | 3446.57 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
27 | 519311360 | 126785 | 16.51 | 4226.05 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
28 | 322256896 | 78676 | 10.24 | 2622.46 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
29 | 523022336 | 127691 | 16.63 | 4256.25 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
30 | 524419072 | 128032 | 16.67 | 4267.61 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
31 | 526954496 | 128651 | 16.75 | 4288.24 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 15284953088 | 3731678 | 485.88 | 124385.72
Write IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
1 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
2 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
3 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
4 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
5 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
6 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
7 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
8 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
9 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
10 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
11 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
12 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
13 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
14 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
15 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
16 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
17 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
18 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
19 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
20 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
21 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
22 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
23 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
24 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
25 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
26 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
27 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
28 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
29 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
30 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
31 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 0 | 0 | 0.00 | 0.00
write iops
Command Line: diskspd.exe -b4k -d30 -o4 -t32 -r -Su -w100 -c10G C:\ClusterStorage\Volume1\iotest.dat
Input parameters:
timespan: 1
-------------
duration: 30s
warm up time: 5s
cool down time: 0s
random seed: 0
path: 'C:\ClusterStorage\Volume1\iotest.dat'
think time: 0ms
burst size: 0
software cache disabled
using hardware write cache, writethrough off
performing write test
block size: 4096
using random I/O (alignment: 4096)
number of outstanding I/O operations: 4
thread stride size: 0
threads per file: 32
using I/O Completion Ports
IO priority: normal
System information:
computer name: HVTEST01
start time: 2020/07/27 18:43:09 UTC
Results for timespan 1:
*******************************************************************************
actual test time: 30.00s
thread count: 32
proc count: 8
CPU | Usage | User | Kernel | Idle
-------------------------------------------
0| 11.46%| 1.77%| 9.69%| 88.54%
1| 9.32%| 1.41%| 7.92%| 90.68%
2| 13.02%| 1.56%| 11.46%| 86.98%
3| 8.39%| 1.15%| 7.24%| 91.61%
4| 13.75%| 0.26%| 13.49%| 86.25%
5| 7.08%| 0.42%| 6.67%| 92.92%
6| 6.20%| 0.36%| 5.83%| 93.80%
7| 6.35%| 0.47%| 5.89%| 93.65%
-------------------------------------------
avg.| 9.45%| 0.92%| 8.52%| 90.55%
Total IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 61890560 | 15110 | 1.97 | 503.66 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
1 | 61554688 | 15028 | 1.96 | 500.93 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
2 | 61644800 | 15050 | 1.96 | 501.66 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
3 | 61874176 | 15106 | 1.97 | 503.53 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
4 | 61566976 | 15031 | 1.96 | 501.03 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
5 | 62095360 | 15160 | 1.97 | 505.33 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
6 | 62324736 | 15216 | 1.98 | 507.20 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
7 | 62148608 | 15173 | 1.98 | 505.76 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
8 | 61898752 | 15112 | 1.97 | 503.73 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
9 | 61767680 | 15080 | 1.96 | 502.66 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
10 | 61870080 | 15105 | 1.97 | 503.50 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
11 | 62222336 | 15191 | 1.98 | 506.36 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
12 | 61886464 | 15109 | 1.97 | 503.63 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
13 | 62029824 | 15144 | 1.97 | 504.80 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
14 | 62164992 | 15177 | 1.98 | 505.90 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
15 | 62124032 | 15167 | 1.97 | 505.56 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
16 | 61771776 | 15081 | 1.96 | 502.70 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
17 | 62070784 | 15154 | 1.97 | 505.13 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
18 | 61718528 | 15068 | 1.96 | 502.26 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
19 | 61804544 | 15089 | 1.96 | 502.96 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
20 | 61603840 | 15040 | 1.96 | 501.33 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
21 | 62234624 | 15194 | 1.98 | 506.46 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
22 | 62201856 | 15186 | 1.98 | 506.20 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
23 | 62054400 | 15150 | 1.97 | 505.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
24 | 61616128 | 15043 | 1.96 | 501.43 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
25 | 61788160 | 15085 | 1.96 | 502.83 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
26 | 61464576 | 15006 | 1.95 | 500.20 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
27 | 61927424 | 15119 | 1.97 | 503.96 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
28 | 61399040 | 14990 | 1.95 | 499.66 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
29 | 61911040 | 15115 | 1.97 | 503.83 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
30 | 61943808 | 15123 | 1.97 | 504.10 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
31 | 62058496 | 15151 | 1.97 | 505.03 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 1980633088 | 483553 | 62.96 | 16118.34
Read IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
1 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
2 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
3 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
4 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
5 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
6 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
7 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
8 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
9 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
10 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
11 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
12 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
13 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
14 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
15 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
16 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
17 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
18 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
19 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
20 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
21 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
22 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
23 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
24 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
25 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
26 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
27 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
28 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
29 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
30 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
31 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 0 | 0 | 0.00 | 0.00
Write IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 61890560 | 15110 | 1.97 | 503.66 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
1 | 61554688 | 15028 | 1.96 | 500.93 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
2 | 61644800 | 15050 | 1.96 | 501.66 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
3 | 61874176 | 15106 | 1.97 | 503.53 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
4 | 61566976 | 15031 | 1.96 | 501.03 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
5 | 62095360 | 15160 | 1.97 | 505.33 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
6 | 62324736 | 15216 | 1.98 | 507.20 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
7 | 62148608 | 15173 | 1.98 | 505.76 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
8 | 61898752 | 15112 | 1.97 | 503.73 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
9 | 61767680 | 15080 | 1.96 | 502.66 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
10 | 61870080 | 15105 | 1.97 | 503.50 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
11 | 62222336 | 15191 | 1.98 | 506.36 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
12 | 61886464 | 15109 | 1.97 | 503.63 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
13 | 62029824 | 15144 | 1.97 | 504.80 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
14 | 62164992 | 15177 | 1.98 | 505.90 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
15 | 62124032 | 15167 | 1.97 | 505.56 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
16 | 61771776 | 15081 | 1.96 | 502.70 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
17 | 62070784 | 15154 | 1.97 | 505.13 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
18 | 61718528 | 15068 | 1.96 | 502.26 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
19 | 61804544 | 15089 | 1.96 | 502.96 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
20 | 61603840 | 15040 | 1.96 | 501.33 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
21 | 62234624 | 15194 | 1.98 | 506.46 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
22 | 62201856 | 15186 | 1.98 | 506.20 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
23 | 62054400 | 15150 | 1.97 | 505.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
24 | 61616128 | 15043 | 1.96 | 501.43 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
25 | 61788160 | 15085 | 1.96 | 502.83 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
26 | 61464576 | 15006 | 1.95 | 500.20 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
27 | 61927424 | 15119 | 1.97 | 503.96 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
28 | 61399040 | 14990 | 1.95 | 499.66 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
29 | 61911040 | 15115 | 1.97 | 503.83 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
30 | 61943808 | 15123 | 1.97 | 504.10 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
31 | 62058496 | 15151 | 1.97 | 505.03 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 1980633088 | 483553 | 62.96 | 16118.34
read latency
Command Line: diskspd.exe -b4k -d30 -o1 -t1 -r -Su -w0 -c10G C:\ClusterStorage\Volume1\iotest.dat
Input parameters:
timespan: 1
-------------
duration: 30s
warm up time: 5s
cool down time: 0s
random seed: 0
path: 'C:\ClusterStorage\Volume1\iotest.dat'
think time: 0ms
burst size: 0
software cache disabled
using hardware write cache, writethrough off
performing read test
block size: 4096
using random I/O (alignment: 4096)
number of outstanding I/O operations: 1
thread stride size: 0
threads per file: 1
IO priority: normal
System information:
computer name: HVTEST01
start time: 2020/07/27 18:45:05 UTC
Results for timespan 1:
*******************************************************************************
actual test time: 30.00s
thread count: 1
proc count: 8
CPU | Usage | User | Kernel | Idle
-------------------------------------------
0| 10.83%| 2.19%| 8.65%| 89.17%
1| 3.65%| 2.76%| 0.89%| 96.35%
2| 1.98%| 0.21%| 1.77%| 98.02%
3| 0.05%| 0.05%| 0.00%| 99.95%
4| 2.14%| 0.00%| 2.14%| 97.86%
5| 0.10%| 0.10%| 0.00%| 99.90%
6| 0.21%| 0.05%| 0.16%| 99.79%
7| 0.10%| 0.00%| 0.10%| 99.90%
-------------------------------------------
avg.| 2.38%| 0.67%| 1.71%| 97.62%
Total IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 254722048 | 62188 | 8.10 | 2072.89 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 254722048 | 62188 | 8.10 | 2072.89
Read IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 254722048 | 62188 | 8.10 | 2072.89 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 254722048 | 62188 | 8.10 | 2072.89
Write IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 0 | 0 | 0.00 | 0.00
write latency
Command Line: diskspd.exe -b4k -d30 -o1 -t1 -r -Su -w100 -c10G C:\ClusterStorage\Volume1\iotest.dat
Input parameters:
timespan: 1
-------------
duration: 30s
warm up time: 5s
cool down time: 0s
random seed: 0
path: 'C:\ClusterStorage\Volume1\iotest.dat'
think time: 0ms
burst size: 0
software cache disabled
using hardware write cache, writethrough off
performing write test
block size: 4096
using random I/O (alignment: 4096)
number of outstanding I/O operations: 1
thread stride size: 0
threads per file: 1
IO priority: normal
System information:
computer name: HVTEST01
start time: 2020/07/27 18:46:41 UTC
Results for timespan 1:
*******************************************************************************
actual test time: 30.00s
thread count: 1
proc count: 8
CPU | Usage | User | Kernel | Idle
-------------------------------------------
0| 6.61%| 2.03%| 4.58%| 93.39%
1| 3.49%| 2.29%| 1.20%| 96.51%
2| 0.26%| 0.16%| 0.10%| 99.74%
3| 0.05%| 0.00%| 0.05%| 99.95%
4| 0.47%| 0.16%| 0.31%| 99.53%
5| 0.10%| 0.10%| 0.00%| 99.90%
6| 0.05%| 0.00%| 0.05%| 99.95%
7| 0.00%| 0.00%| 0.00%| 100.00%
-------------------------------------------
avg.| 1.38%| 0.59%| 0.79%| 98.62%
Total IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 92803072 | 22657 | 2.95 | 755.22 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 92803072 | 22657 | 2.95 | 755.22
Read IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 0 | 0 | 0.00 | 0.00
Write IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 92803072 | 22657 | 2.95 | 755.22 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 92803072 | 22657 | 2.95 | 755.22
This is output from the diskspd.exe utility. Thanks in advance for your time and all the help!
read iops
Command Line: diskspd.exe -b4k -d30 -o4 -t32 -r -Su -w0 -c10G C:\ClusterStorage\Volume1\iotest.dat
Input parameters:
timespan: 1
-------------
duration: 30s
warm up time: 5s
cool down time: 0s
random seed: 0
path: 'C:\ClusterStorage\Volume1\iotest.dat'
think time: 0ms
burst size: 0
software cache disabled
using hardware write cache, writethrough off
performing read test
block size: 4096
using random I/O (alignment: 4096)
number of outstanding I/O operations: 4
thread stride size: 0
threads per file: 32
using I/O Completion Ports
IO priority: normal
System information:
computer name: HVTEST01
start time: 2020/07/27 18:38:46 UTC
Results for timespan 1:
*******************************************************************************
actual test time: 30.00s
thread count: 32
proc count: 8
CPU | Usage | User | Kernel | Idle
-------------------------------------------
0| 79.11%| 3.33%| 75.78%| 20.89%
1| 58.18%| 3.33%| 54.84%| 41.82%
2| 88.07%| 1.51%| 86.56%| 11.93%
3| 59.79%| 4.06%| 55.73%| 40.21%
4| 96.46%| 1.25%| 95.21%| 3.54%
5| 59.79%| 2.97%| 56.82%| 40.21%
6| 55.57%| 3.65%| 51.93%| 44.43%
7| 54.01%| 3.65%| 50.36%| 45.99%
-------------------------------------------
avg.| 68.87%| 2.97%| 65.90%| 31.13%
Total IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 467173376 | 114056 | 14.85 | 3801.76 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
1 | 517079040 | 126240 | 16.44 | 4207.88 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
2 | 424247296 | 103576 | 13.49 | 3452.43 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
3 | 518680576 | 126631 | 16.49 | 4220.91 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
4 | 321974272 | 78607 | 10.23 | 2620.16 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
5 | 522252288 | 127503 | 16.60 | 4249.98 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
6 | 523501568 | 127808 | 16.64 | 4260.15 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
7 | 526270464 | 128484 | 16.73 | 4282.68 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
8 | 466718720 | 113945 | 14.84 | 3798.06 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
9 | 516403200 | 126075 | 16.42 | 4202.38 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
10 | 423788544 | 103464 | 13.47 | 3448.70 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
11 | 519172096 | 126751 | 16.50 | 4224.91 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
12 | 324067328 | 79118 | 10.30 | 2637.19 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
13 | 522407936 | 127541 | 16.61 | 4251.25 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
14 | 523976704 | 127924 | 16.66 | 4264.01 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
15 | 526610432 | 128567 | 16.74 | 4285.44 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
16 | 466575360 | 113910 | 14.83 | 3796.89 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
17 | 516907008 | 126198 | 16.43 | 4206.48 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
18 | 424329216 | 103596 | 13.49 | 3453.10 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
19 | 519110656 | 126736 | 16.50 | 4224.41 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
20 | 321810432 | 78567 | 10.23 | 2618.83 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
21 | 521105408 | 127223 | 16.57 | 4240.65 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
22 | 522428416 | 127546 | 16.61 | 4251.41 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
23 | 526249984 | 128479 | 16.73 | 4282.51 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
24 | 466419712 | 113872 | 14.83 | 3795.63 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
25 | 516202496 | 126026 | 16.41 | 4200.75 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
26 | 423526400 | 103400 | 13.46 | 3446.57 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
27 | 519311360 | 126785 | 16.51 | 4226.05 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
28 | 322256896 | 78676 | 10.24 | 2622.46 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
29 | 523022336 | 127691 | 16.63 | 4256.25 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
30 | 524419072 | 128032 | 16.67 | 4267.61 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
31 | 526954496 | 128651 | 16.75 | 4288.24 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 15284953088 | 3731678 | 485.88 | 124385.72
Read IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 467173376 | 114056 | 14.85 | 3801.76 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
1 | 517079040 | 126240 | 16.44 | 4207.88 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
2 | 424247296 | 103576 | 13.49 | 3452.43 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
3 | 518680576 | 126631 | 16.49 | 4220.91 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
4 | 321974272 | 78607 | 10.23 | 2620.16 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
5 | 522252288 | 127503 | 16.60 | 4249.98 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
6 | 523501568 | 127808 | 16.64 | 4260.15 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
7 | 526270464 | 128484 | 16.73 | 4282.68 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
8 | 466718720 | 113945 | 14.84 | 3798.06 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
9 | 516403200 | 126075 | 16.42 | 4202.38 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
10 | 423788544 | 103464 | 13.47 | 3448.70 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
11 | 519172096 | 126751 | 16.50 | 4224.91 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
12 | 324067328 | 79118 | 10.30 | 2637.19 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
13 | 522407936 | 127541 | 16.61 | 4251.25 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
14 | 523976704 | 127924 | 16.66 | 4264.01 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
15 | 526610432 | 128567 | 16.74 | 4285.44 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
16 | 466575360 | 113910 | 14.83 | 3796.89 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
17 | 516907008 | 126198 | 16.43 | 4206.48 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
18 | 424329216 | 103596 | 13.49 | 3453.10 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
19 | 519110656 | 126736 | 16.50 | 4224.41 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
20 | 321810432 | 78567 | 10.23 | 2618.83 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
21 | 521105408 | 127223 | 16.57 | 4240.65 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
22 | 522428416 | 127546 | 16.61 | 4251.41 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
23 | 526249984 | 128479 | 16.73 | 4282.51 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
24 | 466419712 | 113872 | 14.83 | 3795.63 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
25 | 516202496 | 126026 | 16.41 | 4200.75 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
26 | 423526400 | 103400 | 13.46 | 3446.57 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
27 | 519311360 | 126785 | 16.51 | 4226.05 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
28 | 322256896 | 78676 | 10.24 | 2622.46 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
29 | 523022336 | 127691 | 16.63 | 4256.25 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
30 | 524419072 | 128032 | 16.67 | 4267.61 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
31 | 526954496 | 128651 | 16.75 | 4288.24 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 15284953088 | 3731678 | 485.88 | 124385.72
Write IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
1 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
2 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
3 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
4 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
5 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
6 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
7 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
8 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
9 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
10 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
11 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
12 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
13 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
14 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
15 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
16 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
17 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
18 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
19 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
20 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
21 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
22 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
23 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
24 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
25 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
26 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
27 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
28 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
29 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
30 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
31 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 0 | 0 | 0.00 | 0.00
write iops
Command Line: diskspd.exe -b4k -d30 -o4 -t32 -r -Su -w100 -c10G C:\ClusterStorage\Volume1\iotest.dat
Input parameters:
timespan: 1
-------------
duration: 30s
warm up time: 5s
cool down time: 0s
random seed: 0
path: 'C:\ClusterStorage\Volume1\iotest.dat'
think time: 0ms
burst size: 0
software cache disabled
using hardware write cache, writethrough off
performing write test
block size: 4096
using random I/O (alignment: 4096)
number of outstanding I/O operations: 4
thread stride size: 0
threads per file: 32
using I/O Completion Ports
IO priority: normal
System information:
computer name: HVTEST01
start time: 2020/07/27 18:43:09 UTC
Results for timespan 1:
*******************************************************************************
actual test time: 30.00s
thread count: 32
proc count: 8
CPU | Usage | User | Kernel | Idle
-------------------------------------------
0| 11.46%| 1.77%| 9.69%| 88.54%
1| 9.32%| 1.41%| 7.92%| 90.68%
2| 13.02%| 1.56%| 11.46%| 86.98%
3| 8.39%| 1.15%| 7.24%| 91.61%
4| 13.75%| 0.26%| 13.49%| 86.25%
5| 7.08%| 0.42%| 6.67%| 92.92%
6| 6.20%| 0.36%| 5.83%| 93.80%
7| 6.35%| 0.47%| 5.89%| 93.65%
-------------------------------------------
avg.| 9.45%| 0.92%| 8.52%| 90.55%
Total IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 61890560 | 15110 | 1.97 | 503.66 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
1 | 61554688 | 15028 | 1.96 | 500.93 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
2 | 61644800 | 15050 | 1.96 | 501.66 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
3 | 61874176 | 15106 | 1.97 | 503.53 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
4 | 61566976 | 15031 | 1.96 | 501.03 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
5 | 62095360 | 15160 | 1.97 | 505.33 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
6 | 62324736 | 15216 | 1.98 | 507.20 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
7 | 62148608 | 15173 | 1.98 | 505.76 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
8 | 61898752 | 15112 | 1.97 | 503.73 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
9 | 61767680 | 15080 | 1.96 | 502.66 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
10 | 61870080 | 15105 | 1.97 | 503.50 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
11 | 62222336 | 15191 | 1.98 | 506.36 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
12 | 61886464 | 15109 | 1.97 | 503.63 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
13 | 62029824 | 15144 | 1.97 | 504.80 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
14 | 62164992 | 15177 | 1.98 | 505.90 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
15 | 62124032 | 15167 | 1.97 | 505.56 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
16 | 61771776 | 15081 | 1.96 | 502.70 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
17 | 62070784 | 15154 | 1.97 | 505.13 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
18 | 61718528 | 15068 | 1.96 | 502.26 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
19 | 61804544 | 15089 | 1.96 | 502.96 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
20 | 61603840 | 15040 | 1.96 | 501.33 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
21 | 62234624 | 15194 | 1.98 | 506.46 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
22 | 62201856 | 15186 | 1.98 | 506.20 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
23 | 62054400 | 15150 | 1.97 | 505.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
24 | 61616128 | 15043 | 1.96 | 501.43 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
25 | 61788160 | 15085 | 1.96 | 502.83 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
26 | 61464576 | 15006 | 1.95 | 500.20 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
27 | 61927424 | 15119 | 1.97 | 503.96 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
28 | 61399040 | 14990 | 1.95 | 499.66 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
29 | 61911040 | 15115 | 1.97 | 503.83 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
30 | 61943808 | 15123 | 1.97 | 504.10 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
31 | 62058496 | 15151 | 1.97 | 505.03 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 1980633088 | 483553 | 62.96 | 16118.34
Read IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
1 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
2 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
3 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
4 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
5 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
6 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
7 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
8 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
9 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
10 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
11 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
12 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
13 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
14 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
15 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
16 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
17 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
18 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
19 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
20 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
21 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
22 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
23 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
24 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
25 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
26 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
27 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
28 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
29 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
30 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
31 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 0 | 0 | 0.00 | 0.00
Write IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 61890560 | 15110 | 1.97 | 503.66 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
1 | 61554688 | 15028 | 1.96 | 500.93 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
2 | 61644800 | 15050 | 1.96 | 501.66 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
3 | 61874176 | 15106 | 1.97 | 503.53 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
4 | 61566976 | 15031 | 1.96 | 501.03 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
5 | 62095360 | 15160 | 1.97 | 505.33 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
6 | 62324736 | 15216 | 1.98 | 507.20 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
7 | 62148608 | 15173 | 1.98 | 505.76 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
8 | 61898752 | 15112 | 1.97 | 503.73 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
9 | 61767680 | 15080 | 1.96 | 502.66 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
10 | 61870080 | 15105 | 1.97 | 503.50 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
11 | 62222336 | 15191 | 1.98 | 506.36 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
12 | 61886464 | 15109 | 1.97 | 503.63 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
13 | 62029824 | 15144 | 1.97 | 504.80 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
14 | 62164992 | 15177 | 1.98 | 505.90 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
15 | 62124032 | 15167 | 1.97 | 505.56 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
16 | 61771776 | 15081 | 1.96 | 502.70 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
17 | 62070784 | 15154 | 1.97 | 505.13 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
18 | 61718528 | 15068 | 1.96 | 502.26 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
19 | 61804544 | 15089 | 1.96 | 502.96 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
20 | 61603840 | 15040 | 1.96 | 501.33 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
21 | 62234624 | 15194 | 1.98 | 506.46 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
22 | 62201856 | 15186 | 1.98 | 506.20 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
23 | 62054400 | 15150 | 1.97 | 505.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
24 | 61616128 | 15043 | 1.96 | 501.43 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
25 | 61788160 | 15085 | 1.96 | 502.83 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
26 | 61464576 | 15006 | 1.95 | 500.20 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
27 | 61927424 | 15119 | 1.97 | 503.96 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
28 | 61399040 | 14990 | 1.95 | 499.66 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
29 | 61911040 | 15115 | 1.97 | 503.83 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
30 | 61943808 | 15123 | 1.97 | 504.10 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
31 | 62058496 | 15151 | 1.97 | 505.03 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 1980633088 | 483553 | 62.96 | 16118.34
read latency
Command Line: diskspd.exe -b4k -d30 -o1 -t1 -r -Su -w0 -c10G C:\ClusterStorage\Volume1\iotest.dat
Input parameters:
timespan: 1
-------------
duration: 30s
warm up time: 5s
cool down time: 0s
random seed: 0
path: 'C:\ClusterStorage\Volume1\iotest.dat'
think time: 0ms
burst size: 0
software cache disabled
using hardware write cache, writethrough off
performing read test
block size: 4096
using random I/O (alignment: 4096)
number of outstanding I/O operations: 1
thread stride size: 0
threads per file: 1
IO priority: normal
System information:
computer name: HVTEST01
start time: 2020/07/27 18:45:05 UTC
Results for timespan 1:
*******************************************************************************
actual test time: 30.00s
thread count: 1
proc count: 8
CPU | Usage | User | Kernel | Idle
-------------------------------------------
0| 10.83%| 2.19%| 8.65%| 89.17%
1| 3.65%| 2.76%| 0.89%| 96.35%
2| 1.98%| 0.21%| 1.77%| 98.02%
3| 0.05%| 0.05%| 0.00%| 99.95%
4| 2.14%| 0.00%| 2.14%| 97.86%
5| 0.10%| 0.10%| 0.00%| 99.90%
6| 0.21%| 0.05%| 0.16%| 99.79%
7| 0.10%| 0.00%| 0.10%| 99.90%
-------------------------------------------
avg.| 2.38%| 0.67%| 1.71%| 97.62%
Total IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 254722048 | 62188 | 8.10 | 2072.89 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 254722048 | 62188 | 8.10 | 2072.89
Read IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 254722048 | 62188 | 8.10 | 2072.89 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 254722048 | 62188 | 8.10 | 2072.89
Write IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 0 | 0 | 0.00 | 0.00
write latency
Command Line: diskspd.exe -b4k -d30 -o1 -t1 -r -Su -w100 -c10G C:\ClusterStorage\Volume1\iotest.dat
Input parameters:
timespan: 1
-------------
duration: 30s
warm up time: 5s
cool down time: 0s
random seed: 0
path: 'C:\ClusterStorage\Volume1\iotest.dat'
think time: 0ms
burst size: 0
software cache disabled
using hardware write cache, writethrough off
performing write test
block size: 4096
using random I/O (alignment: 4096)
number of outstanding I/O operations: 1
thread stride size: 0
threads per file: 1
IO priority: normal
System information:
computer name: HVTEST01
start time: 2020/07/27 18:46:41 UTC
Results for timespan 1:
*******************************************************************************
actual test time: 30.00s
thread count: 1
proc count: 8
CPU | Usage | User | Kernel | Idle
-------------------------------------------
0| 6.61%| 2.03%| 4.58%| 93.39%
1| 3.49%| 2.29%| 1.20%| 96.51%
2| 0.26%| 0.16%| 0.10%| 99.74%
3| 0.05%| 0.00%| 0.05%| 99.95%
4| 0.47%| 0.16%| 0.31%| 99.53%
5| 0.10%| 0.10%| 0.00%| 99.90%
6| 0.05%| 0.00%| 0.05%| 99.95%
7| 0.00%| 0.00%| 0.00%| 100.00%
-------------------------------------------
avg.| 1.38%| 0.59%| 0.79%| 98.62%
Total IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 92803072 | 22657 | 2.95 | 755.22 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 92803072 | 22657 | 2.95 | 755.22
Read IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 0 | 0 | 0.00 | 0.00
Write IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 92803072 | 22657 | 2.95 | 755.22 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 92803072 | 22657 | 2.95 | 755.22
admin
2,930 Posts
July 27, 2020, 9:16 pmQuote from admin on July 27, 2020, 9:16 pmThe iSCSI layer is fine, the latency is 0.5 ms read, 1.3 ms write. So if there is a issue with format speed, it is something else.
The iops test with your current setup was 124k read, 16k write. You may get higher write numbers by increasing thread count of the test from 32->128/256. To give further boost, look at the charts for cpu and disk % util ( better run the test for 3-5 min to get nice chart ), most probably your cpu will not be saturated due to the low count of SSDs (2) per node, which means you could add more disks per node and get linear performance increase.
Good luck 🙂
The iSCSI layer is fine, the latency is 0.5 ms read, 1.3 ms write. So if there is a issue with format speed, it is something else.
The iops test with your current setup was 124k read, 16k write. You may get higher write numbers by increasing thread count of the test from 32->128/256. To give further boost, look at the charts for cpu and disk % util ( better run the test for 3-5 min to get nice chart ), most probably your cpu will not be saturated due to the low count of SSDs (2) per node, which means you could add more disks per node and get linear performance increase.
Good luck 🙂
Last edited on July 27, 2020, 9:27 pm by admin · #8
JG
26 Posts
July 28, 2020, 7:19 pmQuote from JG on July 28, 2020, 7:19 pmThank you for your time and responses.
About the problem of the slow formatting of large drives, the problem is not CEPH | iSCSI | the hardware layer but in Windows (at least in Windows Server 2019 (1809) this is a problem and the solution that I leave below solves it)
The solution is "disable the SCSI TRIM and Unmap feature on the Host for the duration of the format" [1]
fsutil behavior set DisableDeleteNotify 1
To enable this feature once the formatting is complete:
fsutil behavior set DisableDeleteNotify 0
To verify the current configuration:
fsutil behavior query DisableDeleteNotify
DisableDeleteNotify = 0 - indicates the 'Trim and Unmap' feature is on (enabled)
DisableDeleteNotify = 1 - indicates the 'Trim and Unmap' feature is off (disabled)
This topic can be considered closed.
Thank you for your time and responses.
About the problem of the slow formatting of large drives, the problem is not CEPH | iSCSI | the hardware layer but in Windows (at least in Windows Server 2019 (1809) this is a problem and the solution that I leave below solves it)
The solution is "disable the SCSI TRIM and Unmap feature on the Host for the duration of the format" [1]
fsutil behavior set DisableDeleteNotify 1
To enable this feature once the formatting is complete:
fsutil behavior set DisableDeleteNotify 0
To verify the current configuration:
fsutil behavior query DisableDeleteNotify
DisableDeleteNotify = 0 - indicates the 'Trim and Unmap' feature is on (enabled)
DisableDeleteNotify = 1 - indicates the 'Trim and Unmap' feature is off (disabled)
This topic can be considered closed.
wid
47 Posts
December 4, 2020, 7:30 pmQuote from wid on December 4, 2020, 7:30 pmHi,
I have the same problem,
iSCSI lun, successfuly conneced (2 path) to windows 2019, format ReFS (5 GB, 10TB, any size) takes about 12-20 hours. Yes, hours.
I tried to switch TRIM (
fsutil behavior query DisableDeleteNotify
NTFS DisableDeleteNotify = 1 (Enabled)
ReFS DisableDeleteNotify = 1 (Enabled)
But no success. Is there any option to debug this case?
Lun is working fine when connected to ESX.
Hi,
I have the same problem,
iSCSI lun, successfuly conneced (2 path) to windows 2019, format ReFS (5 GB, 10TB, any size) takes about 12-20 hours. Yes, hours.
I tried to switch TRIM (
fsutil behavior query DisableDeleteNotify
NTFS DisableDeleteNotify = 1 (Enabled)
ReFS DisableDeleteNotify = 1 (Enabled)
But no success. Is there any option to debug this case?
Lun is working fine when connected to ESX.
Last edited on December 4, 2020, 7:31 pm by wid · #10
Pages: 1 2
iSCSI multi-client access issue, multi-client disk access fails when try validate initial configuration for a hyper-v cluster.
JG
26 Posts
Quote from JG on July 27, 2020, 1:16 pmHello all,
We are trying to configure a two node Hyper-V cluster and we are having the exact same error described here [1]
Implementation details:
Hardware (3 Nodes)
Dell R630 + (2) Xeon E5-2640 v3 + 128 GB RAM + (2) HDD 15k 320 GB RAID 1 + (2) SSD Enterprise (2 TB) for OSDs + (2) 1 Gb / s copper Ethernet ports + ( 4) SFP + 10Gb / s fiber
There are also 2 Dell S4112F-ON switches (linked through 1 QSFP 40 Gb/s port for VLT)
Two bond interfaces were created with the SFP + ports for the public network and for the PetaSAN iSCSI network. Jumbo frames have been enabled.PetaSAN:
Version: 2.5
Linux Kernel: 4.12.14-28-petasan
All nodes are enabled as iSCSI Targets
In PetaSAN there are two iSCSI disks. LUN 00003 1 GB for "quorum" (3 paths) and LUN 00004 (250GB) (3 paths) "VM hdds"
* I know that the storage disk for VMs is small but if I create a multiple TB iSCSI disk in PetaSAN, formatting It takes forever.HyperV-Nodes
(2) Nodes running Windows Server 2019 Datacenter
Windows Version: 1809
WindowsBuildLabEx: 17763.1.amd64fre.rs5_release.180914-1434Issue:
Initial validation for cluster creation fails returning the following error:
Validate Disk Failover
Description: Validate that a disk can fail over successfully with data intact.
Start: 7/27/2020 4:35:14 AM.
Node HVTEST01.mydomain.local holds the SCSI PR on Test Disk 1 and brought the disk online, but failed in its attempt to write file data to partition table entry 1. The disk structure is corrupted and unreadable.
Node HVTEST01.mydomain.local holds the SCSI PR on Test Disk 1 and brought the disk online, but failed in its attempt to write file data to partition table entry 1. The disk structure is corrupted and unreadable.
Stop: 7/27/2020 4:35:18 AM.In PetaSAN nodes the output of the dmesg command shows the following:
[Jul27 07:34] PR register with aptpl unset. Treating as aptpl=1
[ +0.008770] PR register with aptpl unset. Treating as aptpl=1
[ +0.034657] PR register with aptpl unset. Treating as aptpl=1
[ +0.002504] PR register with aptpl unset. Treating as aptpl=1
[ +0.173418] PR register with aptpl unset. Treating as aptpl=1
[ +0.006792] PR register with aptpl unset. Treating as aptpl=1
[ +0.013864] PR register with aptpl unset. Treating as aptpl=1
[ +0.009985] PR register with aptpl unset. Treating as aptpl=1
[ +0.002107] SPC-3 PR: Attempted RESERVE from iqn.1991-05.com.microsoft:hvtest01.mydomain.local,i,0x3430303030313337,iqn.2016-05.com.petasan:00004,t,0x2 while reservation already held by iqn.1991-05.com.microsoft:hvtest02.mydomain.local,i,0x3430303030313337,iqn.2016-05.com.petasan:00004,t,0x2, returning RESERVATION_CONFLICT
[ +0.033549] SPC-3 PR: Attempted RESERVE from iqn.1991-05.com.microsoft:hvtest01.mydomain.local,i,0x3430303030313337,iqn.2016-05.com.petasan:00003,t,0x3 while reservation already held by iqn.1991-05.com.microsoft:hvtest02.mydomain.local,i,0x3430303030313337,iqn.2016-05.com.petasan:00003,t,0x3, returning RESERVATION_CONFLICT
[ +0.021239] PR register with aptpl unset. Treating as aptpl=1
[ +0.005935] PR register with aptpl unset. Treating as aptpl=1
[ +0.005848] SPC-3 PR: Attempted RESERVE from iqn.1991-05.com.microsoft:hvtest02.mydomain.local,i,0x3430303030313337,iqn.2016-05.com.petasan:00004,t,0x2 while reservation already held by iqn.1991-05.com.microsoft:hvtest01.mydomain.local,i,0x3430303030313337,iqn.2016-05.com.petasan:00004,t,0x2, returning RESERVATION_CONFLICT
[ +0.027333] SPC-3 PR: Attempted RESERVE from iqn.1991-05.com.microsoft:hvtest02.mydomain.local,i,0x3430303030313337,iqn.2016-05.com.petasan:00003,t,0x3 while reservation already held by iqn.1991-05.com.microsoft:hvtest01.mydomain.local,i,0x3430303030313337,iqn.2016-05.com.petasan:00003,t,0x2, returning RESERVATION_CONFLICT
[ +0.027573] PR register with aptpl unset. Treating as aptpl=1
[ +0.006129] PR register with aptpl unset. Treating as aptpl=1
...
[ +6.024224] PR register with aptpl unset. Treating as aptpl=1
[ +0.006669] PR register with aptpl unset. Treating as aptpl=1
[ +0.001338] PR register with aptpl unset. Treating as aptpl=1
[ +0.003035] SPC-3 PR: Attempted RESERVE from iqn.1991-05.com.microsoft:hvtest02.rainbowdesign.net,i,0x3430303030313337,iqn.2016-05.com.petasan:00003,t,0x1 while reservation already held by iqn.1991-05.com.microsoft:hvtest01.rainbowdesign.net,i,0x3430303030313337,iqn.2016-05.com.petasan:00003,t,0x2, returning RESERVATION_CONFLICT
[ +6.024011] PR register with aptpl unset. Treating as aptpl=1
[ +0.009249] PR register with aptpl unset. Treating as aptpl=1
[ +0.001279] PR register with aptpl unset. Treating as aptpl=1
[ +6.027354] PR register with aptpl unset. Treating as aptpl=1
[ +0.007597] PR register with aptpl unset. Treating as aptpl=1
[ +0.009238] PR register with aptpl unset. Treating as aptpl=1
[ +2.207343] PR register with aptpl unset. Treating as aptpl=1
[ +0.007402] PR register with aptpl unset. Treating as aptpl=1
[ +0.000881] PR register with aptpl unset. Treating as aptpl=1
[ +1.103159] PR register with aptpl unset. Treating as aptpl=1
[ +0.007697] PR register with aptpl unset. Treating as aptpl=1
[ +1.500708] Unknown VPD Code: 0x89
[ +0.009900] Unknown VPD Code: 0x89
[ +0.266820] Unknown VPD Code: 0x89
[ +0.011765] Unknown VPD Code: 0x89
[ +0.343959] Unknown VPD Code: 0x89
[ +0.011601] Unknown VPD Code: 0x89
...As mentioned here [1] we have run the powershell command Clear-ClusterDiskReservation, with no results.
We have decreased the number of Paths on iSCSI disks from 3 to 2, with no results.Please let me know if I can provide any further info.
Any help will be highly appreciated.
Thank you.
Hello all,
We are trying to configure a two node Hyper-V cluster and we are having the exact same error described here [1]
Implementation details:
Hardware (3 Nodes)
Dell R630 + (2) Xeon E5-2640 v3 + 128 GB RAM + (2) HDD 15k 320 GB RAID 1 + (2) SSD Enterprise (2 TB) for OSDs + (2) 1 Gb / s copper Ethernet ports + ( 4) SFP + 10Gb / s fiber
There are also 2 Dell S4112F-ON switches (linked through 1 QSFP 40 Gb/s port for VLT)
Two bond interfaces were created with the SFP + ports for the public network and for the PetaSAN iSCSI network. Jumbo frames have been enabled.
PetaSAN:
Version: 2.5
Linux Kernel: 4.12.14-28-petasan
All nodes are enabled as iSCSI Targets
In PetaSAN there are two iSCSI disks. LUN 00003 1 GB for "quorum" (3 paths) and LUN 00004 (250GB) (3 paths) "VM hdds"
* I know that the storage disk for VMs is small but if I create a multiple TB iSCSI disk in PetaSAN, formatting It takes forever.
HyperV-Nodes
(2) Nodes running Windows Server 2019 Datacenter
Windows Version: 1809
WindowsBuildLabEx: 17763.1.amd64fre.rs5_release.180914-1434
Issue:
Initial validation for cluster creation fails returning the following error:
Validate Disk Failover
Description: Validate that a disk can fail over successfully with data intact.
Start: 7/27/2020 4:35:14 AM.
Node HVTEST01.mydomain.local holds the SCSI PR on Test Disk 1 and brought the disk online, but failed in its attempt to write file data to partition table entry 1. The disk structure is corrupted and unreadable.
Node HVTEST01.mydomain.local holds the SCSI PR on Test Disk 1 and brought the disk online, but failed in its attempt to write file data to partition table entry 1. The disk structure is corrupted and unreadable.
Stop: 7/27/2020 4:35:18 AM.
In PetaSAN nodes the output of the dmesg command shows the following:
[Jul27 07:34] PR register with aptpl unset. Treating as aptpl=1
[ +0.008770] PR register with aptpl unset. Treating as aptpl=1
[ +0.034657] PR register with aptpl unset. Treating as aptpl=1
[ +0.002504] PR register with aptpl unset. Treating as aptpl=1
[ +0.173418] PR register with aptpl unset. Treating as aptpl=1
[ +0.006792] PR register with aptpl unset. Treating as aptpl=1
[ +0.013864] PR register with aptpl unset. Treating as aptpl=1
[ +0.009985] PR register with aptpl unset. Treating as aptpl=1
[ +0.002107] SPC-3 PR: Attempted RESERVE from iqn.1991-05.com.microsoft:hvtest01.mydomain.local,i,0x3430303030313337,iqn.2016-05.com.petasan:00004,t,0x2 while reservation already held by iqn.1991-05.com.microsoft:hvtest02.mydomain.local,i,0x3430303030313337,iqn.2016-05.com.petasan:00004,t,0x2, returning RESERVATION_CONFLICT
[ +0.033549] SPC-3 PR: Attempted RESERVE from iqn.1991-05.com.microsoft:hvtest01.mydomain.local,i,0x3430303030313337,iqn.2016-05.com.petasan:00003,t,0x3 while reservation already held by iqn.1991-05.com.microsoft:hvtest02.mydomain.local,i,0x3430303030313337,iqn.2016-05.com.petasan:00003,t,0x3, returning RESERVATION_CONFLICT
[ +0.021239] PR register with aptpl unset. Treating as aptpl=1
[ +0.005935] PR register with aptpl unset. Treating as aptpl=1
[ +0.005848] SPC-3 PR: Attempted RESERVE from iqn.1991-05.com.microsoft:hvtest02.mydomain.local,i,0x3430303030313337,iqn.2016-05.com.petasan:00004,t,0x2 while reservation already held by iqn.1991-05.com.microsoft:hvtest01.mydomain.local,i,0x3430303030313337,iqn.2016-05.com.petasan:00004,t,0x2, returning RESERVATION_CONFLICT
[ +0.027333] SPC-3 PR: Attempted RESERVE from iqn.1991-05.com.microsoft:hvtest02.mydomain.local,i,0x3430303030313337,iqn.2016-05.com.petasan:00003,t,0x3 while reservation already held by iqn.1991-05.com.microsoft:hvtest01.mydomain.local,i,0x3430303030313337,iqn.2016-05.com.petasan:00003,t,0x2, returning RESERVATION_CONFLICT
[ +0.027573] PR register with aptpl unset. Treating as aptpl=1
[ +0.006129] PR register with aptpl unset. Treating as aptpl=1
...
[ +6.024224] PR register with aptpl unset. Treating as aptpl=1
[ +0.006669] PR register with aptpl unset. Treating as aptpl=1
[ +0.001338] PR register with aptpl unset. Treating as aptpl=1
[ +0.003035] SPC-3 PR: Attempted RESERVE from iqn.1991-05.com.microsoft:hvtest02.rainbowdesign.net,i,0x3430303030313337,iqn.2016-05.com.petasan:00003,t,0x1 while reservation already held by iqn.1991-05.com.microsoft:hvtest01.rainbowdesign.net,i,0x3430303030313337,iqn.2016-05.com.petasan:00003,t,0x2, returning RESERVATION_CONFLICT
[ +6.024011] PR register with aptpl unset. Treating as aptpl=1
[ +0.009249] PR register with aptpl unset. Treating as aptpl=1
[ +0.001279] PR register with aptpl unset. Treating as aptpl=1
[ +6.027354] PR register with aptpl unset. Treating as aptpl=1
[ +0.007597] PR register with aptpl unset. Treating as aptpl=1
[ +0.009238] PR register with aptpl unset. Treating as aptpl=1
[ +2.207343] PR register with aptpl unset. Treating as aptpl=1
[ +0.007402] PR register with aptpl unset. Treating as aptpl=1
[ +0.000881] PR register with aptpl unset. Treating as aptpl=1
[ +1.103159] PR register with aptpl unset. Treating as aptpl=1
[ +0.007697] PR register with aptpl unset. Treating as aptpl=1
[ +1.500708] Unknown VPD Code: 0x89
[ +0.009900] Unknown VPD Code: 0x89
[ +0.266820] Unknown VPD Code: 0x89
[ +0.011765] Unknown VPD Code: 0x89
[ +0.343959] Unknown VPD Code: 0x89
[ +0.011601] Unknown VPD Code: 0x89
...
As mentioned here [1] we have run the powershell command Clear-ClusterDiskReservation, with no results.
We have decreased the number of Paths on iSCSI disks from 3 to 2, with no results.
Please let me know if I can provide any further info.
Any help will be highly appreciated.
Thank you.
[1] https://www.petasan.org/forums/?view=thread&id=302
admin
2,930 Posts
Quote from admin on July 27, 2020, 1:40 pmThe Windows cluster validation test is something we always do, so it should be working.
I do not know, but the 1G quorum disk maybe too small. Can you try like 64G and see. Also if you say things are very slow then there could be connection issues.
The Windows cluster validation test is something we always do, so it should be working.
I do not know, but the 1G quorum disk maybe too small. Can you try like 64G and see. Also if you say things are very slow then there could be connection issues.
JG
26 Posts
Quote from JG on July 27, 2020, 3:20 pmThanks for your answer.
Increasing the size of the quorum disk fixed the problem. This post could be considered closed.
I wonder if I should create a different thread about the slowness when formatting large iSCSI drives (I wonder what the time it takes for the OS to format the iSCSI drive depends on since there are no physical devices as such but the Formatting the iSCSI disk in PetaSAN happens at the "logical" level) and how long should it take for the OS to actually format an iSCSI drive.
Interestingly, CPU usage and network traffic on both PetaSAN nodes and Hyper-V servers is quite low when formatting large iSCSI drives.
The connectivity tests carried out do not show any problem, the Hyper-V nodes reach the maximum possible transfer in their 10 Gb / s interfaces with the PetaSAN servers as can be seen in the output of iperf3 below, from one of the Hyper-V nodes to one of PetaSAN nodes.[ID] Interval Transfer Bandwidth
[5] 0.00-10.01 sec 0.00 Bytes 0.00 bits / sec sender
[5] 0.00-10.01 sec 3.08 GBytes 2.65 Gbits / sec receiver
[7] 0.00-10.01 sec 0.00 Bytes 0.00 bits / sec sender
[7] 0.00-10.01 sec 2.66 GBytes 2.29 Gbits / sec receiver
[9] 0.00-10.01 sec 0.00 Bytes 0.00 bits / sec sender
[9] 0.00-10.01 sec 3.01 GBytes 2.59 Gbits / sec receiver
[11] 0.00-10.01 sec 0.00 Bytes 0.00 bits / sec sender
[11] 0.00-10.01 sec 2.72 GBytes 2.33 Gbits / sec receiver
[SUM] 0.00-10.01 sec 0.00 Bytes 0.00 bits / sec sender
[SUM] 0.00-10.01 sec 11.5 GBytes 9.85 Gbits / sec receiver
Thanks for your answer.
Increasing the size of the quorum disk fixed the problem. This post could be considered closed.
I wonder if I should create a different thread about the slowness when formatting large iSCSI drives (I wonder what the time it takes for the OS to format the iSCSI drive depends on since there are no physical devices as such but the Formatting the iSCSI disk in PetaSAN happens at the "logical" level) and how long should it take for the OS to actually format an iSCSI drive.
Interestingly, CPU usage and network traffic on both PetaSAN nodes and Hyper-V servers is quite low when formatting large iSCSI drives.
The connectivity tests carried out do not show any problem, the Hyper-V nodes reach the maximum possible transfer in their 10 Gb / s interfaces with the PetaSAN servers as can be seen in the output of iperf3 below, from one of the Hyper-V nodes to one of PetaSAN nodes.
[ID] Interval Transfer Bandwidth
[5] 0.00-10.01 sec 0.00 Bytes 0.00 bits / sec sender
[5] 0.00-10.01 sec 3.08 GBytes 2.65 Gbits / sec receiver
[7] 0.00-10.01 sec 0.00 Bytes 0.00 bits / sec sender
[7] 0.00-10.01 sec 2.66 GBytes 2.29 Gbits / sec receiver
[9] 0.00-10.01 sec 0.00 Bytes 0.00 bits / sec sender
[9] 0.00-10.01 sec 3.01 GBytes 2.59 Gbits / sec receiver
[11] 0.00-10.01 sec 0.00 Bytes 0.00 bits / sec sender
[11] 0.00-10.01 sec 2.72 GBytes 2.33 Gbits / sec receiver
[SUM] 0.00-10.01 sec 0.00 Bytes 0.00 bits / sec sender
[SUM] 0.00-10.01 sec 11.5 GBytes 9.85 Gbits / sec receiver
admin
2,930 Posts
Quote from admin on July 27, 2020, 3:52 pmit could be latency related rather than bandwidth.
Can you run the cluster benchmark from ui, choosing 1 client and 1 thread.
it could be latency related rather than bandwidth.
Can you run the cluster benchmark from ui, choosing 1 client and 1 thread.
JG
26 Posts
Quote from JG on July 27, 2020, 4:12 pmThank you for answer. See benchmark test results below. Thank you.
Cluster IOPS
Write Read 775 3325 Write Resource Load:
Node Memory Util% CPU Util% Network Util% Disks Util% Actions Avg Max Avg Max Journals OSDs Avg Max Avg Max ceph-node2 5 2 7 0 0 0 0 8 10 ceph-node3 6 2 10 0 0 0 0 9 11 Read Resource Load:
Node Memory Util% CPU Util% Network Util% Disks Util% Actions Avg Max Avg Max Journals OSDs Avg Max Avg Max ceph-node2 6 0 3 0 0 0 0 1 1 ceph-node3 6 0 3 0 0 0 0 1 1
Thank you for answer. See benchmark test results below. Thank you.
Cluster IOPS
| Write | Read |
|---|---|
| 775 | 3325 |
Write Resource Load:
| Node | Memory Util% | CPU Util% | Network Util% | Disks Util% | Actions | |||||
|---|---|---|---|---|---|---|---|---|---|---|
| Avg | Max | Avg | Max | Journals | OSDs | |||||
| Avg | Max | Avg | Max | |||||||
| ceph-node2 | 5 | 2 | 7 | 0 | 0 | 0 | 0 | 8 | 10 | |
| ceph-node3 | 6 | 2 | 10 | 0 | 0 | 0 | 0 | 9 | 11 | |
Read Resource Load:
| Node | Memory Util% | CPU Util% | Network Util% | Disks Util% | Actions | |||||
|---|---|---|---|---|---|---|---|---|---|---|
| Avg | Max | Avg | Max | Journals | OSDs | |||||
| Avg | Max | Avg | Max | |||||||
| ceph-node2 | 6 | 0 | 3 | 0 | 0 | 0 | 0 | 1 | 1 | |
| ceph-node3 | 6 | 0 | 3 | 0 | 0 | 0 | 0 | 1 | 1 | |
admin
2,930 Posts
Quote from admin on July 27, 2020, 6:00 pmThe latency numbers are good: 0.3 ms read, 1.3 ms write. These are at the ceph rados level,
can you test at the Windows iSCSI client using Microsoft diskspd:
read iops
diskspd -b4k -d30 -o4 -t32 -r -Su -w0 -c10G Z:\iotest.datwrite iops
diskspd -b4k -d30 -o4 -t32 -r -Su -w100 -c10G Z:\iotest.datread latency
diskspd -b4k -d30 -o1 -t1 -r -Su -w0 -c10G Z:\iotest.datwrite latency
diskspd -b4k -d30 -o1 -t1 -r -Su -w100 -c10G Z:\iotest.dat
The latency numbers are good: 0.3 ms read, 1.3 ms write. These are at the ceph rados level,
can you test at the Windows iSCSI client using Microsoft diskspd:
read iops
diskspd -b4k -d30 -o4 -t32 -r -Su -w0 -c10G Z:\iotest.dat
write iops
diskspd -b4k -d30 -o4 -t32 -r -Su -w100 -c10G Z:\iotest.dat
read latency
diskspd -b4k -d30 -o1 -t1 -r -Su -w0 -c10G Z:\iotest.dat
write latency
diskspd -b4k -d30 -o1 -t1 -r -Su -w100 -c10G Z:\iotest.dat
JG
26 Posts
Quote from JG on July 27, 2020, 6:56 pmThis is output from the diskspd.exe utility. Thanks in advance for your time and all the help!
read iops
Command Line: diskspd.exe -b4k -d30 -o4 -t32 -r -Su -w0 -c10G C:\ClusterStorage\Volume1\iotest.dat
Input parameters:
timespan: 1
-------------
duration: 30s
warm up time: 5s
cool down time: 0s
random seed: 0
path: 'C:\ClusterStorage\Volume1\iotest.dat'
think time: 0ms
burst size: 0
software cache disabled
using hardware write cache, writethrough off
performing read test
block size: 4096
using random I/O (alignment: 4096)
number of outstanding I/O operations: 4
thread stride size: 0
threads per file: 32
using I/O Completion Ports
IO priority: normalSystem information:
computer name: HVTEST01
start time: 2020/07/27 18:38:46 UTCResults for timespan 1:
*******************************************************************************actual test time: 30.00s
thread count: 32
proc count: 8CPU | Usage | User | Kernel | Idle
-------------------------------------------
0| 79.11%| 3.33%| 75.78%| 20.89%
1| 58.18%| 3.33%| 54.84%| 41.82%
2| 88.07%| 1.51%| 86.56%| 11.93%
3| 59.79%| 4.06%| 55.73%| 40.21%
4| 96.46%| 1.25%| 95.21%| 3.54%
5| 59.79%| 2.97%| 56.82%| 40.21%
6| 55.57%| 3.65%| 51.93%| 44.43%
7| 54.01%| 3.65%| 50.36%| 45.99%
-------------------------------------------
avg.| 68.87%| 2.97%| 65.90%| 31.13%Total IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 467173376 | 114056 | 14.85 | 3801.76 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
1 | 517079040 | 126240 | 16.44 | 4207.88 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
2 | 424247296 | 103576 | 13.49 | 3452.43 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
3 | 518680576 | 126631 | 16.49 | 4220.91 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
4 | 321974272 | 78607 | 10.23 | 2620.16 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
5 | 522252288 | 127503 | 16.60 | 4249.98 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
6 | 523501568 | 127808 | 16.64 | 4260.15 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
7 | 526270464 | 128484 | 16.73 | 4282.68 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
8 | 466718720 | 113945 | 14.84 | 3798.06 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
9 | 516403200 | 126075 | 16.42 | 4202.38 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
10 | 423788544 | 103464 | 13.47 | 3448.70 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
11 | 519172096 | 126751 | 16.50 | 4224.91 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
12 | 324067328 | 79118 | 10.30 | 2637.19 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
13 | 522407936 | 127541 | 16.61 | 4251.25 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
14 | 523976704 | 127924 | 16.66 | 4264.01 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
15 | 526610432 | 128567 | 16.74 | 4285.44 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
16 | 466575360 | 113910 | 14.83 | 3796.89 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
17 | 516907008 | 126198 | 16.43 | 4206.48 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
18 | 424329216 | 103596 | 13.49 | 3453.10 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
19 | 519110656 | 126736 | 16.50 | 4224.41 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
20 | 321810432 | 78567 | 10.23 | 2618.83 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
21 | 521105408 | 127223 | 16.57 | 4240.65 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
22 | 522428416 | 127546 | 16.61 | 4251.41 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
23 | 526249984 | 128479 | 16.73 | 4282.51 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
24 | 466419712 | 113872 | 14.83 | 3795.63 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
25 | 516202496 | 126026 | 16.41 | 4200.75 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
26 | 423526400 | 103400 | 13.46 | 3446.57 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
27 | 519311360 | 126785 | 16.51 | 4226.05 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
28 | 322256896 | 78676 | 10.24 | 2622.46 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
29 | 523022336 | 127691 | 16.63 | 4256.25 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
30 | 524419072 | 128032 | 16.67 | 4267.61 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
31 | 526954496 | 128651 | 16.75 | 4288.24 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 15284953088 | 3731678 | 485.88 | 124385.72Read IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 467173376 | 114056 | 14.85 | 3801.76 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
1 | 517079040 | 126240 | 16.44 | 4207.88 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
2 | 424247296 | 103576 | 13.49 | 3452.43 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
3 | 518680576 | 126631 | 16.49 | 4220.91 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
4 | 321974272 | 78607 | 10.23 | 2620.16 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
5 | 522252288 | 127503 | 16.60 | 4249.98 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
6 | 523501568 | 127808 | 16.64 | 4260.15 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
7 | 526270464 | 128484 | 16.73 | 4282.68 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
8 | 466718720 | 113945 | 14.84 | 3798.06 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
9 | 516403200 | 126075 | 16.42 | 4202.38 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
10 | 423788544 | 103464 | 13.47 | 3448.70 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
11 | 519172096 | 126751 | 16.50 | 4224.91 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
12 | 324067328 | 79118 | 10.30 | 2637.19 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
13 | 522407936 | 127541 | 16.61 | 4251.25 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
14 | 523976704 | 127924 | 16.66 | 4264.01 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
15 | 526610432 | 128567 | 16.74 | 4285.44 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
16 | 466575360 | 113910 | 14.83 | 3796.89 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
17 | 516907008 | 126198 | 16.43 | 4206.48 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
18 | 424329216 | 103596 | 13.49 | 3453.10 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
19 | 519110656 | 126736 | 16.50 | 4224.41 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
20 | 321810432 | 78567 | 10.23 | 2618.83 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
21 | 521105408 | 127223 | 16.57 | 4240.65 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
22 | 522428416 | 127546 | 16.61 | 4251.41 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
23 | 526249984 | 128479 | 16.73 | 4282.51 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
24 | 466419712 | 113872 | 14.83 | 3795.63 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
25 | 516202496 | 126026 | 16.41 | 4200.75 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
26 | 423526400 | 103400 | 13.46 | 3446.57 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
27 | 519311360 | 126785 | 16.51 | 4226.05 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
28 | 322256896 | 78676 | 10.24 | 2622.46 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
29 | 523022336 | 127691 | 16.63 | 4256.25 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
30 | 524419072 | 128032 | 16.67 | 4267.61 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
31 | 526954496 | 128651 | 16.75 | 4288.24 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 15284953088 | 3731678 | 485.88 | 124385.72Write IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
1 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
2 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
3 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
4 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
5 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
6 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
7 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
8 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
9 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
10 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
11 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
12 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
13 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
14 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
15 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
16 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
17 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
18 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
19 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
20 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
21 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
22 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
23 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
24 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
25 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
26 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
27 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
28 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
29 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
30 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
31 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 0 | 0 | 0.00 | 0.00
write iops
Command Line: diskspd.exe -b4k -d30 -o4 -t32 -r -Su -w100 -c10G C:\ClusterStorage\Volume1\iotest.dat
Input parameters:
timespan: 1
-------------
duration: 30s
warm up time: 5s
cool down time: 0s
random seed: 0
path: 'C:\ClusterStorage\Volume1\iotest.dat'
think time: 0ms
burst size: 0
software cache disabled
using hardware write cache, writethrough off
performing write test
block size: 4096
using random I/O (alignment: 4096)
number of outstanding I/O operations: 4
thread stride size: 0
threads per file: 32
using I/O Completion Ports
IO priority: normalSystem information:
computer name: HVTEST01
start time: 2020/07/27 18:43:09 UTCResults for timespan 1:
*******************************************************************************actual test time: 30.00s
thread count: 32
proc count: 8CPU | Usage | User | Kernel | Idle
-------------------------------------------
0| 11.46%| 1.77%| 9.69%| 88.54%
1| 9.32%| 1.41%| 7.92%| 90.68%
2| 13.02%| 1.56%| 11.46%| 86.98%
3| 8.39%| 1.15%| 7.24%| 91.61%
4| 13.75%| 0.26%| 13.49%| 86.25%
5| 7.08%| 0.42%| 6.67%| 92.92%
6| 6.20%| 0.36%| 5.83%| 93.80%
7| 6.35%| 0.47%| 5.89%| 93.65%
-------------------------------------------
avg.| 9.45%| 0.92%| 8.52%| 90.55%Total IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 61890560 | 15110 | 1.97 | 503.66 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
1 | 61554688 | 15028 | 1.96 | 500.93 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
2 | 61644800 | 15050 | 1.96 | 501.66 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
3 | 61874176 | 15106 | 1.97 | 503.53 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
4 | 61566976 | 15031 | 1.96 | 501.03 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
5 | 62095360 | 15160 | 1.97 | 505.33 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
6 | 62324736 | 15216 | 1.98 | 507.20 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
7 | 62148608 | 15173 | 1.98 | 505.76 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
8 | 61898752 | 15112 | 1.97 | 503.73 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
9 | 61767680 | 15080 | 1.96 | 502.66 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
10 | 61870080 | 15105 | 1.97 | 503.50 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
11 | 62222336 | 15191 | 1.98 | 506.36 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
12 | 61886464 | 15109 | 1.97 | 503.63 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
13 | 62029824 | 15144 | 1.97 | 504.80 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
14 | 62164992 | 15177 | 1.98 | 505.90 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
15 | 62124032 | 15167 | 1.97 | 505.56 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
16 | 61771776 | 15081 | 1.96 | 502.70 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
17 | 62070784 | 15154 | 1.97 | 505.13 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
18 | 61718528 | 15068 | 1.96 | 502.26 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
19 | 61804544 | 15089 | 1.96 | 502.96 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
20 | 61603840 | 15040 | 1.96 | 501.33 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
21 | 62234624 | 15194 | 1.98 | 506.46 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
22 | 62201856 | 15186 | 1.98 | 506.20 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
23 | 62054400 | 15150 | 1.97 | 505.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
24 | 61616128 | 15043 | 1.96 | 501.43 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
25 | 61788160 | 15085 | 1.96 | 502.83 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
26 | 61464576 | 15006 | 1.95 | 500.20 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
27 | 61927424 | 15119 | 1.97 | 503.96 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
28 | 61399040 | 14990 | 1.95 | 499.66 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
29 | 61911040 | 15115 | 1.97 | 503.83 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
30 | 61943808 | 15123 | 1.97 | 504.10 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
31 | 62058496 | 15151 | 1.97 | 505.03 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 1980633088 | 483553 | 62.96 | 16118.34Read IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
1 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
2 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
3 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
4 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
5 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
6 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
7 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
8 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
9 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
10 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
11 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
12 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
13 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
14 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
15 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
16 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
17 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
18 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
19 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
20 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
21 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
22 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
23 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
24 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
25 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
26 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
27 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
28 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
29 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
30 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
31 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 0 | 0 | 0.00 | 0.00Write IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 61890560 | 15110 | 1.97 | 503.66 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
1 | 61554688 | 15028 | 1.96 | 500.93 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
2 | 61644800 | 15050 | 1.96 | 501.66 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
3 | 61874176 | 15106 | 1.97 | 503.53 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
4 | 61566976 | 15031 | 1.96 | 501.03 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
5 | 62095360 | 15160 | 1.97 | 505.33 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
6 | 62324736 | 15216 | 1.98 | 507.20 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
7 | 62148608 | 15173 | 1.98 | 505.76 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
8 | 61898752 | 15112 | 1.97 | 503.73 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
9 | 61767680 | 15080 | 1.96 | 502.66 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
10 | 61870080 | 15105 | 1.97 | 503.50 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
11 | 62222336 | 15191 | 1.98 | 506.36 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
12 | 61886464 | 15109 | 1.97 | 503.63 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
13 | 62029824 | 15144 | 1.97 | 504.80 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
14 | 62164992 | 15177 | 1.98 | 505.90 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
15 | 62124032 | 15167 | 1.97 | 505.56 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
16 | 61771776 | 15081 | 1.96 | 502.70 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
17 | 62070784 | 15154 | 1.97 | 505.13 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
18 | 61718528 | 15068 | 1.96 | 502.26 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
19 | 61804544 | 15089 | 1.96 | 502.96 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
20 | 61603840 | 15040 | 1.96 | 501.33 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
21 | 62234624 | 15194 | 1.98 | 506.46 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
22 | 62201856 | 15186 | 1.98 | 506.20 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
23 | 62054400 | 15150 | 1.97 | 505.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
24 | 61616128 | 15043 | 1.96 | 501.43 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
25 | 61788160 | 15085 | 1.96 | 502.83 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
26 | 61464576 | 15006 | 1.95 | 500.20 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
27 | 61927424 | 15119 | 1.97 | 503.96 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
28 | 61399040 | 14990 | 1.95 | 499.66 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
29 | 61911040 | 15115 | 1.97 | 503.83 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
30 | 61943808 | 15123 | 1.97 | 504.10 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
31 | 62058496 | 15151 | 1.97 | 505.03 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 1980633088 | 483553 | 62.96 | 16118.34
read latency
Command Line: diskspd.exe -b4k -d30 -o1 -t1 -r -Su -w0 -c10G C:\ClusterStorage\Volume1\iotest.dat
Input parameters:
timespan: 1
-------------
duration: 30s
warm up time: 5s
cool down time: 0s
random seed: 0
path: 'C:\ClusterStorage\Volume1\iotest.dat'
think time: 0ms
burst size: 0
software cache disabled
using hardware write cache, writethrough off
performing read test
block size: 4096
using random I/O (alignment: 4096)
number of outstanding I/O operations: 1
thread stride size: 0
threads per file: 1
IO priority: normalSystem information:
computer name: HVTEST01
start time: 2020/07/27 18:45:05 UTCResults for timespan 1:
*******************************************************************************actual test time: 30.00s
thread count: 1
proc count: 8CPU | Usage | User | Kernel | Idle
-------------------------------------------
0| 10.83%| 2.19%| 8.65%| 89.17%
1| 3.65%| 2.76%| 0.89%| 96.35%
2| 1.98%| 0.21%| 1.77%| 98.02%
3| 0.05%| 0.05%| 0.00%| 99.95%
4| 2.14%| 0.00%| 2.14%| 97.86%
5| 0.10%| 0.10%| 0.00%| 99.90%
6| 0.21%| 0.05%| 0.16%| 99.79%
7| 0.10%| 0.00%| 0.10%| 99.90%
-------------------------------------------
avg.| 2.38%| 0.67%| 1.71%| 97.62%Total IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 254722048 | 62188 | 8.10 | 2072.89 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 254722048 | 62188 | 8.10 | 2072.89Read IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 254722048 | 62188 | 8.10 | 2072.89 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 254722048 | 62188 | 8.10 | 2072.89Write IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 0 | 0 | 0.00 | 0.00
write latency
Command Line: diskspd.exe -b4k -d30 -o1 -t1 -r -Su -w100 -c10G C:\ClusterStorage\Volume1\iotest.dat
Input parameters:
timespan: 1
-------------
duration: 30s
warm up time: 5s
cool down time: 0s
random seed: 0
path: 'C:\ClusterStorage\Volume1\iotest.dat'
think time: 0ms
burst size: 0
software cache disabled
using hardware write cache, writethrough off
performing write test
block size: 4096
using random I/O (alignment: 4096)
number of outstanding I/O operations: 1
thread stride size: 0
threads per file: 1
IO priority: normalSystem information:
computer name: HVTEST01
start time: 2020/07/27 18:46:41 UTCResults for timespan 1:
*******************************************************************************actual test time: 30.00s
thread count: 1
proc count: 8CPU | Usage | User | Kernel | Idle
-------------------------------------------
0| 6.61%| 2.03%| 4.58%| 93.39%
1| 3.49%| 2.29%| 1.20%| 96.51%
2| 0.26%| 0.16%| 0.10%| 99.74%
3| 0.05%| 0.00%| 0.05%| 99.95%
4| 0.47%| 0.16%| 0.31%| 99.53%
5| 0.10%| 0.10%| 0.00%| 99.90%
6| 0.05%| 0.00%| 0.05%| 99.95%
7| 0.00%| 0.00%| 0.00%| 100.00%
-------------------------------------------
avg.| 1.38%| 0.59%| 0.79%| 98.62%Total IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 92803072 | 22657 | 2.95 | 755.22 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 92803072 | 22657 | 2.95 | 755.22Read IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 0 | 0 | 0.00 | 0.00Write IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 92803072 | 22657 | 2.95 | 755.22 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 92803072 | 22657 | 2.95 | 755.22
This is output from the diskspd.exe utility. Thanks in advance for your time and all the help!
read iops
Command Line: diskspd.exe -b4k -d30 -o4 -t32 -r -Su -w0 -c10G C:\ClusterStorage\Volume1\iotest.dat
Input parameters:
timespan: 1
-------------
duration: 30s
warm up time: 5s
cool down time: 0s
random seed: 0
path: 'C:\ClusterStorage\Volume1\iotest.dat'
think time: 0ms
burst size: 0
software cache disabled
using hardware write cache, writethrough off
performing read test
block size: 4096
using random I/O (alignment: 4096)
number of outstanding I/O operations: 4
thread stride size: 0
threads per file: 32
using I/O Completion Ports
IO priority: normal
System information:
computer name: HVTEST01
start time: 2020/07/27 18:38:46 UTC
Results for timespan 1:
*******************************************************************************
actual test time: 30.00s
thread count: 32
proc count: 8
CPU | Usage | User | Kernel | Idle
-------------------------------------------
0| 79.11%| 3.33%| 75.78%| 20.89%
1| 58.18%| 3.33%| 54.84%| 41.82%
2| 88.07%| 1.51%| 86.56%| 11.93%
3| 59.79%| 4.06%| 55.73%| 40.21%
4| 96.46%| 1.25%| 95.21%| 3.54%
5| 59.79%| 2.97%| 56.82%| 40.21%
6| 55.57%| 3.65%| 51.93%| 44.43%
7| 54.01%| 3.65%| 50.36%| 45.99%
-------------------------------------------
avg.| 68.87%| 2.97%| 65.90%| 31.13%
Total IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 467173376 | 114056 | 14.85 | 3801.76 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
1 | 517079040 | 126240 | 16.44 | 4207.88 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
2 | 424247296 | 103576 | 13.49 | 3452.43 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
3 | 518680576 | 126631 | 16.49 | 4220.91 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
4 | 321974272 | 78607 | 10.23 | 2620.16 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
5 | 522252288 | 127503 | 16.60 | 4249.98 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
6 | 523501568 | 127808 | 16.64 | 4260.15 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
7 | 526270464 | 128484 | 16.73 | 4282.68 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
8 | 466718720 | 113945 | 14.84 | 3798.06 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
9 | 516403200 | 126075 | 16.42 | 4202.38 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
10 | 423788544 | 103464 | 13.47 | 3448.70 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
11 | 519172096 | 126751 | 16.50 | 4224.91 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
12 | 324067328 | 79118 | 10.30 | 2637.19 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
13 | 522407936 | 127541 | 16.61 | 4251.25 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
14 | 523976704 | 127924 | 16.66 | 4264.01 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
15 | 526610432 | 128567 | 16.74 | 4285.44 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
16 | 466575360 | 113910 | 14.83 | 3796.89 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
17 | 516907008 | 126198 | 16.43 | 4206.48 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
18 | 424329216 | 103596 | 13.49 | 3453.10 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
19 | 519110656 | 126736 | 16.50 | 4224.41 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
20 | 321810432 | 78567 | 10.23 | 2618.83 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
21 | 521105408 | 127223 | 16.57 | 4240.65 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
22 | 522428416 | 127546 | 16.61 | 4251.41 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
23 | 526249984 | 128479 | 16.73 | 4282.51 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
24 | 466419712 | 113872 | 14.83 | 3795.63 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
25 | 516202496 | 126026 | 16.41 | 4200.75 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
26 | 423526400 | 103400 | 13.46 | 3446.57 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
27 | 519311360 | 126785 | 16.51 | 4226.05 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
28 | 322256896 | 78676 | 10.24 | 2622.46 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
29 | 523022336 | 127691 | 16.63 | 4256.25 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
30 | 524419072 | 128032 | 16.67 | 4267.61 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
31 | 526954496 | 128651 | 16.75 | 4288.24 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 15284953088 | 3731678 | 485.88 | 124385.72
Read IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 467173376 | 114056 | 14.85 | 3801.76 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
1 | 517079040 | 126240 | 16.44 | 4207.88 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
2 | 424247296 | 103576 | 13.49 | 3452.43 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
3 | 518680576 | 126631 | 16.49 | 4220.91 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
4 | 321974272 | 78607 | 10.23 | 2620.16 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
5 | 522252288 | 127503 | 16.60 | 4249.98 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
6 | 523501568 | 127808 | 16.64 | 4260.15 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
7 | 526270464 | 128484 | 16.73 | 4282.68 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
8 | 466718720 | 113945 | 14.84 | 3798.06 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
9 | 516403200 | 126075 | 16.42 | 4202.38 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
10 | 423788544 | 103464 | 13.47 | 3448.70 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
11 | 519172096 | 126751 | 16.50 | 4224.91 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
12 | 324067328 | 79118 | 10.30 | 2637.19 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
13 | 522407936 | 127541 | 16.61 | 4251.25 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
14 | 523976704 | 127924 | 16.66 | 4264.01 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
15 | 526610432 | 128567 | 16.74 | 4285.44 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
16 | 466575360 | 113910 | 14.83 | 3796.89 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
17 | 516907008 | 126198 | 16.43 | 4206.48 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
18 | 424329216 | 103596 | 13.49 | 3453.10 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
19 | 519110656 | 126736 | 16.50 | 4224.41 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
20 | 321810432 | 78567 | 10.23 | 2618.83 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
21 | 521105408 | 127223 | 16.57 | 4240.65 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
22 | 522428416 | 127546 | 16.61 | 4251.41 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
23 | 526249984 | 128479 | 16.73 | 4282.51 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
24 | 466419712 | 113872 | 14.83 | 3795.63 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
25 | 516202496 | 126026 | 16.41 | 4200.75 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
26 | 423526400 | 103400 | 13.46 | 3446.57 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
27 | 519311360 | 126785 | 16.51 | 4226.05 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
28 | 322256896 | 78676 | 10.24 | 2622.46 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
29 | 523022336 | 127691 | 16.63 | 4256.25 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
30 | 524419072 | 128032 | 16.67 | 4267.61 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
31 | 526954496 | 128651 | 16.75 | 4288.24 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 15284953088 | 3731678 | 485.88 | 124385.72
Write IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
1 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
2 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
3 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
4 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
5 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
6 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
7 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
8 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
9 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
10 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
11 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
12 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
13 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
14 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
15 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
16 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
17 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
18 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
19 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
20 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
21 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
22 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
23 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
24 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
25 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
26 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
27 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
28 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
29 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
30 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
31 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 0 | 0 | 0.00 | 0.00
write iops
Command Line: diskspd.exe -b4k -d30 -o4 -t32 -r -Su -w100 -c10G C:\ClusterStorage\Volume1\iotest.dat
Input parameters:
timespan: 1
-------------
duration: 30s
warm up time: 5s
cool down time: 0s
random seed: 0
path: 'C:\ClusterStorage\Volume1\iotest.dat'
think time: 0ms
burst size: 0
software cache disabled
using hardware write cache, writethrough off
performing write test
block size: 4096
using random I/O (alignment: 4096)
number of outstanding I/O operations: 4
thread stride size: 0
threads per file: 32
using I/O Completion Ports
IO priority: normal
System information:
computer name: HVTEST01
start time: 2020/07/27 18:43:09 UTC
Results for timespan 1:
*******************************************************************************
actual test time: 30.00s
thread count: 32
proc count: 8
CPU | Usage | User | Kernel | Idle
-------------------------------------------
0| 11.46%| 1.77%| 9.69%| 88.54%
1| 9.32%| 1.41%| 7.92%| 90.68%
2| 13.02%| 1.56%| 11.46%| 86.98%
3| 8.39%| 1.15%| 7.24%| 91.61%
4| 13.75%| 0.26%| 13.49%| 86.25%
5| 7.08%| 0.42%| 6.67%| 92.92%
6| 6.20%| 0.36%| 5.83%| 93.80%
7| 6.35%| 0.47%| 5.89%| 93.65%
-------------------------------------------
avg.| 9.45%| 0.92%| 8.52%| 90.55%
Total IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 61890560 | 15110 | 1.97 | 503.66 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
1 | 61554688 | 15028 | 1.96 | 500.93 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
2 | 61644800 | 15050 | 1.96 | 501.66 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
3 | 61874176 | 15106 | 1.97 | 503.53 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
4 | 61566976 | 15031 | 1.96 | 501.03 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
5 | 62095360 | 15160 | 1.97 | 505.33 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
6 | 62324736 | 15216 | 1.98 | 507.20 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
7 | 62148608 | 15173 | 1.98 | 505.76 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
8 | 61898752 | 15112 | 1.97 | 503.73 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
9 | 61767680 | 15080 | 1.96 | 502.66 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
10 | 61870080 | 15105 | 1.97 | 503.50 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
11 | 62222336 | 15191 | 1.98 | 506.36 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
12 | 61886464 | 15109 | 1.97 | 503.63 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
13 | 62029824 | 15144 | 1.97 | 504.80 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
14 | 62164992 | 15177 | 1.98 | 505.90 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
15 | 62124032 | 15167 | 1.97 | 505.56 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
16 | 61771776 | 15081 | 1.96 | 502.70 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
17 | 62070784 | 15154 | 1.97 | 505.13 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
18 | 61718528 | 15068 | 1.96 | 502.26 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
19 | 61804544 | 15089 | 1.96 | 502.96 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
20 | 61603840 | 15040 | 1.96 | 501.33 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
21 | 62234624 | 15194 | 1.98 | 506.46 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
22 | 62201856 | 15186 | 1.98 | 506.20 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
23 | 62054400 | 15150 | 1.97 | 505.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
24 | 61616128 | 15043 | 1.96 | 501.43 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
25 | 61788160 | 15085 | 1.96 | 502.83 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
26 | 61464576 | 15006 | 1.95 | 500.20 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
27 | 61927424 | 15119 | 1.97 | 503.96 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
28 | 61399040 | 14990 | 1.95 | 499.66 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
29 | 61911040 | 15115 | 1.97 | 503.83 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
30 | 61943808 | 15123 | 1.97 | 504.10 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
31 | 62058496 | 15151 | 1.97 | 505.03 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 1980633088 | 483553 | 62.96 | 16118.34
Read IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
1 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
2 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
3 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
4 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
5 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
6 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
7 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
8 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
9 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
10 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
11 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
12 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
13 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
14 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
15 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
16 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
17 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
18 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
19 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
20 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
21 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
22 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
23 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
24 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
25 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
26 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
27 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
28 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
29 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
30 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
31 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 0 | 0 | 0.00 | 0.00
Write IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 61890560 | 15110 | 1.97 | 503.66 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
1 | 61554688 | 15028 | 1.96 | 500.93 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
2 | 61644800 | 15050 | 1.96 | 501.66 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
3 | 61874176 | 15106 | 1.97 | 503.53 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
4 | 61566976 | 15031 | 1.96 | 501.03 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
5 | 62095360 | 15160 | 1.97 | 505.33 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
6 | 62324736 | 15216 | 1.98 | 507.20 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
7 | 62148608 | 15173 | 1.98 | 505.76 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
8 | 61898752 | 15112 | 1.97 | 503.73 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
9 | 61767680 | 15080 | 1.96 | 502.66 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
10 | 61870080 | 15105 | 1.97 | 503.50 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
11 | 62222336 | 15191 | 1.98 | 506.36 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
12 | 61886464 | 15109 | 1.97 | 503.63 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
13 | 62029824 | 15144 | 1.97 | 504.80 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
14 | 62164992 | 15177 | 1.98 | 505.90 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
15 | 62124032 | 15167 | 1.97 | 505.56 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
16 | 61771776 | 15081 | 1.96 | 502.70 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
17 | 62070784 | 15154 | 1.97 | 505.13 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
18 | 61718528 | 15068 | 1.96 | 502.26 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
19 | 61804544 | 15089 | 1.96 | 502.96 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
20 | 61603840 | 15040 | 1.96 | 501.33 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
21 | 62234624 | 15194 | 1.98 | 506.46 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
22 | 62201856 | 15186 | 1.98 | 506.20 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
23 | 62054400 | 15150 | 1.97 | 505.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
24 | 61616128 | 15043 | 1.96 | 501.43 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
25 | 61788160 | 15085 | 1.96 | 502.83 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
26 | 61464576 | 15006 | 1.95 | 500.20 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
27 | 61927424 | 15119 | 1.97 | 503.96 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
28 | 61399040 | 14990 | 1.95 | 499.66 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
29 | 61911040 | 15115 | 1.97 | 503.83 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
30 | 61943808 | 15123 | 1.97 | 504.10 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
31 | 62058496 | 15151 | 1.97 | 505.03 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 1980633088 | 483553 | 62.96 | 16118.34
read latency
Command Line: diskspd.exe -b4k -d30 -o1 -t1 -r -Su -w0 -c10G C:\ClusterStorage\Volume1\iotest.dat
Input parameters:
timespan: 1
-------------
duration: 30s
warm up time: 5s
cool down time: 0s
random seed: 0
path: 'C:\ClusterStorage\Volume1\iotest.dat'
think time: 0ms
burst size: 0
software cache disabled
using hardware write cache, writethrough off
performing read test
block size: 4096
using random I/O (alignment: 4096)
number of outstanding I/O operations: 1
thread stride size: 0
threads per file: 1
IO priority: normal
System information:
computer name: HVTEST01
start time: 2020/07/27 18:45:05 UTC
Results for timespan 1:
*******************************************************************************
actual test time: 30.00s
thread count: 1
proc count: 8
CPU | Usage | User | Kernel | Idle
-------------------------------------------
0| 10.83%| 2.19%| 8.65%| 89.17%
1| 3.65%| 2.76%| 0.89%| 96.35%
2| 1.98%| 0.21%| 1.77%| 98.02%
3| 0.05%| 0.05%| 0.00%| 99.95%
4| 2.14%| 0.00%| 2.14%| 97.86%
5| 0.10%| 0.10%| 0.00%| 99.90%
6| 0.21%| 0.05%| 0.16%| 99.79%
7| 0.10%| 0.00%| 0.10%| 99.90%
-------------------------------------------
avg.| 2.38%| 0.67%| 1.71%| 97.62%
Total IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 254722048 | 62188 | 8.10 | 2072.89 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 254722048 | 62188 | 8.10 | 2072.89
Read IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 254722048 | 62188 | 8.10 | 2072.89 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 254722048 | 62188 | 8.10 | 2072.89
Write IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 0 | 0 | 0.00 | 0.00
write latency
Command Line: diskspd.exe -b4k -d30 -o1 -t1 -r -Su -w100 -c10G C:\ClusterStorage\Volume1\iotest.dat
Input parameters:
timespan: 1
-------------
duration: 30s
warm up time: 5s
cool down time: 0s
random seed: 0
path: 'C:\ClusterStorage\Volume1\iotest.dat'
think time: 0ms
burst size: 0
software cache disabled
using hardware write cache, writethrough off
performing write test
block size: 4096
using random I/O (alignment: 4096)
number of outstanding I/O operations: 1
thread stride size: 0
threads per file: 1
IO priority: normal
System information:
computer name: HVTEST01
start time: 2020/07/27 18:46:41 UTC
Results for timespan 1:
*******************************************************************************
actual test time: 30.00s
thread count: 1
proc count: 8
CPU | Usage | User | Kernel | Idle
-------------------------------------------
0| 6.61%| 2.03%| 4.58%| 93.39%
1| 3.49%| 2.29%| 1.20%| 96.51%
2| 0.26%| 0.16%| 0.10%| 99.74%
3| 0.05%| 0.00%| 0.05%| 99.95%
4| 0.47%| 0.16%| 0.31%| 99.53%
5| 0.10%| 0.10%| 0.00%| 99.90%
6| 0.05%| 0.00%| 0.05%| 99.95%
7| 0.00%| 0.00%| 0.00%| 100.00%
-------------------------------------------
avg.| 1.38%| 0.59%| 0.79%| 98.62%
Total IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 92803072 | 22657 | 2.95 | 755.22 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 92803072 | 22657 | 2.95 | 755.22
Read IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 0 | 0 | 0.00 | 0.00 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 0 | 0 | 0.00 | 0.00
Write IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 92803072 | 22657 | 2.95 | 755.22 | C:\ClusterStorage\Volume1\iotest.dat (10240MiB)
------------------------------------------------------------------------------
total: 92803072 | 22657 | 2.95 | 755.22
admin
2,930 Posts
Quote from admin on July 27, 2020, 9:16 pmThe iSCSI layer is fine, the latency is 0.5 ms read, 1.3 ms write. So if there is a issue with format speed, it is something else.
The iops test with your current setup was 124k read, 16k write. You may get higher write numbers by increasing thread count of the test from 32->128/256. To give further boost, look at the charts for cpu and disk % util ( better run the test for 3-5 min to get nice chart ), most probably your cpu will not be saturated due to the low count of SSDs (2) per node, which means you could add more disks per node and get linear performance increase.
Good luck 🙂
The iSCSI layer is fine, the latency is 0.5 ms read, 1.3 ms write. So if there is a issue with format speed, it is something else.
The iops test with your current setup was 124k read, 16k write. You may get higher write numbers by increasing thread count of the test from 32->128/256. To give further boost, look at the charts for cpu and disk % util ( better run the test for 3-5 min to get nice chart ), most probably your cpu will not be saturated due to the low count of SSDs (2) per node, which means you could add more disks per node and get linear performance increase.
Good luck 🙂
JG
26 Posts
Quote from JG on July 28, 2020, 7:19 pmThank you for your time and responses.
About the problem of the slow formatting of large drives, the problem is not CEPH | iSCSI | the hardware layer but in Windows (at least in Windows Server 2019 (1809) this is a problem and the solution that I leave below solves it)
The solution is "disable the SCSI TRIM and Unmap feature on the Host for the duration of the format" [1]fsutil behavior set DisableDeleteNotify 1
To enable this feature once the formatting is complete:
fsutil behavior set DisableDeleteNotify 0
To verify the current configuration:
fsutil behavior query DisableDeleteNotify
DisableDeleteNotify = 0 - indicates the 'Trim and Unmap' feature is on (enabled)
DisableDeleteNotify = 1 - indicates the 'Trim and Unmap' feature is off (disabled)This topic can be considered closed.
Thank you for your time and responses.
About the problem of the slow formatting of large drives, the problem is not CEPH | iSCSI | the hardware layer but in Windows (at least in Windows Server 2019 (1809) this is a problem and the solution that I leave below solves it)
The solution is "disable the SCSI TRIM and Unmap feature on the Host for the duration of the format" [1]
fsutil behavior set DisableDeleteNotify 1
To enable this feature once the formatting is complete:
fsutil behavior set DisableDeleteNotify 0
To verify the current configuration:
fsutil behavior query DisableDeleteNotify
DisableDeleteNotify = 0 - indicates the 'Trim and Unmap' feature is on (enabled)
DisableDeleteNotify = 1 - indicates the 'Trim and Unmap' feature is off (disabled)
This topic can be considered closed.
wid
47 Posts
Quote from wid on December 4, 2020, 7:30 pmHi,
I have the same problem,
iSCSI lun, successfuly conneced (2 path) to windows 2019, format ReFS (5 GB, 10TB, any size) takes about 12-20 hours. Yes, hours.
I tried to switch TRIM (
fsutil behavior query DisableDeleteNotify NTFS DisableDeleteNotify = 1 (Enabled) ReFS DisableDeleteNotify = 1 (Enabled) But no success. Is there any option to debug this case? Lun is working fine when connected to ESX.
Hi,
I have the same problem,
iSCSI lun, successfuly conneced (2 path) to windows 2019, format ReFS (5 GB, 10TB, any size) takes about 12-20 hours. Yes, hours.
I tried to switch TRIM (
fsutil behavior query DisableDeleteNotify
NTFS DisableDeleteNotify = 1 (Enabled)
ReFS DisableDeleteNotify = 1 (Enabled)
But no success. Is there any option to debug this case?
Lun is working fine when connected to ESX.