Forums

Home / Forums

You need to log in to create posts and topics. Login · Register

Failed to migrate VM on VMware to Petasan Storage


zamasaki158
13 Posts
April 24, 2020, 4:28 am

Hi,

I created an ISCSI disk and mounted it into vmware. I tried to migrate VM to  Petasan storage but failed.

ESXi host vmkernel.log:

2020-04-24T04:07:52.024Z cpu8:65616)ScsiDeviceIO: 2965: Cmd(0x439d0b729880) 0x83, CmdSN 0x12e066 from world 70609 to dev "naa.6001405140fdf4400028000000000000" failed H:0x8 D:0x0 P:0x0
2020-04-24T04:07:52.024Z cpu8:65616)WARNING: NMP: nmp_DeviceStartLoop:729: NMP Device "naa.6001405140fdf4400028000000000000" is blocked. Not starting I/O from device.
2020-04-24T04:07:52.024Z cpu8:65616)ScsiDeviceIO: 2965: Cmd(0x439d07a3ce80) 0x83, CmdSN 0x12e067 from world 70609 to dev "naa.6001405140fdf4400028000000000000" failed H:0x8 D:0x0 P:0x0
2020-04-24T04:07:52.024Z cpu8:65616)WARNING: NMP: nmp_DeviceStartLoop:729: NMP Device "naa.6001405140fdf4400028000000000000" is blocked. Not starting I/O from device.
2020-04-24T04:07:52.024Z cpu8:65616)ScsiDeviceIO: 2965: Cmd(0x439d0b794480) 0x83, CmdSN 0x12e068 from world 70609 to dev "naa.6001405140fdf4400028000000000000" failed H:0x8 D:0x0 P:0x0
2020-04-24T04:07:52.024Z cpu8:65616)WARNING: NMP: nmp_DeviceStartLoop:729: NMP Device "naa.6001405140fdf4400028000000000000" is blocked. Not starting I/O from device.
2020-04-24T04:07:52.024Z cpu8:65616)ScsiDeviceIO: 2965: Cmd(0x439d07b09e00) 0x83, CmdSN 0x12e069 from world 70609 to dev "naa.6001405140fdf4400028000000000000" failed H:0x8 D:0x0 P:0x0
2020-04-24T04:07:52.024Z cpu8:65616)WARNING: NMP: nmp_DeviceStartLoop:729: NMP Device "naa.6001405140fdf4400028000000000000" is blocked. Not starting I/O from device.
2020-04-24T04:07:52.024Z cpu8:65616)ScsiDeviceIO: 2965: Cmd(0x439d00950ec0) 0x83, CmdSN 0x12e06a from world 70609 to dev "naa.6001405140fdf4400028000000000000" failed H:0x8 D:0x0 P:0x0
2020-04-24T04:07:52.024Z cpu8:65616)WARNING: NMP: nmp_DeviceStartLoop:729: NMP Device "naa.6001405140fdf4400028000000000000" is blocked. Not starting I/O from device.
2020-04-24T04:07:52.024Z cpu8:65616)WARNING: NMP: nmp_DeviceStartLoop:729: NMP Device "naa.6001405140fdf4400028000000000000" is blocked. Not starting I/O from device.
2020-04-24T04:07:52.024Z cpu8:65616)WARNING: NMP: nmp_DeviceStartLoop:729: NMP Device "naa.6001405140fdf4400028000000000000" is blocked. Not starting I/O from device.
2020-04-24T04:07:52.024Z cpu8:65616)WARNING: NMP: nmp_DeviceStartLoop:729: NMP Device "naa.6001405140fdf4400028000000000000" is blocked. Not starting I/O from device.
2020-04-24T04:07:52.024Z cpu8:65616)WARNING: NMP: nmp_DeviceStartLoop:729: NMP Device "naa.6001405140fdf4400028000000000000" is blocked. Not starting I/O from device.
2020-04-24T04:07:52.024Z cpu8:65616)WARNING: NMP: nmp_DeviceStartLoop:729: NMP Device "naa.6001405140fdf4400028000000000000" is blocked. Not starting I/O from device.
2020-04-24T04:07:52.024Z cpu8:65616)WARNING: NMP: nmp_DeviceStartLoop:729: NMP Device "naa.6001405140fdf4400028000000000000" is blocked. Not starting I/O from device.
2020-04-24T04:07:52.024Z cpu8:65616)WARNING: NMP: nmp_DeviceStartLoop:729: NMP Device "naa.6001405140fdf4400028000000000000" is blocked. Not starting I/O from device.
2020-04-24T04:07:52.024Z cpu8:65616)WARNING: NMP: nmp_DeviceStartLoop:729: NMP Device "naa.6001405140fdf4400028000000000000" is blocked. Not starting I/O from device.
2020-04-24T04:07:52.024Z cpu8:65585)ScsiDeviceIO: 2965: Cmd(0x439d07aee180) 0x28, CmdSN 0x12e073 from world 67159 to dev "naa.6001405140fdf4400028000000000000" failed H:0x8 D:0x0 P:0x0
2020-04-24T04:07:52.024Z cpu9:65586)ScsiDeviceIO: 2965: Cmd(0x439d0091d840) 0x8a, CmdSN 0x12e075 from world 67178 to dev "naa.6001405140fdf4400028000000000000" failed H:0x8 D:0x0 P:0x0
2020-04-24T04:07:52.391Z cpu6:11034352)WARNING: NMP: nmpDeviceAttemptFailover:603: Retry world restore device "naa.6001405140fdf4400028000000000000" - no more commands to retry
2020-04-24T04:07:52.397Z cpu9:65616)HBX: 282: 'LUN1-BCACHE-TURBO-DC4': HB at offset 3383296 - Reclaimed heartbeat [Timeout]:
2020-04-24T04:07:52.397Z cpu9:65616) [HB state abcdef02 offset 3383296 gen 5 stampUS 12831631899332 uuid 5dde9b2f-4a2ad2d6-2ee7-00a0d1eab564 jrnl <FB 42428800> drv 14.81 lockImpl 4 ip 10.0.48.47]

[root@localhost:~] tail -f /var/log/vmkernel.log
2020-04-24T04:09:58.021Z cpu13:65590)ScsiDeviceIO: 3015: Cmd(0x43950edc1580) 0x89, CmdSN 0x12e2b8 from world 65567 to dev "naa.6001405140fdf4400028000000000000" failed H:0x5 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0.
2020-04-24T04:09:58.023Z cpu1:65615)ScsiDeviceIO: 2965: Cmd(0x439d009959c0) 0x83, CmdSN 0x12e2b3 from world 70609 to dev "naa.6001405140fdf4400028000000000000" failed H:0x8 D:0x0 P:0x0
2020-04-24T04:09:58.023Z cpu1:65615)WARNING: NMP: nmp_DeviceStartLoop:729: NMP Device "naa.6001405140fdf4400028000000000000" is blocked. Not starting I/O from device.
2020-04-24T04:09:58.023Z cpu1:65615)ScsiDeviceIO: 2965: Cmd(0x439d0b76e700) 0x83, CmdSN 0x12e2b4 from world 70609 to dev "naa.6001405140fdf4400028000000000000" failed H:0x8 D:0x0 P:0x0
2020-04-24T04:09:58.023Z cpu1:65615)WARNING: NMP: nmp_DeviceStartLoop:729: NMP Device "naa.6001405140fdf4400028000000000000" is blocked. Not starting I/O from device.
2020-04-24T04:09:58.023Z cpu1:65578)ScsiDeviceIO: 2965: Cmd(0x439d07ba8400) 0x28, CmdSN 0x12e2b5 from world 67159 to dev "naa.6001405140fdf4400028000000000000" failed H:0x8 D:0x0 P:0x0
2020-04-24T04:09:58.023Z cpu1:65578)ScsiDeviceIO: 2965: Cmd(0x439d07a90e80) 0x8a, CmdSN 0x12e2b7 from world 67178 to dev "naa.6001405140fdf4400028000000000000" failed H:0x8 D:0x0 P:0x0
2020-04-24T04:09:58.390Z cpu0:11034352)WARNING: NMP: nmpDeviceAttemptFailover:603: Retry world restore device "naa.6001405140fdf4400028000000000000" - no more commands to retry
2020-04-24T04:09:58.402Z cpu1:65615)HBX: 282: 'LUN1-BCACHE-TURBO-DC4': HB at offset 3383296 - Reclaimed heartbeat [Timeout]:
2020-04-24T04:09:58.402Z cpu1:65615) [HB state abcdef02 offset 3383296 gen 5 stampUS 12831757898502 uuid 5dde9b2f-4a2ad2d6-2ee7-00a0d1eab564 jrnl <FB 42428800> drv 14.81 lockImpl 4 ip 10.0.48.47]
2020-04-24T04:10:00.094Z cpu22:67159)DLX: 4312: vol 'LUN3-PSAN-DC4', lock at 264343552: [Req mode 1] Checking liveness:
2020-04-24T04:10:00.094Z cpu22:67159)[type 10c00001 offset 264343552 v 40335350, hb offset 3584512
gen 111, mode 1, owner 5e61b511-81fc5d63-e2b1-0cc47a1da4e2 mtime 2960930
num 0 gblnum 0 gblgen 0 gblbrk 0]
2020-04-24T04:10:01.025Z cpu5:66379)WARNING: NMP: nmp_DeviceRetryCommand:133: Device "naa.6001405140fdf4400028000000000000": awaiting fast path state update for failover with I/O blocked. No prior reservation exists on the device.
2020-04-24T04:10:01.404Z cpu5:66162)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405140fdf4400028000000000000" - issuing command 0x439d0b770680
2020-04-24T04:10:01.404Z cpu5:66379)WARNING: NMP: nmpCompleteRetryForPath:357: Retry cmd 0x83 (0x439d0b770680) to dev "naa.6001405140fdf4400028000000000000" failed on path "vmhba65:C1:T4:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x8 0x0.
2020-04-24T04:10:01.404Z cpu5:66379)WARNING: NMP: nmpCompleteRetryForPath:387: Logical device "naa.6001405140fdf4400028000000000000": awaiting fast path state update before retrying failed command again...
2020-04-24T04:10:02.390Z cpu0:11034352)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405140fdf4400028000000000000" - issuing command 0x439d0b770680
2020-04-24T04:10:02.390Z cpu5:66379)WARNING: NMP: nmpCompleteRetryForPath:357: Retry cmd 0x83 (0x439d0b770680) to dev "naa.6001405140fdf4400028000000000000" failed on path "vmhba65:C1:T4:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x8 0x0.
2020-04-24T04:10:02.390Z cpu5:66379)WARNING: NMP: nmpCompleteRetryForPath:387: Logical device "naa.6001405140fdf4400028000000000000": awaiting fast path state update before retrying failed command again...
2020-04-24T04:10:03.393Z cpu5:66162)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405140fdf4400028000000000000" - issuing command 0x439d0b770680
2020-04-24T04:10:03.394Z cpu5:66379)WARNING: NMP: nmpCompleteRetryForPath:357: Retry cmd 0x83 (0x439d0b770680) to dev "naa.6001405140fdf4400028000000000000" failed on path "vmhba65:C1:T4:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x8 0x0.
2020-04-24T04:10:03.394Z cpu5:66379)WARNING: NMP: nmpCompleteRetryForPath:387: Logical device "naa.6001405140fdf4400028000000000000": awaiting fast path state update before retrying failed command again...
2020-04-24T04:10:04.390Z cpu0:11034352)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405140fdf4400028000000000000" - issuing command 0x439d0b770680
2020-04-24T04:10:04.391Z cpu5:66379)WARNING: NMP: nmpCompleteRetryForPath:357: Retry cmd 0x83 (0x439d0b770680) to dev "naa.6001405140fdf4400028000000000000" failed on path "vmhba65:C1:T4:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x8 0x0.
2020-04-24T04:10:04.391Z cpu5:66379)WARNING: NMP: nmpCompleteRetryForPath:387: Logical device "naa.6001405140fdf4400028000000000000": awaiting fast path state update before retrying failed command again...
2020-04-24T04:10:05.392Z cpu5:66162)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405140fdf4400028000000000000" - issuing command 0x439d0b770680
2020-04-24T04:10:05.392Z cpu0:66379)WARNING: NMP: nmpCompleteRetryForPath:357: Retry cmd 0x83 (0x439d0b770680) to dev "naa.6001405140fdf4400028000000000000" failed on path "vmhba65:C1:T4:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x8 0x0.
2020-04-24T04:10:05.392Z cpu0:66379)WARNING: NMP: nmpCompleteRetryForPath:387: Logical device "naa.6001405140fdf4400028000000000000": awaiting fast path state update before retrying failed command again...
2020-04-24T04:10:06.390Z cpu0:11034352)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405140fdf4400028000000000000" - issuing command 0x439d0b770680
2020-04-24T04:10:06.390Z cpu0:66379)WARNING: NMP: nmpCompleteRetryForPath:357: Retry cmd 0x83 (0x439d0b770680) to dev "naa.6001405140fdf4400028000000000000" failed on path "vmhba65:C1:T4:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x8 0x0.
2020-04-24T04:10:06.390Z cpu0:66379)WARNING: NMP: nmpCompleteRetryForPath:387: Logical device "naa.6001405140fdf4400028000000000000": awaiting fast path state update before retrying failed command again...
2020-04-24T04:10:07.390Z cpu5:66162)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405140fdf4400028000000000000" - issuing command 0x439d0b770680
2020-04-24T04:10:07.391Z cpu5:66379)WARNING: NMP: nmpCompleteRetryForPath:357: Retry cmd 0x83 (0x439d0b770680) to dev "naa.6001405140fdf4400028000000000000" failed on path "vmhba65:C1:T4:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x8 0x0.
2020-04-24T04:10:07.391Z cpu5:66379)WARNING: NMP: nmpCompleteRetryForPath:387: Logical device "naa.6001405140fdf4400028000000000000": awaiting fast path state update before retrying failed command again...
2020-04-24T04:10:08.390Z cpu2:11034352)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405140fdf4400028000000000000" - issuing command 0x439d0b770680
2020-04-24T04:10:08.391Z cpu0:66379)WARNING: NMP: nmpCompleteRetryForPath:357: Retry cmd 0x83 (0x439d0b770680) to dev "naa.6001405140fdf4400028000000000000" failed on path "vmhba65:C1:T4:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x8 0x0.
2020-04-24T04:10:08.391Z cpu0:66379)WARNING: NMP: nmpCompleteRetryForPath:387: Logical device "naa.6001405140fdf4400028000000000000": awaiting fast path state update before retrying failed command again...
2020-04-24T04:10:09.390Z cpu0:66162)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405140fdf4400028000000000000" - issuing command 0x439d0b770680
2020-04-24T04:10:09.390Z cpu9:66379)WARNING: NMP: nmpCompleteRetryForPath:357: Retry cmd 0x83 (0x439d0b770680) to dev "naa.6001405140fdf4400028000000000000" failed on path "vmhba65:C1:T4:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x8 0x0.
2020-04-24T04:10:09.390Z cpu9:66379)WARNING: NMP: nmpCompleteRetryForPath:387: Logical device "naa.6001405140fdf4400028000000000000": awaiting fast path state update before retrying failed command again...
2020-04-24T04:10:10.390Z cpu2:11034352)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405140fdf4400028000000000000" - issuing command 0x439d0b770680
2020-04-24T04:10:10.390Z cpu9:66379)WARNING: NMP: nmpCompleteRetryForPath:357: Retry cmd 0x83 (0x439d0b770680) to dev "naa.6001405140fdf4400028000000000000" failed on path "vmhba65:C1:T4:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x8 0x0.
2020-04-24T04:10:10.390Z cpu9:66379)WARNING: NMP: nmpCompleteRetryForPath:387: Logical device "naa.6001405140fdf4400028000000000000": awaiting fast path state update before retrying failed command again...
2020-04-24T04:10:11.391Z cpu7:66162)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405140fdf4400028000000000000" - issuing command 0x439d0b770680
2020-04-24T04:10:11.391Z cpu8:66379)WARNING: NMP: nmpCompleteRetryForPath:357: Retry cmd 0x83 (0x439d0b770680) to dev "naa.6001405140fdf4400028000000000000" failed on path "vmhba65:C1:T4:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x8 0x0.
2020-04-24T04:10:11.391Z cpu8:66379)WARNING: NMP: nmpCompleteRetryForPath:387: Logical device "naa.6001405140fdf4400028000000000000": awaiting fast path state update before retrying failed command again...
2020-04-24T04:10:12.027Z cpu14:65591)ScsiDeviceIO: 3015: Cmd(0x43950ece5b80) 0x89, CmdSN 0x12e2d3 from world 65567 to dev "naa.6001405140fdf4400028000000000000" failed H:0x5 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0.
2020-04-24T04:10:12.390Z cpu2:11034352)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405140fdf4400028000000000000" - issuing command 0x439d0b770680
2020-04-24T04:10:12.391Z cpu8:66379)WARNING: NMP: nmpCompleteRetryForPath:357: Retry cmd 0x83 (0x439d0b770680) to dev "naa.6001405140fdf4400028000000000000" failed on path "vmhba65:C1:T4:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x8 0x0.
2020-04-24T04:10:12.391Z cpu8:66379)WARNING: NMP: nmpCompleteRetryForPath:387: Logical device "naa.6001405140fdf4400028000000000000": awaiting fast path state update before retrying failed command again...
2020-04-24T04:10:13.021Z cpu12:65589)ScsiDeviceIO: 3015: Cmd(0x43950e06c200) 0x89, CmdSN 0x12e2d6 from world 65567 to dev "naa.6001405140fdf4400028000000000000" failed H:0x5 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0.
2020-04-24T04:10:13.024Z cpu0:65616)ScsiDeviceIO: 2965: Cmd(0x439d0b770680) 0x83, CmdSN 0x12e2d1 from world 70609 to dev "naa.6001405140fdf4400028000000000000" failed H:0x8 D:0x0 P:0x0
2020-04-24T04:10:13.024Z cpu0:65616)WARNING: NMP: nmp_DeviceStartLoop:729: NMP Device "naa.6001405140fdf4400028000000000000" is blocked. Not starting I/O from device.
2020-04-24T04:10:13.024Z cpu0:65616)ScsiDeviceIO: 2965: Cmd(0x439d07b7cf00) 0x83, CmdSN 0x12e2d2 from world 70609 to dev "naa.6001405140fdf4400028000000000000" failed H:0x8 D:0x0 P:0x0
2020-04-24T04:10:13.024Z cpu0:65616)WARNING: NMP: nmp_DeviceStartLoop:729: NMP Device "naa.6001405140fdf4400028000000000000" is blocked. Not starting I/O from device.
2020-04-24T04:10:13.024Z cpu0:65577)ScsiDeviceIO: 2965: Cmd(0x439d07b80700) 0x28, CmdSN 0x12e2d4 from world 11034091 to dev "naa.6001405140fdf4400028000000000000" failed H:0x8 D:0x0 P:0x0
2020-04-24T04:10:13.024Z cpu2:65579)ScsiDeviceIO: 2965: Cmd(0x439d009879c0) 0x8a, CmdSN 0x12e2d5 from world 67178 to dev "naa.6001405140fdf4400028000000000000" failed H:0x8 D:0x0 P:0x0
2020-04-24T04:10:13.391Z cpu7:66162)WARNING: NMP: nmpDeviceAttemptFailover:603: Retry world restore device "naa.6001405140fdf4400028000000000000" - no more commands to retry
2020-04-24T04:10:13.403Z cpu2:65616)HBX: 282: 'LUN1-BCACHE-TURBO-DC4': HB at offset 3383296 - Reclaimed heartbeat [Timeout]:
2020-04-24T04:10:13.403Z cpu2:65616) [HB state abcdef02 offset 3383296 gen 5 stampUS 12831772900332 uuid 5dde9b2f-4a2ad2d6-2ee7-00a0d1eab564 jrnl <FB 42428800> drv 14.81 lockImpl 4 ip 10.0.48.47]
2020-04-24T04:10:14.544Z cpu22:11034091)WARNING: Migrate: 6282: 6120682374460922369 D: Migration considered a failure by the VMX. It is most likely a timeout, but check the VMX log for the true error.
2020-04-24T04:10:14.544Z cpu22:11034091)WARNING: Migrate: 273: 6120682374460922369 D: Failed: Migration determined a failure by the VMX (0xbad0092) @0x41800a32bb41
2020-04-24T04:10:14.544Z cpu22:11034091)WARNING: Migrate: 273: 6120682374460922369 S: Failed: Migration determined a failure by the VMX (0xbad0092) @0x41800b49047f
2020-04-24T04:10:14.544Z cpu12:11034322)WARNING: FSR: 1452: 6120682374460922369 S: Swap copy helper exit requested. Exiting migration.
2020-04-24T04:10:14.544Z cpu12:11034322)WARNING: FSR: 1745: 6120682374460922369 S: Swap copy failed after copy is complete, exiting.
2020-04-24T04:10:14.544Z cpu12:11034322)SVM: 2456: SVM Mirrored mode IO stats for device: 3707d3-7eb7171a-svmmirror
2020-04-24T04:10:14.544Z cpu12:11034322)SVM: 2460: Total # IOs mirrored: 0, Total # IOs sent only to source: 0, Total # IO deferred by lock: 0
2020-04-24T04:10:14.544Z cpu12:11034322)SVM: 2464: Deferred IO stats - Max: 0, Total: 0, Avg: 1 (msec)
2020-04-24T04:10:14.544Z cpu12:11034322)SVM: 2478: Destroyed device 3707d3-7eb7171a-svmmirror
2020-04-24T04:10:14.544Z cpu12:11034322)WARNING: FSR: 1778: 6120682374460922369 S: Swap copy helper failed, exiting.
2020-04-24T04:10:14.544Z cpu22:11034091)Hbr: 3489: Migration end received (worldID=11034094) (migrateType=2) (event=1) (isSource=0) (sharedConfig=0)
2020-04-24T04:10:14.638Z cpu13:68080)Config: 706: "SIOControlFlag2" = 1, Old Value: 2, (Status: 0x0)
2020-04-24T04:10:14.639Z cpu20:70583)WARNING: Migrate: 6282: 6120682374460922369 S: Migration considered a failure by the VMX. It is most likely a timeout, but check the VMX log for the true error.
2020-04-24T04:10:14.639Z cpu20:70583)Hbr: 3489: Migration end received (worldID=70609) (migrateType=2) (event=1) (isSource=1) (sharedConfig=0)
2020-04-24T04:10:14.689Z cpu6:70674)VSCSI: 6590: handle 13668(vscsi0:0):Destroying Device for world 70609 (pendCom 0)
2020-04-24T04:10:14.690Z cpu6:70674)NetPort: 1881: disabled port 0x3000014
2020-04-24T04:10:14.690Z cpu6:70674)NetPort: 1881: disabled port 0x3000013
2020-04-24T04:10:16.056Z cpu10:66379)WARNING: NMP: nmp_DeviceRetryCommand:133: Device "naa.6001405140fdf4400028000000000000": awaiting fast path state update for failover with I/O blocked. No prior reservation exists on the device.
2020-04-24T04:10:16.391Z cpu2:11034352)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405140fdf4400028000000000000" - issuing command 0x439d00896540
2020-04-24T04:10:16.391Z cpu4:66379)WARNING: NMP: nmpCompleteRetryForPath:357: Retry cmd 0x83 (0x439d00896540) to dev "naa.6001405140fdf4400028000000000000" failed on path "vmhba65:C1:T4:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x8 0x0.
2020-04-24T04:10:16.391Z cpu4:66379)WARNING: NMP: nmpCompleteRetryForPath:387: Logical device "naa.6001405140fdf4400028000000000000": awaiting fast path state update before retrying failed command again...
2020-04-24T04:10:17.390Z cpu7:66162)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405140fdf4400028000000000000" - issuing command 0x439d00896540
2020-04-24T04:10:17.390Z cpu4:66379)WARNING: NMP: nmpCompleteRetryForPath:357: Retry cmd 0x83 (0x439d00896540) to dev "naa.6001405140fdf4400028000000000000" failed on path "vmhba65:C1:T4:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x8 0x0.
2020-04-24T04:10:17.390Z cpu4:66379)WARNING: NMP: nmpCompleteRetryForPath:387: Logical device "naa.6001405140fdf4400028000000000000": awaiting fast path state update before retrying failed command again...
2020-04-24T04:10:18.390Z cpu2:11034352)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405140fdf4400028000000000000" - issuing command 0x439d00896540
2020-04-24T04:10:18.390Z cpu0:66379)WARNING: NMP: nmpCompleteRetryForPath:357: Retry cmd 0x83 (0x439d00896540) to dev "naa.6001405140fdf4400028000000000000" failed on path "vmhba65:C1:T4:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x8 0x0.
2020-04-24T04:10:18.390Z cpu0:66379)WARNING: NMP: nmpCompleteRetryForPath:387: Logical device "naa.6001405140fdf4400028000000000000": awaiting fast path state update before retrying failed command again...
2020-04-24T04:10:19.389Z cpu12:66161)NMP: nmp_ResetDeviceLogThrottling:3458: last error status from device naa.6001405140fdf4400028000000000000 repeated 7 times
2020-04-24T04:10:19.390Z cpu7:66162)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405140fdf4400028000000000000" - issuing command 0x439d00896540
2020-04-24T04:10:19.391Z cpu3:66379)WARNING: NMP: nmpCompleteRetryForPath:357: Retry cmd 0x83 (0x439d00896540) to dev "naa.6001405140fdf4400028000000000000" failed on path "vmhba65:C1:T4:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x8 0x0.
2020-04-24T04:10:19.391Z cpu3:66379)WARNING: NMP: nmpCompleteRetryForPath:387: Logical device "naa.6001405140fdf4400028000000000000": awaiting fast path state update before retrying failed command again...
2020-04-24T04:10:20.390Z cpu10:11034352)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405140fdf4400028000000000000" - issuing command 0x439d00896540
2020-04-24T04:10:20.390Z cpu0:66379)WARNING: NMP: nmpCompleteRetryForPath:357: Retry cmd 0x83 (0x439d00896540) to dev "naa.6001405140fdf4400028000000000000" failed on path "vmhba65:C1:T4:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x8 0x0.
2020-04-24T04:10:20.390Z cpu0:66379)WARNING: NMP: nmpCompleteRetryForPath:387: Logical device "naa.6001405140fdf4400028000000000000": awaiting fast path state update before retrying failed command again...
2020-04-24T04:10:21.392Z cpu7:66162)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405140fdf4400028000000000000" - issuing command 0x439d00896540
2020-04-24T04:10:21.393Z cpu0:66379)WARNING: NMP: nmpCompleteRetryForPath:357: Retry cmd 0x83 (0x439d00896540) to dev "naa.6001405140fdf4400028000000000000" failed on path "vmhba65:C1:T4:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x8 0x0.
2020-04-24T04:10:21.393Z cpu0:66379)WARNING: NMP: nmpCompleteRetryForPath:387: Logical device "naa.6001405140fdf4400028000000000000": awaiting fast path state update before retrying failed command again...
2020-04-24T04:10:22.390Z cpu10:11034352)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405140fdf4400028000000000000" - issuing command 0x439d00896540
2020-04-24T04:10:22.391Z cpu0:66379)WARNING: NMP: nmpCompleteRetryForPath:357: Retry cmd 0x83 (0x439d00896540) to dev "naa.6001405140fdf4400028000000000000" failed on path "vmhba65:C1:T4:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x8 0x0.
2020-04-24T04:10:22.391Z cpu0:66379)WARNING: NMP: nmpCompleteRetryForPath:387: Logical device "naa.6001405140fdf4400028000000000000": awaiting fast path state update before retrying failed command again...
2020-04-24T04:10:23.390Z cpu7:66162)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405140fdf4400028000000000000" - issuing command 0x439d00896540
2020-04-24T04:10:23.390Z cpu4:66379)WARNING: NMP: nmpCompleteRetryForPath:357: Retry cmd 0x83 (0x439d00896540) to dev "naa.6001405140fdf4400028000000000000" failed on path "vmhba65:C1:T4:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x8 0x0.
2020-04-24T04:10:23.390Z cpu4:66379)WARNING: NMP: nmpCompleteRetryForPath:387: Logical device "naa.6001405140fdf4400028000000000000": awaiting fast path state update before retrying failed command again...
2020-04-24T04:10:24.390Z cpu10:11034352)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405140fdf4400028000000000000" - issuing command 0x439d00896540
2020-04-24T04:10:24.390Z cpu11:66379)WARNING: NMP: nmpCompleteRetryForPath:357: Retry cmd 0x83 (0x439d00896540) to dev "naa.6001405140fdf4400028000000000000" failed on path "vmhba65:C1:T4:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x8 0x0.
2020-04-24T04:10:24.390Z cpu11:66379)WARNING: NMP: nmpCompleteRetryForPath:387: Logical device "naa.6001405140fdf4400028000000000000": awaiting fast path state update before retrying failed command again...
2020-04-24T04:10:25.391Z cpu7:66162)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405140fdf4400028000000000000" - issuing command 0x439d00896540
2020-04-24T04:10:25.392Z cpu11:66379)WARNING: NMP: nmpCompleteRetryForPath:357: Retry cmd 0x83 (0x439d00896540) to dev "naa.6001405140fdf4400028000000000000" failed on path "vmhba65:C1:T4:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x8 0x0.
2020-04-24T04:10:25.392Z cpu11:66379)WARNING: NMP: nmpCompleteRetryForPath:387: Logical device "naa.6001405140fdf4400028000000000000": awaiting fast path state update before retrying failed command again...
2020-04-24T04:10:26.390Z cpu10:11034352)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405140fdf4400028000000000000" - issuing command 0x439d00896540
2020-04-24T04:10:26.390Z cpu11:66379)WARNING: NMP: nmpCompleteRetryForPath:357: Retry cmd 0x83 (0x439d00896540) to dev "naa.6001405140fdf4400028000000000000" failed on path "vmhba65:C1:T4:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x8 0x0.
2020-04-24T04:10:26.390Z cpu11:66379)WARNING: NMP: nmpCompleteRetryForPath:387: Logical device "naa.6001405140fdf4400028000000000000": awaiting fast path state update before retrying failed command again...
2020-04-24T04:10:27.025Z cpu16:65593)ScsiDeviceIO: 3015: Cmd(0x43950e1e7000) 0x89, CmdSN 0x12e361 from world 65567 to dev "naa.6001405140fdf4400028000000000000" failed H:0x5 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0.
2020-04-24T04:10:27.390Z cpu7:66162)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405140fdf4400028000000000000" - issuing command 0x439d00896540
2020-04-24T04:10:27.390Z cpu11:66379)WARNING: NMP: nmpCompleteRetryForPath:357: Retry cmd 0x83 (0x439d00896540) to dev "naa.6001405140fdf4400028000000000000" failed on path "vmhba65:C1:T4:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x8 0x0.
2020-04-24T04:10:27.390Z cpu11:66379)WARNING: NMP: nmpCompleteRetryForPath:387: Logical device "naa.6001405140fdf4400028000000000000": awaiting fast path state update before retrying failed command again...
2020-04-24T04:10:28.022Z cpu16:65593)ScsiDeviceIO: 3015: Cmd(0x439500ede7c0) 0x89, CmdSN 0x12e364 from world 65567 to dev "naa.6001405140fdf4400028000000000000" failed H:0x5 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0.
2020-04-24T04:10:28.023Z cpu0:65614)ScsiDeviceIO: 2965: Cmd(0x439d00896540) 0x83, CmdSN 0x12e35d from world 70609 to dev "naa.6001405140fdf4400028000000000000" failed H:0x8 D:0x0 P:0x0
2020-04-24T04:10:28.023Z cpu0:65614)WARNING: NMP: nmp_DeviceStartLoop:729: NMP Device "naa.6001405140fdf4400028000000000000" is blocked. Not starting I/O from device.
2020-04-24T04:10:28.023Z cpu0:65614)ScsiDeviceIO: 2965: Cmd(0x439d009e92c0) 0x83, CmdSN 0x12e35e from world 70609 to dev "naa.6001405140fdf4400028000000000000" failed H:0x8 D:0x0 P:0x0
2020-04-24T04:10:28.023Z cpu0:65614)WARNING: NMP: nmp_DeviceStartLoop:729: NMP Device "naa.6001405140fdf4400028000000000000" is blocked. Not starting I/O from device.
2020-04-24T04:10:28.023Z cpu0:65577)ScsiDeviceIO: 2965: Cmd(0x439d07a81280) 0x28, CmdSN 0x12e35f from world 0 to dev "naa.6001405140fdf4400028000000000000" failed H:0x8 D:0x0 P:0x0
2020-04-24T04:10:28.023Z cpu6:65583)ScsiDeviceIO: 2965: Cmd(0x439d00904240) 0x89, CmdSN 0x12e360 from world 66515 to dev "naa.6001405140fdf4400028000000000000" failed H:0x8 D:0x0 P:0x0
2020-04-24T04:10:28.023Z cpu6:65583)ScsiDeviceIO: 2965: Cmd(0x439500f23d40) 0x28, CmdSN 0x12e362 from world 65760 to dev "naa.6001405140fdf4400028000000000000" failed H:0x8 D:0x0 P:0x0
2020-04-24T04:10:28.023Z cpu6:65583)ScsiDeviceIO: 2965: Cmd(0x439d07b86580) 0x8a, CmdSN 0x12e363 from world 67178 to dev "naa.6001405140fdf4400028000000000000" failed H:0x8 D:0x0 P:0x0
2020-04-24T04:10:28.023Z cpu14:66515)ScsiDeviceIO: 3958: Restricting cmd 0x28 (512 bytes) from WID 66515 to quiesced dev naa.6001405140fdf4400028000000000000:1 (vmkCmd=0x439d07bc7880 cmdId: 0x430414f5ac00, 0x12e365)
2020-04-24T04:10:28.023Z cpu14:65591)ScsiDeviceIO: 2965: Cmd(0x439d07bc7880) 0x28, CmdSN 0x12e365 from world 66515 to dev "naa.6001405140fdf4400028000000000000" failed H:0x8 D:0x0 P:0x0
2020-04-24T04:10:28.035Z cpu19:69728)HBX: 2959: 'LUN1-BCACHE-TURBO-DC4': HB at offset 3383296 - Waiting for timed out HB:
2020-04-24T04:10:28.035Z cpu19:69728) [HB state abcdef02 offset 3383296 gen 5 stampUS 12831787529660 uuid 5dde9b2f-4a2ad2d6-2ee7-00a0d1eab564 jrnl <FB 42428800> drv 14.81 lockImpl 4 ip 10.0.48.47]
2020-04-24T04:10:28.390Z cpu10:11034352)WARNING: NMP: nmpDeviceAttemptFailover:603: Retry world restore device "naa.6001405140fdf4400028000000000000" - no more commands to retry
2020-04-24T04:10:28.499Z cpu6:65614)HBX: 282: 'LUN1-BCACHE-TURBO-DC4': HB at offset 3383296 - Reclaimed heartbeat [Timeout]:
2020-04-24T04:10:28.499Z cpu6:65614) [HB state abcdef02 offset 3383296 gen 5 stampUS 12831787898711 uuid 5dde9b2f-4a2ad2d6-2ee7-00a0d1eab564 jrnl <FB 42428800> drv 14.81 lockImpl 4 ip 10.0.48.47]
2020-04-24T04:10:31.092Z cpu0:66379)NMP: nmp_ThrottleLogForDevice:3630: Cmd 0x83 (0x439d0083b8c0, 70609) to dev "naa.6001405140fdf4400028000000000000" on path "vmhba65:C1:T4:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x8 0x0. Act:FAILOVER
2020-04-24T04:10:31.092Z cpu0:66379)WARNING: NMP: nmp_DeviceRetryCommand:133: Device "naa.6001405140fdf4400028000000000000": awaiting fast path state update for failover with I/O blocked. No prior reservation exists on the device.
2020-04-24T04:10:31.391Z cpu7:66162)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405140fdf4400028000000000000" - issuing command 0x439d0083b8c0
2020-04-24T04:10:31.391Z cpu0:66379)WARNING: NMP: nmpCompleteRetryForPath:357: Retry cmd 0x83 (0x439d0083b8c0) to dev "naa.6001405140fdf4400028000000000000" failed on path "vmhba65:C1:T4:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x8 0x0.
2020-04-24T04:10:31.391Z cpu0:66379)WARNING: NMP: nmpCompleteRetryForPath:387: Logical device "naa.6001405140fdf4400028000000000000": awaiting fast path state update before retrying failed command again...
2020-04-24T04:10:32.390Z cpu10:11034352)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405140fdf4400028000000000000" - issuing command 0x439d0083b8c0
2020-04-24T04:10:32.390Z cpu9:66379)WARNING: NMP: nmpCompleteRetryForPath:357: Retry cmd 0x83 (0x439d0083b8c0) to dev "naa.6001405140fdf4400028000000000000" failed on path "vmhba65:C1:T4:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x8 0x0.
2020-04-24T04:10:32.390Z cpu9:66379)WARNING: NMP: nmpCompleteRetryForPath:387: Logical device "naa.6001405140fdf4400028000000000000": awaiting fast path state update before retrying failed command again...
2020-04-24T04:10:33.391Z cpu0:66162)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405140fdf4400028000000000000" - issuing command 0x439d0083b8c0
2020-04-24T04:10:33.391Z cpu9:66379)WARNING: NMP: nmpCompleteRetryForPath:357: Retry cmd 0x83 (0x439d0083b8c0) to dev "naa.6001405140fdf4400028000000000000" failed on path "vmhba65:C1:T4:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x8 0x0.
2020-04-24T04:10:33.391Z cpu9:66379)WARNING: NMP: nmpCompleteRetryForPath:387: Logical device "naa.6001405140fdf4400028000000000000": awaiting fast path state update before retrying failed command again...
2020-04-24T04:10:34.391Z cpu10:11034352)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405140fdf4400028000000000000" - issuing command 0x439d0083b8c0
2020-04-24T04:10:34.391Z cpu9:66379)WARNING: NMP: nmpCompleteRetryForPath:357: Retry cmd 0x83 (0x439d0083b8c0) to dev "naa.6001405140fdf4400028000000000000" failed on path "vmhba65:C1:T4:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x8 0x0.
2020-04-24T04:10:34.391Z cpu9:66379)WARNING: NMP: nmpCompleteRetryForPath:387: Logical device "naa.6001405140fdf4400028000000000000": awaiting fast path state update before retrying failed command again...
2020-04-24T04:10:35.393Z cpu0:66162)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405140fdf4400028000000000000" - issuing command 0x439d0083b8c0
2020-04-24T04:10:35.393Z cpu1:66379)WARNING: NMP: nmpCompleteRetryForPath:357: Retry cmd 0x83 (0x439d0083b8c0) to dev "naa.6001405140fdf4400028000000000000" failed on path "vmhba65:C1:T4:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x8 0x0.
2020-04-24T04:10:35.393Z cpu1:66379)WARNING: NMP: nmpCompleteRetryForPath:387: Logical device "naa.6001405140fdf4400028000000000000": awaiting fast path state update before retrying failed command again...
2020-04-24T04:10:36.390Z cpu10:11034352)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405140fdf4400028000000000000" - issuing command 0x439d0083b8c0
2020-04-24T04:10:36.390Z cpu9:66379)WARNING: NMP: nmpCompleteRetryForPath:357: Retry cmd 0x83 (0x439d0083b8c0) to dev "naa.6001405140fdf4400028000000000000" failed on path "vmhba65:C1:T4:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x8 0x0.
2020-04-24T04:10:36.390Z cpu9:66379)WARNING: NMP: nmpCompleteRetryForPath:387: Logical device "naa.6001405140fdf4400028000000000000": awaiting fast path state update before retrying failed command again...
2020-04-24T04:10:37.390Z cpu0:66162)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405140fdf4400028000000000000" - issuing command 0x439d0083b8c0
2020-04-24T04:10:37.391Z cpu9:66379)WARNING: NMP: nmpCompleteRetryForPath:357: Retry cmd 0x83 (0x439d0083b8c0) to dev "naa.6001405140fdf4400028000000000000" failed on path "vmhba65:C1:T4:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x8 0x0.
2020-04-24T04:10:37.391Z cpu9:66379)WARNING: NMP: nmpCompleteRetryForPath:387: Logical device "naa.6001405140fdf4400028000000000000": awaiting fast path state update before retrying failed command again...
2020-04-24T04:10:38.390Z cpu5:11034352)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405140fdf4400028000000000000" - issuing command 0x439d0083b8c0
2020-04-24T04:10:38.390Z cpu9:66379)WARNING: NMP: nmpCompleteRetryForPath:357: Retry cmd 0x83 (0x439d0083b8c0) to dev "naa.6001405140fdf4400028000000000000" failed on path "vmhba65:C1:T4:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x8 0x0.
2020-04-24T04:10:38.390Z cpu9:66379)WARNING: NMP: nmpCompleteRetryForPath:387: Logical device "naa.6001405140fdf4400028000000000000": awaiting fast path state update before retrying failed command again...
2020-04-24T04:10:39.390Z cpu0:66162)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405140fdf4400028000000000000" - issuing command 0x439d0083b8c0
2020-04-24T04:10:39.390Z cpu9:66379)WARNING: NMP: nmpCompleteRetryForPath:357: Retry cmd 0x83 (0x439d0083b8c0) to dev "naa.6001405140fdf4400028000000000000" failed on path "vmhba65:C1:T4:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x8 0x0.
2020-04-24T04:10:39.390Z cpu9:66379)WARNING: NMP: nmpCompleteRetryForPath:387: Logical device "naa.6001405140fdf4400028000000000000": awaiting fast path state update before retrying failed command again...
2020-04-24T04:10:40.391Z cpu0:11034352)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405140fdf4400028000000000000" - issuing command 0x439d0083b8c0
2020-04-24T04:10:40.391Z cpu9:66379)WARNING: NMP: nmpCompleteRetryForPath:357: Retry cmd 0x83 (0x439d0083b8c0) to dev "naa.6001405140fdf4400028000000000000" failed on path "vmhba65:C1:T4:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x8 0x0.
2020-04-24T04:10:40.391Z cpu9:66379)WARNING: NMP: nmpCompleteRetryForPath:387: Logical device "naa.6001405140fdf4400028000000000000": awaiting fast path state update before retrying failed command again...
2020-04-24T04:10:41.390Z cpu1:66162)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405140fdf4400028000000000000" - issuing command 0x439d0083b8c0
2020-04-24T04:10:41.390Z cpu9:66379)WARNING: NMP: nmpCompleteRetryForPath:357: Retry cmd 0x83 (0x439d0083b8c0) to dev "naa.6001405140fdf4400028000000000000" failed on path "vmhba65:C1:T4:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x8 0x0.
2020-04-24T04:10:41.390Z cpu9:66379)WARNING: NMP: nmpCompleteRetryForPath:387: Logical device "naa.6001405140fdf4400028000000000000": awaiting fast path state update before retrying failed command again...
2020-04-24T04:10:42.027Z cpu19:65596)ScsiDeviceIO: 3015: Cmd(0x43950e1dfc80) 0x89, CmdSN 0x12e43d from world 65567 to dev "naa.6001405140fdf4400028000000000000" failed H:0x5 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0.
2020-04-24T04:10:42.390Z cpu0:11034352)WARNING: NMP: nmpDeviceAttemptFailover:640: Retry world failover device "naa.6001405140fdf4400028000000000000" - issuing command 0x439d0083b8c0
2020-04-24T04:10:42.390Z cpu8:66379)WARNING: NMP: nmpCompleteRetryForPath:357: Retry cmd 0x83 (0x439d0083b8c0) to dev "naa.6001405140fdf4400028000000000000" failed on path "vmhba65:C1:T4:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x8 0x0.
2020-04-24T04:10:42.390Z cpu8:66379)WARNING: NMP: nmpCompleteRetryForPath:387: Logical device "naa.6001405140fdf4400028000000000000": awaiting fast path state update before retrying failed command again...
2020-04-24T04:10:43.022Z cpu22:65599)ScsiDeviceIO: 3015: Cmd(0x439500ea0240) 0x89, CmdSN 0x12e441 from world 65567 to dev "naa.6001405140fdf4400028000000000000" failed H:0x5 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0.
2020-04-24T04:10:43.024Z cpu1:65615)ScsiDeviceIO: 2965: Cmd(0x439d0083b8c0) 0x83, CmdSN 0x12e438 from world 70609 to dev "naa.6001405140fdf4400028000000000000" failed H:0x8 D:0x0 P:0x0
2020-04-24T04:10:43.024Z cpu1:65615)WARNING: NMP: nmp_DeviceStartLoop:729: NMP Device "naa.6001405140fdf4400028000000000000" is blocked. Not starting I/O from device.
2020-04-24T04:10:43.024Z cpu1:65615)ScsiDeviceIO: 2965: Cmd(0x439d00855940) 0x83, CmdSN 0x12e439 from world 70609 to dev "naa.6001405140fdf4400028000000000000" failed H:0x8 D:0x0 P:0x0
2020-04-24T04:10:43.024Z cpu1:65615)WARNING: NMP: nmp_DeviceStartLoop:729: NMP Device "naa.6001405140fdf4400028000000000000" is blocked. Not starting I/O from device.
2020-04-24T04:10:43.024Z cpu22:11034351)FS3DM: 2203: status Busy copying 1 extents between two files, bytesTransferred = 0 extentsTransferred: 0
2020-04-24T04:10:43.024Z cpu22:11034351)SVM: 2743: scsi0:0 Completed copy in 287045 ms. vmmLeaderID = 70609.
2020-04-24T04:10:43.024Z cpu1:65578)ScsiDeviceIO: 2965: Cmd(0x439d00810740) 0x28, CmdSN 0x12e43b from world 11034091 to dev "naa.6001405140fdf4400028000000000000" failed H:0x8 D:0x0 P:0x0
2020-04-24T04:10:43.024Z cpu1:65578)ScsiDeviceIO: 2965: Cmd(0x439d0080c140) 0x89, CmdSN 0x12e43c from world 66523 to dev "naa.6001405140fdf4400028000000000000" failed H:0x8 D:0x0 P:0x0
2020-04-24T04:10:43.024Z cpu2:70674)SVM: 2456: SVM Mirrored mode IO stats for device: 50379b0f-800b171d-svmmirror
2020-04-24T04:10:43.024Z cpu2:70674)SVM: 2460: Total # IOs mirrored: 94, Total # IOs sent only to source: 77, Total # IO deferred by lock: 8981
2020-04-24T04:10:43.024Z cpu2:70674)SVM: 2464: Deferred IO stats - Max: 91, Total: 8981, Avg: 1 (msec)
2020-04-24T04:10:43.024Z cpu2:70674)SVM: 2478: Destroyed device 50379b0f-800b171d-svmmirror
2020-04-24T04:10:43.024Z cpu1:65578)ScsiDeviceIO: 2965: Cmd(0x439d00969340) 0x8a, CmdSN 0x12e43e from world 67178 to dev "naa.6001405140fdf4400028000000000000" failed H:0x8 D:0x0 P:0x0
2020-04-24T04:10:43.024Z cpu1:65578)ScsiDeviceIO: 2965: Cmd(0x439d00877b40) 0x28, CmdSN 0x12e43f from world 67159 to dev "naa.6001405140fdf4400028000000000000" failed H:0x8 D:0x0 P:0x0
2020-04-24T04:10:43.024Z cpu1:65578)ScsiDeviceIO: 2965: Cmd(0x43950ecbb800) 0x2a, CmdSN 0x12e440 from world 65567 to dev "naa.6001405140fdf4400028000000000000" failed H:0x8 D:0x0 P:0x0
2020-04-24T04:10:43.025Z cpu2:70674)CBT: 722: Disconnecting the cbt device 50379b0f-cbt with filehandle 1345821455
2020-04-24T04:10:43.052Z cpu22:69739)HBX: 2959: 'LUN1-BCACHE-TURBO-DC4': HB at offset 3383296 - Waiting for timed out HB:
2020-04-24T04:10:43.052Z cpu22:69739) [HB state abcdef02 offset 3383296 gen 5 stampUS 12831802530006 uuid 5dde9b2f-4a2ad2d6-2ee7-00a0d1eab564 jrnl <FB 42428800> drv 14.81 lockImpl 4 ip 10.0.48.47]
2020-04-24T04:10:43.055Z cpu2:70674)SVM: 2456: SVM Mirrored mode IO stats for device: 5e969b10-92e1171f-svmmirror
2020-04-24T04:10:43.055Z cpu2:70674)SVM: 2460: Total # IOs mirrored: 0, Total # IOs sent only to source: 6, Total # IO deferred by lock: 0
2020-04-24T04:10:43.055Z cpu2:70674)SVM: 2464: Deferred IO stats - Max: 0, Total: 0, Avg: 1 (msec)
2020-04-24T04:10:43.055Z cpu2:70674)SVM: 2478: Destroyed device 5e969b10-92e1171f-svmmirror
2020-04-24T04:10:43.060Z cpu2:70674)SVM: 2456: SVM Mirrored mode IO stats for device: 63759b12-76c41722-svmmirror
2020-04-24T04:10:43.060Z cpu2:70674)SVM: 2460: Total # IOs mirrored: 0, Total # IOs sent only to source: 0, Total # IO deferred by lock: 0
2020-04-24T04:10:43.060Z cpu2:70674)SVM: 2464: Deferred IO stats - Max: 0, Total: 0, Avg: 1 (msec)
2020-04-24T04:10:43.060Z cpu2:70674)SVM: 2478: Destroyed device 63759b12-76c41722-svmmirror
2020-04-24T04:10:43.060Z cpu0:10884647)HBX: 2959: 'LUN1-BCACHE-TURBO-DC4': HB at offset 3383296 - Waiting for timed out HB:
2020-04-24T04:10:43.060Z cpu0:10884647) [HB state abcdef02 offset 3383296 gen 5 stampUS 12831802530006 uuid 5dde9b2f-4a2ad2d6-2ee7-00a0d1eab564 jrnl <FB 42428800> drv 14.81 lockImpl 4 ip 10.0.48.47]
2020-04-24T04:10:43.157Z cpu8:70674)CBT: 1341: Created device 5eca9b27-cbt for cbt driver with filehandle 1590336295
2020-04-24T04:10:43.157Z cpu8:70674)WARNING: CBT: 1133: Unsupported ioctl 63
2020-04-24T04:10:43.157Z cpu8:70674)WARNING: CBT: 1133: Unsupported ioctl 62
2020-04-24T04:10:43.157Z cpu8:70674)VSCSI: 3801: handle 13669(vscsi0:0):Creating Virtual Device for world 70609 (FSS handle 1393335081) numBlocks=20971520 (bs=512)
2020-04-24T04:10:43.157Z cpu8:70674)VSCSI: 273: handle 13669(vscsi0:0):Input values: res=0 limit=300 bw=-1 Shares=1000
2020-04-24T04:10:43.157Z cpu8:70674)Vmxnet3: 17336: Disable Rx queuing; queue size 256 is larger than Vmxnet3RxQueueLimit limit of 64.
2020-04-24T04:10:43.157Z cpu8:70674)Vmxnet3: 17695: Using default queue delivery for vmxnet3 for port 0x3000014
2020-04-24T04:10:43.157Z cpu8:70674)NetPort: 3208: resuming traffic on DV port 2477
2020-04-24T04:10:43.157Z cpu8:70674)Team.etherswitch: TeamESPolicySet:5996: Port 0x3000014 frp numUplinks 2 active 2(max 2) standby 0
2020-04-24T04:10:43.157Z cpu8:70674)Team.etherswitch: TeamESPolicySet:6004: Update: Port 0x3000014 frp numUplinks 2 active 2(max 2) standby 0
2020-04-24T04:10:43.157Z cpu8:70674)NetPort: 1662: enabled port 0x3000014 with mac 00:50:56:0a:02:1f
2020-04-24T04:10:43.158Z cpu8:70674)Vmxnet3: 17336: Disable Rx queuing; queue size 256 is larger than Vmxnet3RxQueueLimit limit of 64.
2020-04-24T04:10:43.158Z cpu8:70674)Vmxnet3: 17695: Using default queue delivery for vmxnet3 for port 0x3000013
2020-04-24T04:10:43.158Z cpu8:70674)NetPort: 3208: resuming traffic on DV port 2535
2020-04-24T04:10:43.158Z cpu8:70674)Team.etherswitch: TeamESPolicySet:5996: Port 0x3000013 frp numUplinks 2 active 2(max 2) standby 0
2020-04-24T04:10:43.158Z cpu8:70674)Team.etherswitch: TeamESPolicySet:6004: Update: Port 0x3000013 frp numUplinks 2 active 2(max 2) standby 0
2020-04-24T04:10:43.158Z cpu8:70674)NetPort: 1662: enabled port 0x3000013 with mac 00:50:56:0a:01:f5
2020-04-24T04:10:43.164Z cpu2:70677)Vmxnet3: 15952: There is still packet not transmitted when the device isdisabled, port:0x3000014, queue: 0
2020-04-24T04:10:43.165Z cpu2:70677)NetPort: 1881: disabled port 0x3000014
2020-04-24T04:10:43.165Z cpu2:70677)Vmxnet3: 17336: Disable Rx queuing; queue size 256 is larger than Vmxnet3RxQueueLimit limit of 64.
2020-04-24T04:10:43.165Z cpu2:70677)Vmxnet3: 17695: Using default queue delivery for vmxnet3 for port 0x3000014
2020-04-24T04:10:43.165Z cpu2:70677)NetPort: 3208: resuming traffic on DV port 2477
2020-04-24T04:10:43.165Z cpu2:70677)Team.etherswitch: TeamESPolicySet:5996: Port 0x3000014 frp numUplinks 2 active 2(max 2) standby 0
2020-04-24T04:10:43.165Z cpu2:70677)Team.etherswitch: TeamESPolicySet:6004: Update: Port 0x3000014 frp numUplinks 2 active 2(max 2) standby 0
2020-04-24T04:10:43.165Z cpu2:70677)NetPort: 1662: enabled port 0x3000014 with mac 00:50:56:0a:02:1f
2020-04-24T04:10:43.392Z cpu0:66162)WARNING: NMP: nmpDeviceAttemptFailover:603: Retry world restore device "naa.6001405140fdf4400028000000000000" - no more commands to retry
2020-04-24T04:10:43.411Z cpu0:65615)HBX: 282: 'LUN1-BCACHE-TURBO-DC4': HB at offset 3383296 - Reclaimed heartbeat [Timeout]:
2020-04-24T04:10:43.411Z cpu0:65615) [HB state abcdef02 offset 3383296 gen 5 stampUS 12831802900615 uuid 5dde9b2f-4a2ad2d6-2ee7-00a0d1eab564 jrnl <FB 42428800> drv 14.81 lockImpl 4 ip 10.0.48.47]
2020-04-24T04:10:47.749Z cpu2:68036)Config: 706: "SIOControlFlag2" = 0, Old Value: 1, (Status: 0x0)
2020-04-24T04:11:19.244Z cpu13:67178)DLX: 4312: vol 'LUN3-PSAN-DC4', lock at 264343552: [Req mode 1] Checking liveness:
2020-04-24T04:11:19.244Z cpu13:67178)[type 10c00001 offset 264343552 v 40335820, hb offset 3387392
gen 99, mode 1, owner 5e612b8e-f3947fdb-bd79-0025b5020b2f mtime 2666986
num 0 gblnum 0 gblgen 0 gblbrk 0]
2020-04-24T04:11:19.390Z cpu12:66161)NMP: nmp_ResetDeviceLogThrottling:3458: last error status from device naa.6001405140fdf4400028000000000000 repeated 1 times

 

ISCSI Server kern.log:

Apr 24 10:59:26 dell4-n3-psn3 kernel: [ 615.767489] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 10:59:26 dell4-n3-psn3 kernel: [ 615.767493] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 10:59:26 dell4-n3-psn3 kernel: [ 615.767495] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 10:59:26 dell4-n3-psn3 kernel: [ 615.767497] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 10:59:26 dell4-n3-psn3 kernel: [ 615.767498] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 10:59:26 dell4-n3-psn3 kernel: [ 615.767501] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 10:59:26 dell4-n3-psn3 kernel: [ 615.767502] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 10:59:26 dell4-n3-psn3 kernel: [ 615.767504] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 10:59:26 dell4-n3-psn3 kernel: [ 615.767506] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 10:59:26 dell4-n3-psn3 kernel: [ 615.767508] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 10:59:32 dell4-n3-psn3 kernel: [ 621.132863] target_xcopy_do_work: 11 callbacks suppressed
Apr 24 10:59:32 dell4-n3-psn3 kernel: [ 621.132865] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 10:59:33 dell4-n3-psn3 kernel: [ 622.133313] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 10:59:34 dell4-n3-psn3 kernel: [ 623.132922] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 10:59:35 dell4-n3-psn3 kernel: [ 624.133860] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 10:59:36 dell4-n3-psn3 kernel: [ 625.133531] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 10:59:37 dell4-n3-psn3 kernel: [ 626.132656] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 10:59:38 dell4-n3-psn3 kernel: [ 627.133535] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 10:59:41 dell4-n3-psn3 kernel: [ 630.767232] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 10:59:41 dell4-n3-psn3 kernel: [ 630.767236] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 10:59:41 dell4-n3-psn3 kernel: [ 630.767238] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 10:59:41 dell4-n3-psn3 kernel: [ 630.767240] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 10:59:41 dell4-n3-psn3 kernel: [ 630.767242] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 10:59:41 dell4-n3-psn3 kernel: [ 630.767244] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 10:59:41 dell4-n3-psn3 kernel: [ 630.767245] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 10:59:41 dell4-n3-psn3 kernel: [ 630.767247] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 10:59:41 dell4-n3-psn3 kernel: [ 630.767249] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 10:59:44 dell4-n3-psn3 kernel: [ 633.132796] target_xcopy_do_work: 9 callbacks suppressed
Apr 24 10:59:44 dell4-n3-psn3 kernel: [ 633.132798] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 10:59:45 dell4-n3-psn3 kernel: [ 634.132952] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 10:59:46 dell4-n3-psn3 kernel: [ 635.132959] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 10:59:47 dell4-n3-psn3 kernel: [ 636.132863] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 10:59:48 dell4-n3-psn3 kernel: [ 637.132883] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 10:59:49 dell4-n3-psn3 kernel: [ 638.135971] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 10:59:50 dell4-n3-psn3 kernel: [ 639.133028] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 10:59:51 dell4-n3-psn3 kernel: [ 640.132696] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 10:59:52 dell4-n3-psn3 kernel: [ 641.132643] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 10:59:53 dell4-n3-psn3 kernel: [ 642.132842] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 10:59:56 dell4-n3-psn3 kernel: [ 645.888191] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 10:59:56 dell4-n3-psn3 kernel: [ 645.888195] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 10:59:56 dell4-n3-psn3 kernel: [ 645.888197] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 10:59:56 dell4-n3-psn3 kernel: [ 645.888199] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 10:59:56 dell4-n3-psn3 kernel: [ 645.888200] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 10:59:56 dell4-n3-psn3 kernel: [ 645.888202] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 10:59:56 dell4-n3-psn3 kernel: [ 645.888204] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 10:59:56 dell4-n3-psn3 kernel: [ 645.888206] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 10:59:56 dell4-n3-psn3 kernel: [ 645.888208] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 10:59:56 dell4-n3-psn3 kernel: [ 645.888210] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 11:00:02 dell4-n3-psn3 kernel: [ 651.132902] target_xcopy_do_work: 11 callbacks suppressed
Apr 24 11:00:02 dell4-n3-psn3 kernel: [ 651.132904] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 11:00:03 dell4-n3-psn3 kernel: [ 652.133276] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 11:00:04 dell4-n3-psn3 kernel: [ 653.133090] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 11:00:05 dell4-n3-psn3 kernel: [ 654.133005] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 11:00:06 dell4-n3-psn3 kernel: [ 655.135560] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 11:00:07 dell4-n3-psn3 kernel: [ 656.135373] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 11:00:08 dell4-n3-psn3 kernel: [ 657.134507] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 11:00:09 dell4-n3-psn3 kernel: [ 658.519681] rbd: rbd0: compare-and-write 200 at fa83000 (283000)
Apr 24 11:00:09 dell4-n3-psn3 kernel: [ 658.519684] rbd: rbd0: result -4151 xferred 200
Apr 24 11:00:09 dell4-n3-psn3 kernel: [ 658.519685] COMPARE_AND_WRITE: miscompare at offset 0
Apr 24 11:00:11 dell4-n3-psn3 kernel: [ 660.828539] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 11:00:11 dell4-n3-psn3 kernel: [ 660.828543] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 11:00:11 dell4-n3-psn3 kernel: [ 660.828545] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 11:00:11 dell4-n3-psn3 kernel: [ 660.828546] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 11:00:11 dell4-n3-psn3 kernel: [ 660.828645] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 11:00:11 dell4-n3-psn3 kernel: [ 660.828648] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 11:00:11 dell4-n3-psn3 kernel: [ 660.828650] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 11:00:11 dell4-n3-psn3 kernel: [ 660.828651] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 11:00:13 dell4-n3-psn3 kernel: [ 662.135321] target_xcopy_do_work: 9 callbacks suppressed
Apr 24 11:00:13 dell4-n3-psn3 kernel: [ 662.135322] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 11:00:14 dell4-n3-psn3 kernel: [ 663.133565] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 11:00:15 dell4-n3-psn3 kernel: [ 664.138092] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 11:00:16 dell4-n3-psn3 kernel: [ 665.132976] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 11:00:17 dell4-n3-psn3 kernel: [ 666.134118] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 11:00:18 dell4-n3-psn3 kernel: [ 667.133120] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 11:00:19 dell4-n3-psn3 kernel: [ 668.133089] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 11:00:20 dell4-n3-psn3 kernel: [ 669.134720] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 11:00:21 dell4-n3-psn3 kernel: [ 670.133297] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 11:00:22 dell4-n3-psn3 kernel: [ 671.133046] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 11:00:23 dell4-n3-psn3 kernel: [ 672.132869] target_xcopy_do_work: rc: 0, Setting X-COPY CHECK_CONDITION -> sending response
Apr 24 11:00:24 dell4-n3-psn3 kernel: [ 673.320098] rbd: rbd0: compare-and-write 200 at 31f6400 (1f6400)
Apr 24 11:00:24 dell4-n3-psn3 kernel: [ 673.320101] rbd: rbd0: result -4151 xferred 200
Apr 24 11:00:24 dell4-n3-psn3 kernel: [ 673.320102] COMPARE_AND_WRITE: miscompare at offset 0
Apr 24 11:00:24 dell4-n3-psn3 kernel: [ 673.495235] rbd: rbd0: compare-and-write 200 at 31f6000 (1f6000)
Apr 24 11:00:24 dell4-n3-psn3 kernel: [ 673.495239] rbd: rbd0: result -4151 xferred 200
Apr 24 11:00:24 dell4-n3-psn3 kernel: [ 673.495240] COMPARE_AND_WRITE: miscompare at offset 0
Apr 24 11:00:24 dell4-n3-psn3 kernel: [ 673.562526] rbd: rbd0: compare-and-write 200 at ff53000 (353000)
Apr 24 11:00:24 dell4-n3-psn3 kernel: [ 673.562529] rbd: rbd0: result -4151 xferred 200
Apr 24 11:00:24 dell4-n3-psn3 kernel: [ 673.562533] COMPARE_AND_WRITE: miscompare at offset 0
Apr 24 11:00:25 dell4-n3-psn3 kernel: [ 674.674501] rbd: rbd0: compare-and-write 200 at 7024000 (24000)
Apr 24 11:00:25 dell4-n3-psn3 kernel: [ 674.674505] rbd: rbd0: result -4108 xferred 200
Apr 24 11:00:25 dell4-n3-psn3 kernel: [ 674.674506] COMPARE_AND_WRITE: miscompare at offset 0
Apr 24 11:00:41 dell4-n3-psn3 kernel: [ 690.614083] rbd: rbd0: compare-and-write 200 at 7024000 (24000)
Apr 24 11:00:41 dell4-n3-psn3 kernel: [ 690.614085] rbd: rbd0: result -4108 xferred 200
Apr 24 11:00:41 dell4-n3-psn3 kernel: [ 690.614086] COMPARE_AND_WRITE: miscompare at offset 0
Apr 24 11:01:15 dell4-n3-psn3 kernel: [ 724.532836] iSCSI/iqn.1998-01.com.vmware:5cdb81f0-0321-c5d4-4f83-00a0d1eabf08-58df7f54: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
Apr 24 11:01:15 dell4-n3-psn3 kernel: [ 724.532991] iSCSI/iqn.1998-01.com.vmware:5cdb81f0-0321-c5d4-4f83-00a0d1eabf08-58df7f54: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
Apr 24 11:01:39 dell4-n3-psn3 kernel: [ 748.174762] iSCSI/iqn.1998-01.com.vmware:sm-bld15-18-79: Unsupported SCSI Opcode 0x4d, sending CHECK_CONDITION.
Apr 24 11:01:48 dell4-n3-psn3 kernel: [ 757.463140] iSCSI/iqn.1998-01.com.vmware:bld3-18-133: Unsupported SCSI Opcode 0x4d, sending CHECK_CONDITION.
Apr 24 11:01:52 dell4-n3-psn3 kernel: [ 761.159024] iSCSI/iqn.1998-01.com.vmware:bld2-18-134: Unsupported SCSI Opcode 0x4d, sending CHECK_CONDITION.
Apr 24 11:02:11 dell4-n3-psn3 kernel: [ 780.494682] rbd: rbd0: compare-and-write 200 at 7024000 (24000)
Apr 24 11:02:11 dell4-n3-psn3 kernel: [ 780.494685] rbd: rbd0: result -4108 xferred 200
Apr 24 11:02:11 dell4-n3-psn3 kernel: [ 780.494687] COMPARE_AND_WRITE: miscompare at offset 0
Apr 24 11:02:25 dell4-n3-psn3 kernel: [ 794.780962] iSCSI/iqn.1998-01.com.vmware:5d4ce620-6323-11e2-9fc8-0025902c5602-0af40d15: Unsupported SCSI Opcode 0x4d, sending CHECK_CONDITION.
Apr 24 11:02:27 dell4-n3-psn3 kernel: [ 796.162315] iSCSI/iqn.1998-01.com.vmware:5d4e318e-3e32-1990-ebec-00266cfb4bec-2c061885: Unsupported SCSI Opcode 0x4d, sending CHECK_CONDITION.
Apr 24 11:02:30 dell4-n3-psn3 kernel: [ 799.018523] iSCSI/iqn.1998-01.com.vmware:5d4ce19a-1f04-da88-09a8-0025902c5602-0e778a45: Unsupported SCSI Opcode 0x4d, sending CHECK_CONDITION.
Apr 24 11:02:33 dell4-n3-psn3 kernel: [ 802.170186] iSCSI/iqn.1998-01.com.vmware:5cdb81f0-0321-c5d4-4f83-00a0d1eabf08-58df7f54: Unsupported SCSI Opcode 0x4d, sending CHECK_CONDITION.
Apr 24 11:02:34 dell4-n3-psn3 kernel: [ 803.641207] iSCSI/iqn.1998-01.com.vmware:5dad17af-58cb-2b60-73eb-00266cffdc54-258a5808: Unsupported SCSI Opcode 0x4d, sending CHECK_CONDITION.
Apr 24 11:02:35 dell4-n3-psn3 kernel: [ 804.054206] iSCSI/iqn.1998-01.com.vmware:5cdb74d8-4e11-014c-f30a-008cfa04684c-0d0d492c: Unsupported SCSI Opcode 0x4d, sending CHECK_CONDITION.
Apr 24 11:02:37 dell4-n3-psn3 kernel: [ 806.308144] rbd: rbd0: compare-and-write 200 at 7024000 (24000)
Apr 24 11:02:37 dell4-n3-psn3 kernel: [ 806.308146] rbd: rbd0: result -4108 xferred 200
Apr 24 11:02:37 dell4-n3-psn3 kernel: [ 806.308150] COMPARE_AND_WRITE: miscompare at offset 0
Apr 24 11:02:37 dell4-n3-psn3 kernel: [ 806.483093] iSCSI/iqn.1998-01.com.vmware:5cdb84d6-5c2c-3d47-985d-00a0d1eab3ec-4ffcc033: Unsupported SCSI Opcode 0x4d, sending CHECK_CONDITION.
Apr 24 11:02:37 dell4-n3-psn3 kernel: [ 806.517129] rbd: rbd0: compare-and-write 200 at 7024000 (24000)
Apr 24 11:02:37 dell4-n3-psn3 kernel: [ 806.517132] rbd: rbd0: result -4108 xferred 200
Apr 24 11:02:37 dell4-n3-psn3 kernel: [ 806.517144] COMPARE_AND_WRITE: miscompare at offset 0
Apr 24 11:02:37 dell4-n3-psn3 kernel: [ 806.559025] rbd: rbd0: compare-and-write 200 at 7024000 (24000)
Apr 24 11:02:37 dell4-n3-psn3 kernel: [ 806.559027] rbd: rbd0: result -4108 xferred 200
Apr 24 11:02:37 dell4-n3-psn3 kernel: [ 806.559029] COMPARE_AND_WRITE: miscompare at offset 0
Apr 24 11:02:37 dell4-n3-psn3 kernel: [ 806.559103] rbd: rbd0: compare-and-write 200 at 7024000 (24000)
Apr 24 11:02:37 dell4-n3-psn3 kernel: [ 806.559105] rbd: rbd0: result -4108 xferred 200
Apr 24 11:02:37 dell4-n3-psn3 kernel: [ 806.559109] COMPARE_AND_WRITE: miscompare at offset 0
Apr 24 11:02:39 dell4-n3-psn3 kernel: [ 808.440126] iSCSI/iqn.1998-01.com.vmware:5cdb8de8-ff5b-9bdc-ad0e-00a0d1eb20a4-1223e026: Unsupported SCSI Opcode 0x4d, sending CHECK_CONDITION.
Apr 24 11:02:40 dell4-n3-psn3 kernel: [ 809.000684] iSCSI/iqn.1998-01.com.vmware:5cdbdbe6-4270-27d6-d332-00a0d1ec082c-5879ba60: Unsupported SCSI Opcode 0x4d, sending CHECK_CONDITION.
Apr 24 11:02:41 dell4-n3-psn3 kernel: [ 810.581262] rbd: rbd0: compare-and-write 200 at 7024000 (24000)
Apr 24 11:02:41 dell4-n3-psn3 kernel: [ 810.581266] rbd: rbd0: result -4108 xferred 200
Apr 24 11:02:41 dell4-n3-psn3 kernel: [ 810.581268] COMPARE_AND_WRITE: miscompare at offset 0
Apr 24 11:02:49 dell4-n3-psn3 kernel: [ 818.809266] rbd: rbd0: compare-and-write 200 at 7024000 (24000)
Apr 24 11:02:49 dell4-n3-psn3 kernel: [ 818.809269] rbd: rbd0: result -4108 xferred 200
Apr 24 11:02:49 dell4-n3-psn3 kernel: [ 818.809270] COMPARE_AND_WRITE: miscompare at offset

PING from ESXi to ISCSI Server :

[root@localhost:~] vmkping -I vmk4 -s 8972 10.0.94.27 -c 50
PING 10.0.94.27 (10.0.94.27): 8972 data bytes
8980 bytes from 10.0.94.27: icmp_seq=0 ttl=64 time=0.320 ms
8980 bytes from 10.0.94.27: icmp_seq=1 ttl=64 time=0.266 ms
8980 bytes from 10.0.94.27: icmp_seq=2 ttl=64 time=0.303 ms
8980 bytes from 10.0.94.27: icmp_seq=3 ttl=64 time=0.368 ms
8980 bytes from 10.0.94.27: icmp_seq=4 ttl=64 time=0.234 ms
8980 bytes from 10.0.94.27: icmp_seq=5 ttl=64 time=0.271 ms
8980 bytes from 10.0.94.27: icmp_seq=6 ttl=64 time=0.263 ms
8980 bytes from 10.0.94.27: icmp_seq=7 ttl=64 time=0.275 ms
8980 bytes from 10.0.94.27: icmp_seq=8 ttl=64 time=0.231 ms
8980 bytes from 10.0.94.27: icmp_seq=9 ttl=64 time=0.329 ms
8980 bytes from 10.0.94.27: icmp_seq=10 ttl=64 time=0.317 ms
8980 bytes from 10.0.94.27: icmp_seq=11 ttl=64 time=0.306 ms
8980 bytes from 10.0.94.27: icmp_seq=12 ttl=64 time=0.286 ms
8980 bytes from 10.0.94.27: icmp_seq=13 ttl=64 time=0.473 ms
8980 bytes from 10.0.94.27: icmp_seq=14 ttl=64 time=0.256 ms
8980 bytes from 10.0.94.27: icmp_seq=15 ttl=64 time=0.883 ms
8980 bytes from 10.0.94.27: icmp_seq=16 ttl=64 time=0.267 ms
8980 bytes from 10.0.94.27: icmp_seq=17 ttl=64 time=0.296 ms
8980 bytes from 10.0.94.27: icmp_seq=18 ttl=64 time=0.250 ms
8980 bytes from 10.0.94.27: icmp_seq=19 ttl=64 time=0.249 ms
8980 bytes from 10.0.94.27: icmp_seq=20 ttl=64 time=0.286 ms
8980 bytes from 10.0.94.27: icmp_seq=21 ttl=64 time=0.456 ms
8980 bytes from 10.0.94.27: icmp_seq=22 ttl=64 time=0.247 ms
8980 bytes from 10.0.94.27: icmp_seq=23 ttl=64 time=0.464 ms
8980 bytes from 10.0.94.27: icmp_seq=24 ttl=64 time=0.402 ms
8980 bytes from 10.0.94.27: icmp_seq=25 ttl=64 time=0.275 ms
8980 bytes from 10.0.94.27: icmp_seq=26 ttl=64 time=0.224 ms
8980 bytes from 10.0.94.27: icmp_seq=27 ttl=64 time=0.409 ms
8980 bytes from 10.0.94.27: icmp_seq=28 ttl=64 time=0.250 ms
8980 bytes from 10.0.94.27: icmp_seq=29 ttl=64 time=0.456 ms
8980 bytes from 10.0.94.27: icmp_seq=30 ttl=64 time=0.224 ms
8980 bytes from 10.0.94.27: icmp_seq=31 ttl=64 time=0.321 ms
8980 bytes from 10.0.94.27: icmp_seq=32 ttl=64 time=0.321 ms
8980 bytes from 10.0.94.27: icmp_seq=33 ttl=64 time=0.471 ms
8980 bytes from 10.0.94.27: icmp_seq=34 ttl=64 time=0.740 ms
8980 bytes from 10.0.94.27: icmp_seq=35 ttl=64 time=0.220 ms
8980 bytes from 10.0.94.27: icmp_seq=36 ttl=64 time=0.317 ms
8980 bytes from 10.0.94.27: icmp_seq=37 ttl=64 time=0.234 ms
8980 bytes from 10.0.94.27: icmp_seq=38 ttl=64 time=0.377 ms
8980 bytes from 10.0.94.27: icmp_seq=39 ttl=64 time=0.321 ms
8980 bytes from 10.0.94.27: icmp_seq=40 ttl=64 time=0.242 ms
8980 bytes from 10.0.94.27: icmp_seq=41 ttl=64 time=0.316 ms
8980 bytes from 10.0.94.27: icmp_seq=42 ttl=64 time=0.237 ms
8980 bytes from 10.0.94.27: icmp_seq=43 ttl=64 time=0.277 ms
8980 bytes from 10.0.94.27: icmp_seq=44 ttl=64 time=0.243 ms
8980 bytes from 10.0.94.27: icmp_seq=45 ttl=64 time=0.249 ms
8980 bytes from 10.0.94.27: icmp_seq=46 ttl=64 time=0.238 ms
8980 bytes from 10.0.94.27: icmp_seq=47 ttl=64 time=0.277 ms
8980 bytes from 10.0.94.27: icmp_seq=48 ttl=64 time=0.290 ms
8980 bytes from 10.0.94.27: icmp_seq=49 ttl=64 time=0.248 ms

--- 10.0.94.27 ping statistics ---
50 packets transmitted, 50 packets received, 0% packet loss
round-trip min/avg/max = 0.220/0.322/0.883 ms

Looks like iscsi doesn't write data to rbd disk.

Can you help me troubleshoot it?

Thanks.

#1

admin
2,930 Posts
April 24, 2020, 11:13 am

If you are migrating vms from PetaSAN datastore to another PetaSAN datastore, make sure both source and target iSCSI disks are served by the same hosts, if source disk paths are on hostA and hostB, destination disk paths should also be on hostA and hostB, else you use the path assignment to move paths before migration.  This is explained in the vmware guide.

Double check VAAI ATS is supported via:

esxcli storage core device vaai status get

If you one of your datastores your are live migrating from is not PetaSAN, VAAI is not used, which will slow things down, and depending on how fast/slow your hardware is and what load is currently on the hosts and vm,  things could timeout. Veeam do have a migration tool which if you already have, try it in this case, it may be better.

 

 

#2

zamasaki158
13 Posts
April 26, 2020, 3:28 am

I appreciate it. It works

#3