Forums

Home / Forums

You need to log in to create posts and topics. Login · Register

assigned node iqn is changing constantly

Pages: 1 2

fredohouse
13 Posts
February 25, 2023, 10:57 am

Hello 😉

I'm installed esxi 7U3 and configured an iscsi volume on petasan, with 2 subnet.

I can the the assigned node is changing constantly in path of the volume. The effect is esxi vmkernel.log show lot of errors.

Anyone as an idea ? Thanks a lot !

2023-02-25T10:54:06.414Z cpu3:2097965)WARNING: iscsi_vmk: iscsivmk_StartConnection:919: vmhba64:CH:0 T:0 CN:0: iSCSI connection is being marked "ONLINE"
2023-02-25T10:54:06.414Z cpu3:2097965)WARNING: iscsi_vmk: iscsivmk_StartConnection:920: Sess [ISID: 00023d000001 TARGET: iqn.2016-05.infra.vcd.isci:civavsan:00003 TPGT: 1 TSIH: 0]
2023-02-25T10:54:06.414Z cpu3:2097965)WARNING: iscsi_vmk: iscsivmk_StartConnection:921: Conn [CID: 0 L: 10.98.95.11:26077 R: 10.98.95.1:3260]
2023-02-25T10:54:07.262Z cpu19:2097363)StorageDevice: 7060: End path evaluation for device naa.6001405ce0495ed00003000000000000
2023-02-25T10:54:23.089Z cpu0:2097893)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic:481: vmhba64:CH:0 T:0 CN:0: Failed to receive data: Connection reset by peer
2023-02-25T10:54:23.089Z cpu0:2097893)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic:484: Sess [ISID: 00023d000001 TARGET: iqn.2016-05.infra.vcd.isci:civavsan:00003 TPGT: 1 TSIH: 0]
2023-02-25T10:54:23.089Z cpu0:2097893)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic:485: Conn [CID: 0 L: 10.98.95.11:26077 R: 10.98.95.1:3260]
2023-02-25T10:54:23.089Z cpu0:2097893)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1236: vmhba64:CH:0 T:0 CN:0: Connection rx notifying failure: Failed to Receive. State=Online
2023-02-25T10:54:23.089Z cpu0:2097893)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1237: Sess [ISID: 00023d000001 TARGET: iqn.2016-05.infra.vcd.isci:civavsan:00003 TPGT: 1 TSIH: 0]
2023-02-25T10:54:23.089Z cpu0:2097893)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1238: Conn [CID: 0 L: 10.98.95.11:26077 R: 10.98.95.1:3260]
2023-02-25T10:54:23.089Z cpu10:2097652)WARNING: iscsi_vmk: iscsivmk_StopConnection:738: vmhba64:CH:0 T:0 CN:0: iSCSI connection is being marked "OFFLINE" (Event:6)
2023-02-25T10:54:23.089Z cpu10:2097652)WARNING: iscsi_vmk: iscsivmk_StopConnection:739: Sess [ISID: 00023d000001 TARGET: iqn.2016-05.infra.vcd.isci:civavsan:00003 TPGT: 1 TSIH: 0]
2023-02-25T10:54:23.089Z cpu10:2097652)WARNING: iscsi_vmk: iscsivmk_StopConnection:740: Conn [CID: 0 L: 10.98.95.11:26077 R: 10.98.95.1:3260]
2023-02-25T10:54:23.103Z cpu10:2097667)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6001405ce0495ed00003000000000000" state in doubt; requested fast path state update...
2023-02-25T10:54:23.103Z cpu10:2097667)ScsiDeviceIO: 4124: Cmd(0x45b946954588) 0x2a, CmdSN 0xd6 from world 2099283 to dev "naa.6001405ce0495ed00003000000000000" failed H:0x2 D:0x0 P:0x0
2023-02-25T10:54:23.263Z cpu0:2105991)WARNING: VMW_SATP_ALUA: satp_alua_getTargetPortInfo:160: Could not get page 83 INQUIRY data for path "vmhba64:C0:T0:L0" - Transient storage condition, suggest retry (195887294)
2023-02-25T10:54:23.263Z cpu0:2105991)StorageDevice: 7060: End path evaluation for device naa.6001405ce0495ed00003000000000000
2023-02-25T10:54:23.277Z cpu10:2097667)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6001405ce0495ed00003000000000000" state in doubt; requested fast path state update...
2023-02-25T10:54:24.263Z cpu3:2102195)WARNING: VMW_SATP_ALUA: satp_alua_getTargetPortInfo:160: Could not get page 83 INQUIRY data for path "vmhba64:C0:T0:L0" - Transient storage condition, suggest retry (195887294)
2023-02-25T10:54:24.263Z cpu3:2102195)StorageDevice: 7060: End path evaluation for device naa.6001405ce0495ed00003000000000000
2023-02-25T10:54:24.265Z cpu10:2097667)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6001405ce0495ed00003000000000000" state in doubt; requested fast path state update...
2023-02-25T10:54:24.774Z cpu10:2097667)NMP: nmp_ThrottleLogForDevice:3798: last error status from device naa.6001405ce0495ed00003000000000000 repeated 640 times
2023-02-25T10:54:25.263Z cpu14:2102555)WARNING: VMW_SATP_ALUA: satp_alua_getTargetPortInfo:160: Could not get page 83 INQUIRY data for path "vmhba64:C0:T0:L0" - Transient storage condition, suggest retry (195887294)
2023-02-25T10:54:25.263Z cpu14:2102555)StorageDevice: 7060: End path evaluation for device naa.6001405ce0495ed00003000000000000
2023-02-25T10:54:25.266Z cpu10:2097667)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6001405ce0495ed00003000000000000" state in doubt; requested fast path state update...

#1

admin
2,930 Posts
February 25, 2023, 3:19 pm

from dashboard..what is the status of the cluster ?

#2

fredohouse
13 Posts
February 25, 2023, 3:27 pm

All is ok. No errors.

#3

admin
2,930 Posts
February 25, 2023, 5:01 pm

is the iqn changing or is it the ip is moving to a different node  ?

what do you see on the PetaSAN side rather than on vmware side ?  in PetaSAN ui do you see ip switching nodes or is it fixed ? do you see any errors in UI or in the PetaSAN log ?

 

#4

fredohouse
13 Posts
February 25, 2023, 7:53 pm

the ip is moving on different node. we can see in the log of the node (petasan) that the iqn is removed. on the vmware node, the volume appears degraded. when I try to format the volume, there is an error (in vmware)

Also, vmware said : "Échec de la création de la banque de données VMFS - Un paramètre spécifié était incorrect : devname"

you can see here that vmware as connectivity failure :

2023-02-25T19:49:32.186Z cpu13:2097965)WARNING: iscsi_vmk: iscsivmk_StartConnection:919: vmhba64:CH:1 T:0 CN:0: iSCSI connection is being marked "ONLINE"
2023-02-25T19:49:32.186Z cpu13:2097965)WARNING: iscsi_vmk: iscsivmk_StartConnection:920: Sess [ISID: 00023d000004 TARGET: iqn.2016-05.infra.vcd.isci:civavsan:00004 TPGT: 2 TSIH: 0]
2023-02-25T19:49:32.186Z cpu13:2097965)WARNING: iscsi_vmk: iscsivmk_StartConnection:921: Conn [CID: 0 L: 10.98.96.11:61238 R: 10.98.96.2:3260]
2023-02-25T19:49:38.261Z cpu16:2097618)NMP: nmp_ResetDeviceLogThrottling:3784: last error status from device naa.6001405ce0495ed00004000000000000 repeated 98 times
2023-02-25T19:49:48.127Z cpu11:2097901)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic:481: vmhba64:CH:1 T:0 CN:0: Failed to receive data: Connection reset by peer
2023-02-25T19:49:48.127Z cpu11:2097901)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic:484: Sess [ISID: 00023d000004 TARGET: iqn.2016-05.infra.vcd.isci:civavsan:00004 TPGT: 2 TSIH: 0]
2023-02-25T19:49:48.127Z cpu11:2097901)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic:485: Conn [CID: 0 L: 10.98.96.11:61238 R: 10.98.96.2:3260]
2023-02-25T19:49:48.127Z cpu11:2097901)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1236: vmhba64:CH:1 T:0 CN:0: Connection rx notifying failure: Failed to Receive. State=Online
2023-02-25T19:49:48.127Z cpu11:2097901)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1237: Sess [ISID: 00023d000004 TARGET: iqn.2016-05.infra.vcd.isci:civavsan:00004 TPGT: 2 TSIH: 0]
2023-02-25T19:49:48.127Z cpu11:2097901)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1238: Conn [CID: 0 L: 10.98.96.11:61238 R: 10.98.96.2:3260]
2023-02-25T19:49:48.127Z cpu11:2097652)WARNING: iscsi_vmk: iscsivmk_StopConnection:738: vmhba64:CH:1 T:0 CN:0: iSCSI connection is being marked "OFFLINE" (Event:6)
2023-02-25T19:49:48.127Z cpu11:2097652)WARNING: iscsi_vmk: iscsivmk_StopConnection:739: Sess [ISID: 00023d000004 TARGET: iqn.2016-05.infra.vcd.isci:civavsan:00004 TPGT: 2 TSIH: 0]
2023-02-25T19:49:48.127Z cpu11:2097652)WARNING: iscsi_vmk: iscsivmk_StopConnection:740: Conn [CID: 0 L: 10.98.96.11:61238 R: 10.98.96.2:3260]
2023-02-25T19:49:50.949Z cpu13:2097965)iscsi_vmk: iscsivmk_ConnNetRegister:1898: socket 0x431594036b20 network resource pool netsched.pools.persist.iscsi associated
2023-02-25T19:49:50.949Z cpu13:2097965)iscsi_vmk: iscsivmk_ConnNetRegister:1926: socket 0x431594036b20 network tracker id 934179698 tracker.iSCSI.10.98.96.2 associated
2023-02-25T19:49:51.203Z cpu13:2097965)WARNING: iscsi_vmk: iscsivmk_StartConnection:919: vmhba64:CH:1 T:0 CN:0: iSCSI connection is being marked "ONLINE"
2023-02-25T19:49:51.203Z cpu13:2097965)WARNING: iscsi_vmk: iscsivmk_StartConnection:920: Sess [ISID: 00023d000004 TARGET: iqn.2016-05.infra.vcd.isci:civavsan:00004 TPGT: 2 TSIH: 0]
2023-02-25T19:49:51.203Z cpu13:2097965)WARNING: iscsi_vmk: iscsivmk_StartConnection:921: Conn [CID: 0 L: 10.98.96.11:54337 R: 10.98.96.2:3260]

I think that is because the iqn is remove on an node, an recreated on another node. Between this time, it's seems that the iqn is not available.. but maybe I'm in false...

Thank you for your help 🙂

 

#5

admin
2,930 Posts
February 25, 2023, 8:28 pm

i am more interested in the PetaSAN.log rather than vmware log

#6

fredohouse
13 Posts
February 25, 2023, 8:51 pm

Sure I understand.

log from node 1 :

25/02/2023 21:48:25 INFO Image image-00004 unmapped successfully.

25/02/2023 21:48:25 INFO PetaSAN Cleaned rbd backstores.

25/02/2023 21:48:25 INFO LIO deleted backstore image image-00004

25/02/2023 21:48:23 INFO PetaSAN cleaned iqns.

25/02/2023 21:48:23 ERROR Cannot unmap image image-civavsan. error, no mapped images found.

25/02/2023 21:48:23 INFO LIO deleted Target iqn.2016-05.infra.vcd.isci:civavsan:00004

25/02/2023 21:48:20 INFO PetaSAN unlocked any consul locks not configured in this node.

25/02/2023 21:48:20 INFO Unlock path 00004/1

25/02/2023 21:48:20 INFO PetaSAN cleaned local paths not locked by this node in consul.

25/02/2023 21:48:20 INFO Cleaned disk path civavsan/1.

25/02/2023 21:48:20 INFO Path 00004/1 acquired successfully

25/02/2023 21:48:20 INFO LIO add_target() disk wwn is ce0495ed00004

25/02/2023 21:48:17 INFO Image image-00004 mapped successfully.

25/02/2023 21:48:17 INFO Found pool:rbd for disk:00004 via consul

25/02/2023 21:48:12 INFO Image image-00004 unmapped successfully.

log from node 2 :

25/02/2023 21:48:25 INFO Image image-00004 unmapped successfully.

25/02/2023 21:48:25 INFO PetaSAN Cleaned rbd backstores.

25/02/2023 21:48:25 INFO LIO deleted backstore image image-00004

25/02/2023 21:48:23 INFO PetaSAN cleaned iqns.

25/02/2023 21:48:23 ERROR Cannot unmap image image-civavsan. error, no mapped images found.

25/02/2023 21:48:23 INFO LIO deleted Target iqn.2016-05.infra.vcd.isci:civavsan:00004

25/02/2023 21:48:20 INFO PetaSAN unlocked any consul locks not configured in this node.

25/02/2023 21:48:20 INFO Unlock path 00004/1

25/02/2023 21:48:20 INFO PetaSAN cleaned local paths not locked by this node in consul.

25/02/2023 21:48:20 INFO Cleaned disk path civavsan/1.

25/02/2023 21:48:20 INFO Path 00004/1 acquired successfully

25/02/2023 21:48:20 INFO LIO add_target() disk wwn is ce0495ed00004

25/02/2023 21:48:17 INFO Image image-00004 mapped successfully.

25/02/2023 21:48:17 INFO Found pool:rbd for disk:00004 via consul

25/02/2023 21:48:12 INFO Image image-00004 unmapped successfully.

log from node 3 :

25/02/2023 21:50:14 INFO PetaSAN Cleaned rbd backstores.

25/02/2023 21:50:14 INFO LIO deleted backstore image image-00004

25/02/2023 21:50:11 INFO PetaSAN cleaned iqns.

25/02/2023 21:50:11 ERROR Cannot unmap image image-civavsan. error, no mapped images found.

25/02/2023 21:50:11 INFO LIO deleted Target iqn.2016-05.infra.vcd.isci:civavsan:00004

25/02/2023 21:50:09 INFO PetaSAN unlocked any consul locks not configured in this node.

25/02/2023 21:50:09 INFO Unlock path 00004/2

25/02/2023 21:50:07 INFO PetaSAN cleaned local paths not locked by this node in consul.

25/02/2023 21:50:07 INFO Cleaned disk path civavsan/2.

25/02/2023 21:50:04 INFO Path 00004/2 acquired successfully

25/02/2023 21:50:04 INFO LIO add_target() disk wwn is ce0495ed00004

25/02/2023 21:50:01 INFO Image image-00004 mapped successfully.

25/02/2023 21:50:01 INFO Found pool:rbd for disk:00004 via consul

25/02/2023 21:49:57 INFO Image image-00004 unmapped successfully.

Hope it's help...

#7

admin
2,930 Posts
February 25, 2023, 9:06 pm

1.  Was this working before then stopped working by itself or is this a new install that never worked ?

2.  What is image-civavsan ? is this an image you manually created ? or was this created from UI ?

3. Have you changed any hostnames for any nodes ?

#8

fredohouse
13 Posts
February 25, 2023, 9:56 pm

1 : never work. new install

2: image-civasan don't know. but iqn name is :iqn.2016-05.infra.vcd.isci:CIVAVSAN and the iscsi disk is : VSAN-DATASTORE

was created by UI

3: no

#9

admin
2,930 Posts
February 25, 2023, 10:30 pm

The IQN name is strange iqn.2016-05.infra.vcd.isci:CIVAVSAN.  The :CIVAVSAN  part looks woing. After the : there should be a number like 00001, 00002 which is the iqn for the individual disks. Each disk has its own iqn.

In the iSCSI Settings, what is the IQN base prefix you defined ?

 

#10
Pages: 1 2