iSCSI Path assignment does not always re-assign the paths.
karad
12 Posts
August 23, 2023, 4:37 pmQuote from karad on August 23, 2023, 4:37 pmI'm been seeing since version 3.2.0 that sometimes when I'm trying to reassign some of the iSCSI paths from one node to another it does not always perform the operation.
The last time that I tried to do it, I ended up losing one of the paths of the iSCSI disk.
I was trying to check what I can do to bring the path up again but didn't find any info in the forums.
The only that I found was the issue reported that looks similar to what I'm seeing https://www.petasan.org/forums/?view=thread&id=440.
My questions are:
How can I get it back to the path of the iSCSI disk that was lost?
It is normal that sometimes the path re-assignment is never complete and the path remains on the same host? Is any bug still present?
Below are the logs of Petasan when I tried to perform the path re-assignment and I lost the path
Cluster info:
4 nodes, 127 OSD backed by HDD drives
Each node 2x Xeon E5-2640, and 384 Gb of RAM. 1 bond (2x10G) network for the backend and 2x10 10G for iSCSI services
version: 3.2.1
Node-03 (origin node)
21/08/2023 12:21:49 INFO Found pool:rbd for disk:00004 via consul
21/08/2023 12:21:49 INFO Image image-00004 mapped successfully.
21/08/2023 12:21:52 INFO LIO add_target() disk wwn is 2f13efd300004
21/08/2023 12:21:52 INFO Path 00004/2 acquired successfully
21/08/2023 13:54:22 INFO Found pool:rbd for disk:00004 via consul
21/08/2023 13:54:22 INFO Path 00004/1 acquired successfully
21/08/2023 13:55:24 INFO Found pool:rbd_ssd for disk:00019 via consul
21/08/2023 13:55:24 INFO Image image-00019 mapped successfully.
21/08/2023 13:55:27 INFO LIO add_target() disk wwn is 2f13efd300019
21/08/2023 13:55:27 INFO Path 00019/2 acquired successfully
21/08/2023 13:55:27 INFO Updating path 10.10.81.50 status to 2
21/08/2023 13:55:27 INFO Path 10.10.81.50 status updated to 2
21/08/2023 13:56:21 INFO Reassignment paths script invoked to run clean action.
21/08/2023 13:56:21 INFO Updating path 10.10.82.53 status to 0
21/08/2023 13:56:21 INFO Path 10.10.82.53 status updated to 0
21/08/2023 13:56:21 INFO Found pool:rbd for disk:00004 via consul
21/08/2023 13:56:21 INFO Move action,cleaned disk 00004 path 0.
21/08/2023 13:56:21 INFO Move action,clean newtwork config for disk 00004 path 0.
21/08/2023 13:56:21 INFO {}
21/08/2023 13:56:21 INFO Move action,release disk 00004 path 1.
21/08/2023 13:57:02 INFO Found pool:rbd for disk:00004 via consul
21/08/2023 13:57:02 INFO Path 00004/1 acquired successfully
22/08/2023 06:25:26 INFO GlusterFS mount attempt
23/08/2023 06:25:29 INFO GlusterFS mount attempt
23/08/2023 10:33:20 INFO Cleaned disk path 00019/2.
23/08/2023 10:33:20 INFO PetaSAN cleaned local paths not locked by this node in consul.
23/08/2023 10:33:20 INFO Stopping disk 00019
23/08/2023 10:33:20 ERROR Could not find ips for image-00019
23/08/2023 10:44:50 INFO Reassignment paths script invoked to run clean action.
23/08/2023 10:44:50 INFO Updating path 10.10.82.53 status to 0
23/08/2023 10:44:50 INFO Path 10.10.82.53 status updated to 0
23/08/2023 10:44:50 INFO Found pool:rbd for disk:00004 via consul
23/08/2023 10:44:50 INFO Move action,cleaned disk 00004 path 0.
23/08/2023 10:44:50 INFO Move action,clean newtwork config for disk 00004 path 0.
23/08/2023 10:44:50 INFO {}
23/08/2023 10:44:50 INFO Move action,release disk 00004 path 1.
Node-06 (destination node)
21/08/2023 06:25:24 INFO GlusterFS mount attempt
21/08/2023 12:21:55 INFO Found pool:rbd for disk:00004 via consul
21/08/2023 12:21:55 INFO Could not lock path 00004/2 with session 34d05d80-4dee-34da-e3b7-7e5831599ba0.
21/08/2023 13:53:41 INFO Reassignment paths script invoked to run clean action.
21/08/2023 13:53:41 INFO Updating path 10.10.82.53 status to 0
21/08/2023 13:53:41 INFO Path 10.10.82.53 status updated to 0
21/08/2023 13:53:41 INFO Found pool:rbd for disk:00004 via consul
21/08/2023 13:53:41 INFO Move action,cleaned disk 00004 path 0.
21/08/2023 13:53:41 INFO Move action,clean newtwork config for disk 00004 path 0.
21/08/2023 13:53:41 INFO {}
21/08/2023 13:53:41 INFO Move action,release disk 00004 path 1.
22/08/2023 06:25:28 INFO GlusterFS mount attempt
23/08/2023 06:25:02 INFO GlusterFS mount attempt
I'm been seeing since version 3.2.0 that sometimes when I'm trying to reassign some of the iSCSI paths from one node to another it does not always perform the operation.
The last time that I tried to do it, I ended up losing one of the paths of the iSCSI disk.
I was trying to check what I can do to bring the path up again but didn't find any info in the forums.
The only that I found was the issue reported that looks similar to what I'm seeing https://www.petasan.org/forums/?view=thread&id=440.
My questions are:
How can I get it back to the path of the iSCSI disk that was lost?
It is normal that sometimes the path re-assignment is never complete and the path remains on the same host? Is any bug still present?
Below are the logs of Petasan when I tried to perform the path re-assignment and I lost the path
Cluster info:
4 nodes, 127 OSD backed by HDD drives
Each node 2x Xeon E5-2640, and 384 Gb of RAM. 1 bond (2x10G) network for the backend and 2x10 10G for iSCSI services
version: 3.2.1
Node-03 (origin node)
21/08/2023 12:21:49 INFO Found pool:rbd for disk:00004 via consul
21/08/2023 12:21:49 INFO Image image-00004 mapped successfully.
21/08/2023 12:21:52 INFO LIO add_target() disk wwn is 2f13efd300004
21/08/2023 12:21:52 INFO Path 00004/2 acquired successfully
21/08/2023 13:54:22 INFO Found pool:rbd for disk:00004 via consul
21/08/2023 13:54:22 INFO Path 00004/1 acquired successfully
21/08/2023 13:55:24 INFO Found pool:rbd_ssd for disk:00019 via consul
21/08/2023 13:55:24 INFO Image image-00019 mapped successfully.
21/08/2023 13:55:27 INFO LIO add_target() disk wwn is 2f13efd300019
21/08/2023 13:55:27 INFO Path 00019/2 acquired successfully
21/08/2023 13:55:27 INFO Updating path 10.10.81.50 status to 2
21/08/2023 13:55:27 INFO Path 10.10.81.50 status updated to 2
21/08/2023 13:56:21 INFO Reassignment paths script invoked to run clean action.
21/08/2023 13:56:21 INFO Updating path 10.10.82.53 status to 0
21/08/2023 13:56:21 INFO Path 10.10.82.53 status updated to 0
21/08/2023 13:56:21 INFO Found pool:rbd for disk:00004 via consul
21/08/2023 13:56:21 INFO Move action,cleaned disk 00004 path 0.
21/08/2023 13:56:21 INFO Move action,clean newtwork config for disk 00004 path 0.
21/08/2023 13:56:21 INFO {}
21/08/2023 13:56:21 INFO Move action,release disk 00004 path 1.
21/08/2023 13:57:02 INFO Found pool:rbd for disk:00004 via consul
21/08/2023 13:57:02 INFO Path 00004/1 acquired successfully
22/08/2023 06:25:26 INFO GlusterFS mount attempt
23/08/2023 06:25:29 INFO GlusterFS mount attempt
23/08/2023 10:33:20 INFO Cleaned disk path 00019/2.
23/08/2023 10:33:20 INFO PetaSAN cleaned local paths not locked by this node in consul.
23/08/2023 10:33:20 INFO Stopping disk 00019
23/08/2023 10:33:20 ERROR Could not find ips for image-00019
23/08/2023 10:44:50 INFO Reassignment paths script invoked to run clean action.
23/08/2023 10:44:50 INFO Updating path 10.10.82.53 status to 0
23/08/2023 10:44:50 INFO Path 10.10.82.53 status updated to 0
23/08/2023 10:44:50 INFO Found pool:rbd for disk:00004 via consul
23/08/2023 10:44:50 INFO Move action,cleaned disk 00004 path 0.
23/08/2023 10:44:50 INFO Move action,clean newtwork config for disk 00004 path 0.
23/08/2023 10:44:50 INFO {}
23/08/2023 10:44:50 INFO Move action,release disk 00004 path 1.
Node-06 (destination node)
21/08/2023 06:25:24 INFO GlusterFS mount attempt
21/08/2023 12:21:55 INFO Found pool:rbd for disk:00004 via consul
21/08/2023 12:21:55 INFO Could not lock path 00004/2 with session 34d05d80-4dee-34da-e3b7-7e5831599ba0.
21/08/2023 13:53:41 INFO Reassignment paths script invoked to run clean action.
21/08/2023 13:53:41 INFO Updating path 10.10.82.53 status to 0
21/08/2023 13:53:41 INFO Path 10.10.82.53 status updated to 0
21/08/2023 13:53:41 INFO Found pool:rbd for disk:00004 via consul
21/08/2023 13:53:41 INFO Move action,cleaned disk 00004 path 0.
21/08/2023 13:53:41 INFO Move action,clean newtwork config for disk 00004 path 0.
21/08/2023 13:53:41 INFO {}
21/08/2023 13:53:41 INFO Move action,release disk 00004 path 1.
22/08/2023 06:25:28 INFO GlusterFS mount attempt
23/08/2023 06:25:02 INFO GlusterFS mount attempt
karad
12 Posts
August 23, 2023, 8:07 pmQuote from karad on August 23, 2023, 8:07 pmHi, I'm using 3.2.1. I mentioned version 3.2.0 to reference that the issue with unable to reassign paths has also been seen since that version.
Hi, I'm using 3.2.1. I mentioned version 3.2.0 to reference that the issue with unable to reassign paths has also been seen since that version.
iSCSI Path assignment does not always re-assign the paths.
karad
12 Posts
Quote from karad on August 23, 2023, 4:37 pmI'm been seeing since version 3.2.0 that sometimes when I'm trying to reassign some of the iSCSI paths from one node to another it does not always perform the operation.
The last time that I tried to do it, I ended up losing one of the paths of the iSCSI disk.I was trying to check what I can do to bring the path up again but didn't find any info in the forums.
The only that I found was the issue reported that looks similar to what I'm seeing https://www.petasan.org/forums/?view=thread&id=440.My questions are:
How can I get it back to the path of the iSCSI disk that was lost?
It is normal that sometimes the path re-assignment is never complete and the path remains on the same host? Is any bug still present?Below are the logs of Petasan when I tried to perform the path re-assignment and I lost the path
Cluster info:
4 nodes, 127 OSD backed by HDD drives
Each node 2x Xeon E5-2640, and 384 Gb of RAM. 1 bond (2x10G) network for the backend and 2x10 10G for iSCSI services
version: 3.2.1Node-03 (origin node)
21/08/2023 12:21:49 INFO Found pool:rbd for disk:00004 via consul
21/08/2023 12:21:49 INFO Image image-00004 mapped successfully.
21/08/2023 12:21:52 INFO LIO add_target() disk wwn is 2f13efd300004
21/08/2023 12:21:52 INFO Path 00004/2 acquired successfully
21/08/2023 13:54:22 INFO Found pool:rbd for disk:00004 via consul
21/08/2023 13:54:22 INFO Path 00004/1 acquired successfully
21/08/2023 13:55:24 INFO Found pool:rbd_ssd for disk:00019 via consul
21/08/2023 13:55:24 INFO Image image-00019 mapped successfully.
21/08/2023 13:55:27 INFO LIO add_target() disk wwn is 2f13efd300019
21/08/2023 13:55:27 INFO Path 00019/2 acquired successfully
21/08/2023 13:55:27 INFO Updating path 10.10.81.50 status to 2
21/08/2023 13:55:27 INFO Path 10.10.81.50 status updated to 2
21/08/2023 13:56:21 INFO Reassignment paths script invoked to run clean action.
21/08/2023 13:56:21 INFO Updating path 10.10.82.53 status to 0
21/08/2023 13:56:21 INFO Path 10.10.82.53 status updated to 0
21/08/2023 13:56:21 INFO Found pool:rbd for disk:00004 via consul
21/08/2023 13:56:21 INFO Move action,cleaned disk 00004 path 0.
21/08/2023 13:56:21 INFO Move action,clean newtwork config for disk 00004 path 0.
21/08/2023 13:56:21 INFO {}
21/08/2023 13:56:21 INFO Move action,release disk 00004 path 1.
21/08/2023 13:57:02 INFO Found pool:rbd for disk:00004 via consul
21/08/2023 13:57:02 INFO Path 00004/1 acquired successfully
22/08/2023 06:25:26 INFO GlusterFS mount attempt
23/08/2023 06:25:29 INFO GlusterFS mount attempt
23/08/2023 10:33:20 INFO Cleaned disk path 00019/2.
23/08/2023 10:33:20 INFO PetaSAN cleaned local paths not locked by this node in consul.
23/08/2023 10:33:20 INFO Stopping disk 00019
23/08/2023 10:33:20 ERROR Could not find ips for image-00019
23/08/2023 10:44:50 INFO Reassignment paths script invoked to run clean action.
23/08/2023 10:44:50 INFO Updating path 10.10.82.53 status to 0
23/08/2023 10:44:50 INFO Path 10.10.82.53 status updated to 0
23/08/2023 10:44:50 INFO Found pool:rbd for disk:00004 via consul
23/08/2023 10:44:50 INFO Move action,cleaned disk 00004 path 0.
23/08/2023 10:44:50 INFO Move action,clean newtwork config for disk 00004 path 0.
23/08/2023 10:44:50 INFO {}
23/08/2023 10:44:50 INFO Move action,release disk 00004 path 1.
Node-06 (destination node)
21/08/2023 06:25:24 INFO GlusterFS mount attempt
21/08/2023 12:21:55 INFO Found pool:rbd for disk:00004 via consul
21/08/2023 12:21:55 INFO Could not lock path 00004/2 with session 34d05d80-4dee-34da-e3b7-7e5831599ba0.
21/08/2023 13:53:41 INFO Reassignment paths script invoked to run clean action.
21/08/2023 13:53:41 INFO Updating path 10.10.82.53 status to 0
21/08/2023 13:53:41 INFO Path 10.10.82.53 status updated to 0
21/08/2023 13:53:41 INFO Found pool:rbd for disk:00004 via consul
21/08/2023 13:53:41 INFO Move action,cleaned disk 00004 path 0.
21/08/2023 13:53:41 INFO Move action,clean newtwork config for disk 00004 path 0.
21/08/2023 13:53:41 INFO {}
21/08/2023 13:53:41 INFO Move action,release disk 00004 path 1.
22/08/2023 06:25:28 INFO GlusterFS mount attempt
23/08/2023 06:25:02 INFO GlusterFS mount attempt
I'm been seeing since version 3.2.0 that sometimes when I'm trying to reassign some of the iSCSI paths from one node to another it does not always perform the operation.
The last time that I tried to do it, I ended up losing one of the paths of the iSCSI disk.
I was trying to check what I can do to bring the path up again but didn't find any info in the forums.
The only that I found was the issue reported that looks similar to what I'm seeing https://www.petasan.org/forums/?view=thread&id=440.
My questions are:
How can I get it back to the path of the iSCSI disk that was lost?
It is normal that sometimes the path re-assignment is never complete and the path remains on the same host? Is any bug still present?
Below are the logs of Petasan when I tried to perform the path re-assignment and I lost the path
Cluster info:
4 nodes, 127 OSD backed by HDD drives
Each node 2x Xeon E5-2640, and 384 Gb of RAM. 1 bond (2x10G) network for the backend and 2x10 10G for iSCSI services
version: 3.2.1
Node-03 (origin node)
21/08/2023 12:21:49 INFO Found pool:rbd for disk:00004 via consul
21/08/2023 12:21:49 INFO Image image-00004 mapped successfully.
21/08/2023 12:21:52 INFO LIO add_target() disk wwn is 2f13efd300004
21/08/2023 12:21:52 INFO Path 00004/2 acquired successfully
21/08/2023 13:54:22 INFO Found pool:rbd for disk:00004 via consul
21/08/2023 13:54:22 INFO Path 00004/1 acquired successfully
21/08/2023 13:55:24 INFO Found pool:rbd_ssd for disk:00019 via consul
21/08/2023 13:55:24 INFO Image image-00019 mapped successfully.
21/08/2023 13:55:27 INFO LIO add_target() disk wwn is 2f13efd300019
21/08/2023 13:55:27 INFO Path 00019/2 acquired successfully
21/08/2023 13:55:27 INFO Updating path 10.10.81.50 status to 2
21/08/2023 13:55:27 INFO Path 10.10.81.50 status updated to 2
21/08/2023 13:56:21 INFO Reassignment paths script invoked to run clean action.
21/08/2023 13:56:21 INFO Updating path 10.10.82.53 status to 0
21/08/2023 13:56:21 INFO Path 10.10.82.53 status updated to 0
21/08/2023 13:56:21 INFO Found pool:rbd for disk:00004 via consul
21/08/2023 13:56:21 INFO Move action,cleaned disk 00004 path 0.
21/08/2023 13:56:21 INFO Move action,clean newtwork config for disk 00004 path 0.
21/08/2023 13:56:21 INFO {}
21/08/2023 13:56:21 INFO Move action,release disk 00004 path 1.
21/08/2023 13:57:02 INFO Found pool:rbd for disk:00004 via consul
21/08/2023 13:57:02 INFO Path 00004/1 acquired successfully
22/08/2023 06:25:26 INFO GlusterFS mount attempt
23/08/2023 06:25:29 INFO GlusterFS mount attempt
23/08/2023 10:33:20 INFO Cleaned disk path 00019/2.
23/08/2023 10:33:20 INFO PetaSAN cleaned local paths not locked by this node in consul.
23/08/2023 10:33:20 INFO Stopping disk 00019
23/08/2023 10:33:20 ERROR Could not find ips for image-00019
23/08/2023 10:44:50 INFO Reassignment paths script invoked to run clean action.
23/08/2023 10:44:50 INFO Updating path 10.10.82.53 status to 0
23/08/2023 10:44:50 INFO Path 10.10.82.53 status updated to 0
23/08/2023 10:44:50 INFO Found pool:rbd for disk:00004 via consul
23/08/2023 10:44:50 INFO Move action,cleaned disk 00004 path 0.
23/08/2023 10:44:50 INFO Move action,clean newtwork config for disk 00004 path 0.
23/08/2023 10:44:50 INFO {}
23/08/2023 10:44:50 INFO Move action,release disk 00004 path 1.
Node-06 (destination node)
21/08/2023 06:25:24 INFO GlusterFS mount attempt
21/08/2023 12:21:55 INFO Found pool:rbd for disk:00004 via consul
21/08/2023 12:21:55 INFO Could not lock path 00004/2 with session 34d05d80-4dee-34da-e3b7-7e5831599ba0.
21/08/2023 13:53:41 INFO Reassignment paths script invoked to run clean action.
21/08/2023 13:53:41 INFO Updating path 10.10.82.53 status to 0
21/08/2023 13:53:41 INFO Path 10.10.82.53 status updated to 0
21/08/2023 13:53:41 INFO Found pool:rbd for disk:00004 via consul
21/08/2023 13:53:41 INFO Move action,cleaned disk 00004 path 0.
21/08/2023 13:53:41 INFO Move action,clean newtwork config for disk 00004 path 0.
21/08/2023 13:53:41 INFO {}
21/08/2023 13:53:41 INFO Move action,release disk 00004 path 1.
22/08/2023 06:25:28 INFO GlusterFS mount attempt
23/08/2023 06:25:02 INFO GlusterFS mount attempt
karad
12 Posts
Quote from karad on August 23, 2023, 8:07 pmHi, I'm using 3.2.1. I mentioned version 3.2.0 to reference that the issue with unable to reassign paths has also been seen since that version.
Hi, I'm using 3.2.1. I mentioned version 3.2.0 to reference that the issue with unable to reassign paths has also been seen since that version.