Increase iSCSI log verbosity
quantumschema
7 Posts
November 10, 2020, 2:43 pmQuote from quantumschema on November 10, 2020, 2:43 pmHey everyone,
Our PetaSAN environment is getting hammered with failed iSCSI login negotiations for 3 IQNs/LUNs that don't exist anymore. The ESX environment that uses the PetaSAN storage doesn't show any mappings, dismounted volumes, etc. that would be still trying to access these.
We believe this flood of failed logins is causing a spiral condition on the nodes that eventually causes consul and the boxes to hit 100% CPU utilization and then the ESX environment starts to drop LUN mounts.
Is there a way to increase the iSCSI log verbosity to identify which host is failing authentication?
I'd expect to see "kernel: [37421.369509] iSCSI Login negotiation failed. Host: 192.168.0.1" or similar.
Besides combing through any host that could possibly be using iSCSI and talking to PetaSAN, is there a way to identify the host?
Thanks!
Hey everyone,
Our PetaSAN environment is getting hammered with failed iSCSI login negotiations for 3 IQNs/LUNs that don't exist anymore. The ESX environment that uses the PetaSAN storage doesn't show any mappings, dismounted volumes, etc. that would be still trying to access these.
We believe this flood of failed logins is causing a spiral condition on the nodes that eventually causes consul and the boxes to hit 100% CPU utilization and then the ESX environment starts to drop LUN mounts.
Is there a way to increase the iSCSI log verbosity to identify which host is failing authentication?
I'd expect to see "kernel: [37421.369509] iSCSI Login negotiation failed. Host: 192.168.0.1" or similar.
Besides combing through any host that could possibly be using iSCSI and talking to PetaSAN, is there a way to identify the host?
Thanks!
admin
2,930 Posts
November 10, 2020, 4:48 pmQuote from admin on November 10, 2020, 4:48 pmThe LIO iSCSI target is a kernel module, you can view its logs via dmesg it will show login failure messages and show the client ip. You can change kernel log levels but the default will show the login error so there is no need.
Most probably these could be caused by deleting an iSCSI disk, so it is no longer available, yet you have some old client initiators who still have the old disk configured and keep trying to login, so you need to remove the disks from the clients. It could also happen if clients trying to access deleted disks from a previous PetaSAN cluster setup.
The LIO iSCSI target is a kernel module, you can view its logs via dmesg it will show login failure messages and show the client ip. You can change kernel log levels but the default will show the login error so there is no need.
Most probably these could be caused by deleting an iSCSI disk, so it is no longer available, yet you have some old client initiators who still have the old disk configured and keep trying to login, so you need to remove the disks from the clients. It could also happen if clients trying to access deleted disks from a previous PetaSAN cluster setup.
Increase iSCSI log verbosity
quantumschema
7 Posts
Quote from quantumschema on November 10, 2020, 2:43 pmHey everyone,
Our PetaSAN environment is getting hammered with failed iSCSI login negotiations for 3 IQNs/LUNs that don't exist anymore. The ESX environment that uses the PetaSAN storage doesn't show any mappings, dismounted volumes, etc. that would be still trying to access these.
We believe this flood of failed logins is causing a spiral condition on the nodes that eventually causes consul and the boxes to hit 100% CPU utilization and then the ESX environment starts to drop LUN mounts.
Is there a way to increase the iSCSI log verbosity to identify which host is failing authentication?
I'd expect to see "kernel: [37421.369509] iSCSI Login negotiation failed. Host: 192.168.0.1" or similar.
Besides combing through any host that could possibly be using iSCSI and talking to PetaSAN, is there a way to identify the host?
Thanks!
Hey everyone,
Our PetaSAN environment is getting hammered with failed iSCSI login negotiations for 3 IQNs/LUNs that don't exist anymore. The ESX environment that uses the PetaSAN storage doesn't show any mappings, dismounted volumes, etc. that would be still trying to access these.
We believe this flood of failed logins is causing a spiral condition on the nodes that eventually causes consul and the boxes to hit 100% CPU utilization and then the ESX environment starts to drop LUN mounts.
Is there a way to increase the iSCSI log verbosity to identify which host is failing authentication?
I'd expect to see "kernel: [37421.369509] iSCSI Login negotiation failed. Host: 192.168.0.1" or similar.
Besides combing through any host that could possibly be using iSCSI and talking to PetaSAN, is there a way to identify the host?
Thanks!
admin
2,930 Posts
Quote from admin on November 10, 2020, 4:48 pmThe LIO iSCSI target is a kernel module, you can view its logs via dmesg it will show login failure messages and show the client ip. You can change kernel log levels but the default will show the login error so there is no need.
Most probably these could be caused by deleting an iSCSI disk, so it is no longer available, yet you have some old client initiators who still have the old disk configured and keep trying to login, so you need to remove the disks from the clients. It could also happen if clients trying to access deleted disks from a previous PetaSAN cluster setup.
The LIO iSCSI target is a kernel module, you can view its logs via dmesg it will show login failure messages and show the client ip. You can change kernel log levels but the default will show the login error so there is no need.
Most probably these could be caused by deleting an iSCSI disk, so it is no longer available, yet you have some old client initiators who still have the old disk configured and keep trying to login, so you need to remove the disks from the clients. It could also happen if clients trying to access deleted disks from a previous PetaSAN cluster setup.