ISCSI lost connection
shadowlin
67 Posts
January 21, 2019, 8:11 amQuote from shadowlin on January 21, 2019, 8:11 amI am hitting a problem this morning.I found the iscsi triffic stopped from our moniter software.
There are alot of errors on the server which runs the iscsi targert:
[Mon Jan 21 16:03:35 2019] iSCSI Login negotiation failed.
[Mon Jan 21 16:03:50 2019] iSCSI Login timeout on Network Portal 10.50.7.247:3260
[Mon Jan 21 16:03:50 2019] tx_data returned -104, expecting 48.
[Mon Jan 21 16:03:50 2019] iSCSI Login negotiation failed.
[Mon Jan 21 16:04:05 2019] iSCSI Login timeout on Network Portal 10.50.7.247:3260
[Mon Jan 21 16:04:05 2019] tx_data returned -104, expecting 48.
[Mon Jan 21 16:04:05 2019] iSCSI Login negotiation failed.
What could cause this? The ceph end seems ok and the service has been running for a long time without any incident
I am hitting a problem this morning.I found the iscsi triffic stopped from our moniter software.
There are alot of errors on the server which runs the iscsi targert:
[Mon Jan 21 16:03:35 2019] iSCSI Login negotiation failed.
[Mon Jan 21 16:03:50 2019] iSCSI Login timeout on Network Portal 10.50.7.247:3260
[Mon Jan 21 16:03:50 2019] tx_data returned -104, expecting 48.
[Mon Jan 21 16:03:50 2019] iSCSI Login negotiation failed.
[Mon Jan 21 16:04:05 2019] iSCSI Login timeout on Network Portal 10.50.7.247:3260
[Mon Jan 21 16:04:05 2019] tx_data returned -104, expecting 48.
[Mon Jan 21 16:04:05 2019] iSCSI Login negotiation failed.
What could cause this? The ceph end seems ok and the service has been running for a long time without any incident
admin
2,930 Posts
January 21, 2019, 8:43 amQuote from admin on January 21, 2019, 8:43 amIf you login from a different client machine to the same iscsi server and same lun, does it work ?
If you login from a different client machine to the same iscsi server and same lun, does it work ?
admin
2,930 Posts
January 21, 2019, 9:49 amQuote from admin on January 21, 2019, 9:49 amThis is a "Connection reset by peer" error:
#define ECONNRESET 104 /* Connection reset by peer */
happens at the lower tcp send message level when trying to transmit back iSCSI login parameters back to the initiator.
Check your network, connections..
This is a "Connection reset by peer" error:
#define ECONNRESET 104 /* Connection reset by peer */
happens at the lower tcp send message level when trying to transmit back iSCSI login parameters back to the initiator.
Check your network, connections..
shadowlin
67 Posts
January 22, 2019, 5:59 amQuote from shadowlin on January 22, 2019, 5:59 amThank you admin.
It turns out that the ceph cluster was not working properly there were some request were blocked which caused this error.
The error itself is kinda confusing why an unresponsive ceph cluster can cause this problem?
Thank you admin.
It turns out that the ceph cluster was not working properly there were some request were blocked which caused this error.
The error itself is kinda confusing why an unresponsive ceph cluster can cause this problem?
admin
2,930 Posts
January 22, 2019, 7:54 amQuote from admin on January 22, 2019, 7:54 amAs per my earlier post, the error is coming from the tcp stack due to a connection reset by peer while trying to transmit back iSCSI login parameters.
This is not related to iSCSI or Ceph, but something at the network/hardware. It can cause Ceph or iSCSI to fail but not the other way.
As per my earlier post, the error is coming from the tcp stack due to a connection reset by peer while trying to transmit back iSCSI login parameters.
This is not related to iSCSI or Ceph, but something at the network/hardware. It can cause Ceph or iSCSI to fail but not the other way.
shadowlin
67 Posts
February 16, 2019, 1:37 amQuote from shadowlin on February 16, 2019, 1:37 am@admin
But after I fix ceph cluster this error went away.
The root cause was ceph had some error that blocked all incoming request.After this error was solved everything became ok automatically
@admin
But after I fix ceph cluster this error went away.
The root cause was ceph had some error that blocked all incoming request.After this error was solved everything became ok automatically
ISCSI lost connection
shadowlin
67 Posts
Quote from shadowlin on January 21, 2019, 8:11 amI am hitting a problem this morning.I found the iscsi triffic stopped from our moniter software.
There are alot of errors on the server which runs the iscsi targert:
[Mon Jan 21 16:03:35 2019] iSCSI Login negotiation failed.
[Mon Jan 21 16:03:50 2019] iSCSI Login timeout on Network Portal 10.50.7.247:3260
[Mon Jan 21 16:03:50 2019] tx_data returned -104, expecting 48.
[Mon Jan 21 16:03:50 2019] iSCSI Login negotiation failed.
[Mon Jan 21 16:04:05 2019] iSCSI Login timeout on Network Portal 10.50.7.247:3260
[Mon Jan 21 16:04:05 2019] tx_data returned -104, expecting 48.
[Mon Jan 21 16:04:05 2019] iSCSI Login negotiation failed.
What could cause this? The ceph end seems ok and the service has been running for a long time without any incident
I am hitting a problem this morning.I found the iscsi triffic stopped from our moniter software.
There are alot of errors on the server which runs the iscsi targert:
[Mon Jan 21 16:03:35 2019] iSCSI Login negotiation failed.
[Mon Jan 21 16:03:50 2019] iSCSI Login timeout on Network Portal 10.50.7.247:3260
[Mon Jan 21 16:03:50 2019] tx_data returned -104, expecting 48.
[Mon Jan 21 16:03:50 2019] iSCSI Login negotiation failed.
[Mon Jan 21 16:04:05 2019] iSCSI Login timeout on Network Portal 10.50.7.247:3260
[Mon Jan 21 16:04:05 2019] tx_data returned -104, expecting 48.
[Mon Jan 21 16:04:05 2019] iSCSI Login negotiation failed.
What could cause this? The ceph end seems ok and the service has been running for a long time without any incident
admin
2,930 Posts
Quote from admin on January 21, 2019, 8:43 amIf you login from a different client machine to the same iscsi server and same lun, does it work ?
If you login from a different client machine to the same iscsi server and same lun, does it work ?
admin
2,930 Posts
Quote from admin on January 21, 2019, 9:49 amThis is a "Connection reset by peer" error:
#define ECONNRESET 104 /* Connection reset by peer */
happens at the lower tcp send message level when trying to transmit back iSCSI login parameters back to the initiator.
Check your network, connections..
This is a "Connection reset by peer" error:
#define ECONNRESET 104 /* Connection reset by peer */
happens at the lower tcp send message level when trying to transmit back iSCSI login parameters back to the initiator.
Check your network, connections..
shadowlin
67 Posts
Quote from shadowlin on January 22, 2019, 5:59 amThank you admin.
It turns out that the ceph cluster was not working properly there were some request were blocked which caused this error.
The error itself is kinda confusing why an unresponsive ceph cluster can cause this problem?
Thank you admin.
It turns out that the ceph cluster was not working properly there were some request were blocked which caused this error.
The error itself is kinda confusing why an unresponsive ceph cluster can cause this problem?
admin
2,930 Posts
Quote from admin on January 22, 2019, 7:54 amAs per my earlier post, the error is coming from the tcp stack due to a connection reset by peer while trying to transmit back iSCSI login parameters.
This is not related to iSCSI or Ceph, but something at the network/hardware. It can cause Ceph or iSCSI to fail but not the other way.
As per my earlier post, the error is coming from the tcp stack due to a connection reset by peer while trying to transmit back iSCSI login parameters.
This is not related to iSCSI or Ceph, but something at the network/hardware. It can cause Ceph or iSCSI to fail but not the other way.
shadowlin
67 Posts
Quote from shadowlin on February 16, 2019, 1:37 am@admin
But after I fix ceph cluster this error went away.
The root cause was ceph had some error that blocked all incoming request.After this error was solved everything became ok automatically
@admin
But after I fix ceph cluster this error went away.
The root cause was ceph had some error that blocked all incoming request.After this error was solved everything became ok automatically