Forums

Home / Forums

You need to log in to create posts and topics. Login · Register

dmesg logs after upgrade to 2.3


therm
121 Posts
May 16, 2019, 8:17 am

Hi,

this week we upgraded from 2.1 to 2.2 and finally to 2.3. Today there are new kinds of messages in dmesg log:

on iscsi-server-1:

[Thu May 16 07:36:21 2019] rbd: rbd10: compare-and-write 200 at 71dd000 (1dd000)
[Thu May 16 07:36:21 2019] rbd: rbd10: result -4108 xferred 200
[Thu May 16 07:36:21 2019] COMPARE_AND_WRITE: miscompare at offset 0
[Thu May 16 07:46:16 2019] Unable to recover from DataOut timeout while in ERL=0, closing iSCSI connection for I_T Nexus iqn.1986-03.com.sun:01:002128f9dace.51416e32,i,0x4000002a0000,iqn.2016-05.com.petasan:00013,t,0x01
[Thu May 16 07:46:16 2019] Unable to recover from DataOut timeout while in ERL=0, closing iSCSI connection for I_T Nexus iqn.1986-03.com.sun:01:002128f9dace.51416e32,i,0x4000002a0000,iqn.2016-05.com.petasan:00013,t,0x01

on iscsi-server-2:

[Thu May 16 07:21:14 2019] rbd: rbd9: result -4108 xferred 200
[Thu May 16 07:21:14 2019] COMPARE_AND_WRITE: miscompare at offset 0
[Thu May 16 07:25:11 2019] Unable to recover from DataOut timeout while in ERL=0, closing iSCSI connection for I_T Nexus iqn.1986-03.com.sun:01:002128f9dace.51416e32,i,0x4000002a0000,iqn.2016-05.com.petasan:00012,t,0x01
[Thu May 16 07:25:53 2019] rbd: rbd9: compare-and-write 200 at 71dd000 (1dd000)
[Thu May 16 07:25:53 2019] rbd: rbd9: result -4109 xferred 200
[Thu May 16 07:25:53 2019] COMPARE_AND_WRITE: miscompare at offset 0

Is this something to worry about and what does it mean?

Regards,

Dennis

Last edited on May 16, 2019, 8:19 am by therm · #1

admin
2,930 Posts
May 16, 2019, 9:43 am

Nothing worrying, it means the client iSCSI initiator is sending too much data without being requested to send, so the iSCSI target server is not able to handle in time. The data will be re-tried using a different connection.

This dmesg message was added in kernel for 2.3,  before it used to be a debug level message  so  it would not show up:

https://patchwork.kernel.org/patch/10638261/
https://github.com/openSUSE/kernel/commit/d9a771fd42712cd530544674b04bc3e42cc7927a#diff-57894aeacbdaf6d2ab5751f0fba60d25

Another factor is our recent iSCSI tuning params  in 2.3 which allow the client to send more data in advance for better performance, the upgrade would have chanced the older value.  To revert back to the older (more conservative) values, place the following json in  /opt/petasan/config/tuning/current/lio_tunings:

 

{
"storage_objects": [
{
"attributes": {
"block_size": 512,
"emulate_3pc": 1,
"emulate_caw": 1,
"emulate_tpu": 0,
"queue_depth": 128
}
}
],

"targets": [
{

"tpgs": [
{
"attributes": {
"default_cmdsn_depth": 64
},

"parameters": {
"DataDigest": "None",
"DefaultTime2Retain": "20",
"DefaultTime2Wait": "2",
"FirstBurstLength": "65536",
"HeaderDigest": "None",
"ImmediateData": "Yes",
"InitialR2T": "No",
"MaxBurstLength": "1048576",
"MaxOutstandingR2T": "8",
"MaxRecvDataSegmentLength": "262144",
"MaxXmitDataSegmentLength": "262144"
}
}

]
}
]
}

 

Before you doing this, try to also tune the client vmware side , we have updated the vmware guide for our new recommendations, else you may revert the old json back.

Last edited on May 16, 2019, 9:49 am by admin · #2

therm
121 Posts
May 16, 2019, 12:28 pm

Don`t know yet if it solves the problem, but setting

esxcli system settings advanced set -o /ISCSI/MaxIoSizeKB -i 512

on the vmware host increased throughput enormously!

Regards,

Dennis

Last edited on May 16, 2019, 12:36 pm by therm · #3

admin
2,930 Posts
May 16, 2019, 12:39 pm

you mean it is a good thing ..correct ?  also if you do vmotion  frequently we have also recommendation in the guide to get high performance.

Last edited on May 16, 2019, 12:47 pm by admin · #4