Forums

Home / Forums

You need to log in to create posts and topics. Login · Register

iscsi keep restarting randomly.


emanzx
11 Posts
April 26, 2018, 4:34 am

Hi,

I new to petaSAN and just deploy 3 host with 6 total OSD. I dont notice any problem before until recently our VM keep hangup for some reason. I did some investigation and found out that the iscsi is disconnect and reconnecting and it happen alot.

error log from xenserver

Apr 26 12:23:11 xenserver-01 iscsid: Kernel reported iSCSI connection 2:0 error (1011 - ISCSI_ERR_CONN_FAILED: iSCSI connection failed) state (3)

Apr 26 12:23:24 xenserver-01 xapi: [ info|xenserver-01|3150 INET :::80|session.logout D:7ba7f7ecb48a|xapi] Session.destroy trackid=2a3b64d1a9c3a842ab12cd08c74298c3

Apr 26 12:23:25 xenserver-01 kernel: [3608407.604452]  session2: session recovery timed out after 15 secs

Apr 26 12:23:25 xenserver-01 multipathd: 8:32: mark as failed

Apr 26 12:23:25 xenserver-01 multipathd: 360014050000600000000000000000000: remaining active paths: 1

Apr 26 12:23:25 xenserver-01 multipathd: Path event for 360014050000600000000000000000000, request call of mpathcount

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608476] sd 19:0:0:0: rejecting I/O to offline device

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608484] sd 19:0:0:0: [sdc] killing request

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608486] sd 19:0:0:0: rejecting I/O to offline device

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608489] sd 19:0:0:0: [sdc] killing request

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608491] sd 19:0:0:0: rejecting I/O to offline device

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608493] sd 19:0:0:0: [sdc] killing request

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608494] sd 19:0:0:0: rejecting I/O to offline device

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608497] sd 19:0:0:0: [sdc] killing request

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608498] sd 19:0:0:0: rejecting I/O to offline device

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608500] sd 19:0:0:0: [sdc] killing request

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608501] sd 19:0:0:0: rejecting I/O to offline device

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608504] sd 19:0:0:0: [sdc] killing request

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608505] sd 19:0:0:0: rejecting I/O to offline device

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608507] sd 19:0:0:0: [sdc] killing request

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608516] sd 19:0:0:0: [sdc] Unhandled error code

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608517] sd 19:0:0:0: [sdc] 

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608518] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608519] sd 19:0:0:0: [sdc] CDB:

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608520] Write(10): 2a 00 04 d3 e2 20 00 00 28 00

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608525] end_request: I/O error, dev sdc, sector 80994848

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608533] sd 19:0:0:0: [sdc] Unhandled error code

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608534] sd 19:0:0:0: [sdc] 

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608535] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608536] sd 19:0:0:0: [sdc] CDB:

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608536] Write(10): 2a 00 04 64 ae 08 00 01 08 00

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608540] end_request: I/O error, dev sdc, sector 73707016

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608545] sd 19:0:0:0: [sdc] Unhandled error code

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608546] sd 19:0:0:0: [sdc] 

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608546] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608547] sd 19:0:0:0: [sdc] CDB:

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608548] Write(10): 2a 00 05 69 57 18 00 00 20 00

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608552] end_request: I/O error, dev sdc, sector 90789656

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608556] sd 19:0:0:0: [sdc] Unhandled error code

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608556] sd 19:0:0:0: [sdc] 

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608557] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608558] sd 19:0:0:0: [sdc] CDB:

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608559] Write(10): 2a 00 05 1f e0 b8 00 00 08 00

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608562] end_request: I/O error, dev sdc, sector 85975224

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608566] sd 19:0:0:0: [sdc] Unhandled error code

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608567] sd 19:0:0:0: [sdc] 

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608568] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608569] sd 19:0:0:0: [sdc] CDB:

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608569] Write(10): 2a 00 05 70 cb 10 00 00 20 00

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608573] end_request: I/O error, dev sdc, sector 91278096

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608577] sd 19:0:0:0: [sdc] Unhandled error code

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608578] sd 19:0:0:0: [sdc] 

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608578] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608579] sd 19:0:0:0: [sdc] CDB:

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608580] Write(10): 2a 00 04 77 c6 a8 00 00 08 00

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608584] end_request: I/O error, dev sdc, sector 74958504

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608587] sd 19:0:0:0: [sdc] Unhandled error code

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608588] sd 19:0:0:0: [sdc] 

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608589] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608590] sd 19:0:0:0: [sdc] CDB:

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608590] Write(10): 2a 00 06 14 cb df 00 00 01 00

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608594] end_request: I/O error, dev sdc, sector 102026207

Apr 26 12:23:25 xenserver-01 kernel: [3608407.608598] device-mapper: multipath: Failing path 8:32.

Petasan iscsi keep restarting

log from petasan

Apr 26 12:26:57 petasan-01 kernel: [591973.516762] rbd: rbd2: capacity 1099511627776 features 0x1

Apr 26 12:26:58 petasan-01 ntpd[1039]: Listen normally on 2565 eth1 10.0.3.103:123

Apr 26 12:26:58 petasan-01 ntpd[1039]: 26 Apr 12:26:58 ntpd[1039]: Listen normally on 2565 eth1 10.0.3.103:123

Apr 26 12:26:58 petasan-01 ntpd[1039]: 26 Apr 12:26:58 ntpd[1039]: new interface(s) found: waking up resolver

Apr 26 12:26:58 petasan-01 ntpd[1039]: new interface(s) found: waking up resolver

Apr 26 12:27:01 petasan-01 iscsi_service.py[1273]: ARPING 10.0.3.103 from 10.0.3.103 eth1

Apr 26 12:27:01 petasan-01 iscsi_service.py[1273]: Sent 5 probes (5 broadcast(s))

Apr 26 12:27:01 petasan-01 iscsi_service.py[1273]: Received 0 response(s)

Apr 26 12:27:03 petasan-01 iscsi_service.py[1273]: 0 rbd image-00004 - /dev/rbd0

Apr 26 12:27:03 petasan-01 iscsi_service.py[1273]: 1 rbd image-00005 - /dev/rbd1

Apr 26 12:27:03 petasan-01 iscsi_service.py[1273]: 2 rbd image-00006 - /dev/rbd2

Apr 26 12:27:10 petasan-01 iscsi_service.py[1273]: ARPING 10.0.3.103 from 10.0.3.103 eth1

Apr 26 12:27:10 petasan-01 iscsi_service.py[1273]: Sent 5 probes (5 broadcast(s))

Apr 26 12:27:10 petasan-01 iscsi_service.py[1273]: Received 0 response(s)

Apr 26 12:27:18 petasan-01 iscsi_service.py[1273]: 0 rbd image-00004 - /dev/rbd0

Apr 26 12:27:18 petasan-01 iscsi_service.py[1273]: 1 rbd image-00005 - /dev/rbd1

Apr 26 12:27:18 petasan-01 iscsi_service.py[1273]: 2 rbd image-00006 - /dev/rbd2

Apr 26 12:27:19 petasan-01 iscsi_service.py[1273]: ARPING 10.0.3.103 from 10.0.3.103 eth1

Apr 26 12:27:19 petasan-01 iscsi_service.py[1273]: Sent 5 probes (5 broadcast(s))

Apr 26 12:27:19 petasan-01 iscsi_service.py[1273]: Received 0 response(s)

Apr 26 12:27:25 petasan-01 iscsi_service.py[1273]: 0 rbd image-00004 - /dev/rbd0

Apr 26 12:27:25 petasan-01 iscsi_service.py[1273]: 1 rbd image-00005 - /dev/rbd1

Apr 26 12:27:25 petasan-01 iscsi_service.py[1273]: 2 rbd image-00006 - /dev/rbd2

Apr 26 12:27:28 petasan-01 iscsi_service.py[1273]: ARPING 10.0.3.103 from 10.0.3.103 eth1

Apr 26 12:27:28 petasan-01 iscsi_service.py[1273]: Sent 5 probes (5 broadcast(s))

Apr 26 12:27:28 petasan-01 iscsi_service.py[1273]: Received 0 response(s)

Apr 26 12:27:34 petasan-01 iscsi_service.py[1273]: 0 rbd image-00004 - /dev/rbd0

Apr 26 12:27:34 petasan-01 iscsi_service.py[1273]: 1 rbd image-00005 - /dev/rbd1

Apr 26 12:27:34 petasan-01 iscsi_service.py[1273]: 2 rbd image-00006 - /dev/rbd2

Apr 26 12:27:37 petasan-01 iscsi_service.py[1273]: ARPING 10.0.3.103 from 10.0.3.103 eth1

Apr 26 12:27:37 petasan-01 iscsi_service.py[1273]: Sent 5 probes (5 broadcast(s))

Apr 26 12:27:37 petasan-01 iscsi_service.py[1273]: Received 0 response(s)

Apr 26 12:27:45 petasan-01 iscsi_service.py[1273]: 0 rbd image-00004 - /dev/rbd0

Apr 26 12:27:45 petasan-01 iscsi_service.py[1273]: 1 rbd image-00005 - /dev/rbd1

Apr 26 12:27:45 petasan-01 iscsi_service.py[1273]: 2 rbd image-00006 - /dev/rbd2

Apr 26 12:27:58 petasan-01 iscsi_service.py[1273]: 0 rbd image-00004 - /dev/rbd0

Apr 26 12:27:58 petasan-01 iscsi_service.py[1273]: 1 rbd image-00005 - /dev/rbd1

Apr 26 12:27:58 petasan-01 iscsi_service.py[1273]: 2 rbd image-00006 - /dev/rbd2

Apr 26 12:28:13 petasan-01 iscsi_service.py[1273]: 0 rbd image-00004 - /dev/rbd0

Apr 26 12:28:13 petasan-01 iscsi_service.py[1273]: 1 rbd image-00005 - /dev/rbd1

Apr 26 12:28:13 petasan-01 iscsi_service.py[1273]: 2 rbd image-00006 - /dev/rbd2

Apr 26 12:28:30 petasan-01 iscsi_service.py[1273]: 0 rbd image-00004 - /dev/rbd0

Apr 26 12:28:30 petasan-01 iscsi_service.py[1273]: 1 rbd image-00005 - /dev/rbd1

Apr 26 12:28:30 petasan-01 iscsi_service.py[1273]: 2 rbd image-00006 - /dev/rbd2

Apr 26 12:28:49 petasan-01 iscsi_service.py[1273]: 0 rbd image-00004 - /dev/rbd0

Apr 26 12:28:49 petasan-01 iscsi_service.py[1273]: 1 rbd image-00005 - /dev/rbd1

Apr 26 12:28:49 petasan-01 iscsi_service.py[1273]: 2 rbd image-00006 - /dev/rbd2

Apr 26 12:29:11 petasan-01 iscsi_service.py[1273]: 0 rbd image-00004 - /dev/rbd0

Apr 26 12:29:11 petasan-01 iscsi_service.py[1273]: 1 rbd image-00005 - /dev/rbd1

Apr 26 12:29:11 petasan-01 iscsi_service.py[1273]: 2 rbd image-00006 - /dev/rbd2

Apr 26 12:29:34 petasan-01 iscsi_service.py[1273]: 0 rbd image-00004 - /dev/rbd0

Apr 26 12:29:34 petasan-01 iscsi_service.py[1273]: 1 rbd image-00005 - /dev/rbd1

Apr 26 12:29:34 petasan-01 iscsi_service.py[1273]: 2 rbd image-00006 - /dev/rbd2

Apr 26 12:29:49 petasan-01 iscsi_service.py[1273]: 0 rbd image-00004 - /dev/rbd0

Apr 26 12:29:49 petasan-01 iscsi_service.py[1273]: 1 rbd image-00005 - /dev/rbd1

Apr 26 12:29:49 petasan-01 iscsi_service.py[1273]: 2 rbd image-00006 - /dev/rbd2

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: ('data', [{u'LockIndex': 0, u'ModifyIndex': 76, u'Value': 'MTkyLjE2OC4xMC43ICAgcGV0YXNhbi0wMwoxMjcuMC4wLjEgICBsb2NhbGhvc3QKMTkyLjE2OC4xMC41ICAgcGV0YXNhbi0wM

QoxOTIuMTY4LjEwLjYgICBwZXRhc2FuLTAyCg==', u'Flags': 0, u'Key': u'PetaSAN/Config/Files/etc/hosts', u'CreateIndex': 76}])

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: -------------------------------------

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: ('updated current_index: ', '76')

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: file_path

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: /etc/hosts

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: Gonna write file

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: wrote file

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: ('index', '76')

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: ('data', [{u'LockIndex': 0, u'ModifyIndex': 76, u'Value': 'MTkyLjE2OC4xMC43ICAgcGV0YXNhbi0wMwoxMjcuMC4wLjEgICBsb2NhbGhvc3QKMTkyLjE2OC4xMC41ICAgcGV0YXNhbi0wM

QoxOTIuMTY4LjEwLjYgICBwZXRhc2FuLTAyCg==', u'Flags': 0, u'Key': u'PetaSAN/Config/Files/etc/hosts', u'CreateIndex': 76}])

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: -------------------------------------

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: ('updated current_index: ', '76')

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: file_path

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: /etc/hosts

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: Gonna write file

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: wrote file

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: ('index', '76')

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: ('data', [{u'LockIndex': 0, u'ModifyIndex': 76, u'Value': 'MTkyLjE2OC4xMC43ICAgcGV0YXNhbi0wMwoxMjcuMC4wLjEgICBsb2NhbGhvc3QKMTkyLjE2OC4xMC41ICAgcGV0YXNhbi0wM

QoxOTIuMTY4LjEwLjYgICBwZXRhc2FuLTAyCg==', u'Flags': 0, u'Key': u'PetaSAN/Config/Files/etc/hosts', u'CreateIndex': 76}])

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: -------------------------------------

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: ('updated current_index: ', '76')

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: file_path

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: /etc/hosts

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: Gonna write file

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: wrote file

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: ('index', '76')

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: ('data', [{u'LockIndex': 0, u'ModifyIndex': 76, u'Value': 'MTkyLjE2OC4xMC43ICAgcGV0YXNhbi0wMwoxMjcuMC4wLjEgICBsb2NhbGhvc3QKMTkyLjE2OC4xMC41ICAgcGV0YXNhbi0wM

QoxOTIuMTY4LjEwLjYgICBwZXRhc2FuLTAyCg==', u'Flags': 0, u'Key': u'PetaSAN/Config/Files/etc/hosts', u'CreateIndex': 76}])

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: -------------------------------------

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: ('updated current_index: ', '76')

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: file_path

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: /etc/hosts

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: Gonna write file

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: wrote file

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: ('index', '76')

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: ('data', [{u'LockIndex': 0, u'ModifyIndex': 76, u'Value': 'MTkyLjE2OC4xMC43ICAgcGV0YXNhbi0wMwoxMjcuMC4wLjEgICBsb2NhbGhvc3QKMTkyLjE2OC4xMC41ICAgcGV0YXNhbi0wM

QoxOTIuMTY4LjEwLjYgICBwZXRhc2FuLTAyCg==', u'Flags': 0, u'Key': u'PetaSAN/Config/Files/etc/hosts', u'CreateIndex': 76}])

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: -------------------------------------

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: ('updated current_index: ', '76')

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: file_path

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: /etc/hosts

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: Gonna write file

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: wrote file

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: ('index', '76')

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: ('data', [{u'LockIndex': 0, u'ModifyIndex': 76, u'Value': 'MTkyLjE2OC4xMC43ICAgcGV0YXNhbi0wMwoxMjcuMC4wLjEgICBsb2NhbGhvc3QKMTkyLjE2OC4xMC41ICAgcGV0YXNhbi0wM

QoxOTIuMTY4LjEwLjYgICBwZXRhc2FuLTAyCg==', u'Flags': 0, u'Key': u'PetaSAN/Config/Files/etc/hosts', u'CreateIndex': 76}])

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: -------------------------------------

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: ('updated current_index: ', '76')

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: file_path

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: /etc/hosts

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: Gonna write file

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: wrote file

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: ('index', '76')

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: ('data', [{u'LockIndex': 0, u'ModifyIndex': 76, u'Value': 'MTkyLjE2OC4xMC43ICAgcGV0YXNhbi0wMwoxMjcuMC4wLjEgICBsb2NhbGhvc3QKMTkyLjE2OC4xMC41ICAgcGV0YXNhbi0wM

QoxOTIuMTY4LjEwLjYgICBwZXRhc2FuLTAyCg==', u'Flags': 0, u'Key': u'PetaSAN/Config/Files/etc/hosts', u'CreateIndex': 76}])

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: -------------------------------------

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: ('updated current_index: ', '76')

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: file_path

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: /etc/hosts

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: Gonna write file

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: wrote file

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: ('index', '76')

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: ('data', [{u'LockIndex': 0, u'ModifyIndex': 76, u'Value': 'MTkyLjE2OC4xMC43ICAgcGV0YXNhbi0wMwoxMjcuMC4wLjEgICBsb2NhbGhvc3QKMTkyLjE2OC4xMC41ICAgcGV0YXNhbi0wM

QoxOTIuMTY4LjEwLjYgICBwZXRhc2FuLTAyCg==', u'Flags': 0, u'Key': u'PetaSAN/Config/Files/etc/hosts', u'CreateIndex': 76}])

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: -------------------------------------

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: ('updated current_index: ', '76')

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: file_path

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: /etc/hosts

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: Gonna write file

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: wrote file

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: ('index', '76')

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: ('data', [{u'LockIndex': 0, u'ModifyIndex': 76, u'Value': 'MTkyLjE2OC4xMC43ICAgcGV0YXNhbi0wMwoxMjcuMC4wLjEgICBsb2NhbGhvc3QKMTkyLjE2OC4xMC41ICAgcGV0YXNhbi0wM

QoxOTIuMTY4LjEwLjYgICBwZXRhc2FuLTAyCg==', u'Flags': 0, u'Key': u'PetaSAN/Config/Files/etc/hosts', u'CreateIndex': 76}])

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: -------------------------------------

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: ('updated current_index: ', '76')

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: file_path

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: /etc/hosts

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: Gonna write file

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: wrote file

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: ('index', '76')

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: ('data', [{u'LockIndex': 0, u'ModifyIndex': 76, u'Value': 'MTkyLjE2OC4xMC43ICAgcGV0YXNhbi0wMwoxMjcuMC4wLjEgICBsb2NhbGhvc3QKMTkyLjE2OC4xMC41ICAgcGV0YXNhbi0wM

QoxOTIuMTY4LjEwLjYgICBwZXRhc2FuLTAyCg==', u'Flags': 0, u'Key': u'PetaSAN/Config/Files/etc/hosts', u'CreateIndex': 76}])

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: -------------------------------------

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: ('updated current_index: ', '76')

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: file_path

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: /etc/hosts

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: Gonna write file

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: wrote file

Apr 26 12:29:55 petasan-01 files_sync.py[1246]: ('index', '76')

Apr 26 12:29:56 petasan-01 iscsi_service.py[1273]: 0 rbd image-00004 - /dev/rbd0

Apr 26 12:29:56 petasan-01 iscsi_service.py[1273]: 1 rbd image-00005 - /dev/rbd1

Apr 26 12:29:56 petasan-01 iscsi_service.py[1273]: 2 rbd image-00006 - /dev/rbd2

Apr 26 12:30:05 petasan-01 iscsi_service.py[1273]: 0 rbd image-00004 - /dev/rbd0

Apr 26 12:30:05 petasan-01 iscsi_service.py[1273]: 1 rbd image-00005 - /dev/rbd1

Apr 26 12:30:05 petasan-01 iscsi_service.py[1273]: 2 rbd image-00006 - /dev/rbd2

Apr 26 12:30:16 petasan-01 iscsi_service.py[1273]: 0 rbd image-00004 - /dev/rbd0

Apr 26 12:30:16 petasan-01 iscsi_service.py[1273]: 1 rbd image-00005 - /dev/rbd1

Apr 26 12:30:16 petasan-01 iscsi_service.py[1273]: 2 rbd image-00006 - /dev/rbd2

Apr 26 12:30:29 petasan-01 iscsi_service.py[1273]: 0 rbd image-00004 - /dev/rbd0

Apr 26 12:30:29 petasan-01 iscsi_service.py[1273]: 1 rbd image-00005 - /dev/rbd1

Apr 26 12:30:29 petasan-01 iscsi_service.py[1273]: 2 rbd image-00006 - /dev/rbd2

Apr 26 12:30:44 petasan-01 iscsi_service.py[1273]: 0 rbd image-00004 - /dev/rbd0

Apr 26 12:30:44 petasan-01 iscsi_service.py[1273]: 1 rbd image-00005 - /dev/rbd1

Apr 26 12:30:44 petasan-01 iscsi_service.py[1273]: 2 rbd image-00006 - /dev/rbd2

Apr 26 12:31:01 petasan-01 iscsi_service.py[1273]: 0 rbd image-00004 - /dev/rbd0

Apr 26 12:31:01 petasan-01 iscsi_service.py[1273]: 1 rbd image-00005 - /dev/rbd1

Apr 26 12:31:01 petasan-01 iscsi_service.py[1273]: 2 rbd image-00006 - /dev/rbd2

Apr 26 12:31:20 petasan-01 iscsi_service.py[1273]: 0 rbd image-00004 - /dev/rbd0

Apr 26 12:31:20 petasan-01 iscsi_service.py[1273]: 1 rbd image-00005 - /dev/rbd1

Apr 26 12:31:20 petasan-01 iscsi_service.py[1273]: 2 rbd image-00006 - /dev/rbd2

Apr 26 12:31:41 petasan-01 iscsi_service.py[1273]: 0 rbd image-00004 - /dev/rbd0

Apr 26 12:31:41 petasan-01 iscsi_service.py[1273]: 1 rbd image-00005 - /dev/rbd1

Apr 26 12:31:41 petasan-01 iscsi_service.py[1273]: 2 rbd image-00006 - /dev/rbd2

Apr 26 12:32:05 petasan-01 iscsi_service.py[1273]: 0 rbd image-00004 - /dev/rbd0

Apr 26 12:32:05 petasan-01 iscsi_service.py[1273]: 1 rbd image-00005 - /dev/rbd1

Apr 26 12:32:05 petasan-01 iscsi_service.py[1273]: 2 rbd image-00006 - /dev/rbd2

Apr 26 12:32:20 petasan-01 iscsi_service.py[1273]: 0 rbd image-00004 - /dev/rbd0

Apr 26 12:32:20 petasan-01 iscsi_service.py[1273]: 1 rbd image-00005 - /dev/rbd1

Apr 26 12:32:20 petasan-01 iscsi_service.py[1273]: 2 rbd image-00006 - /dev/rbd2

Apr 26 12:32:27 petasan-01 iscsi_service.py[1273]: 0 rbd image-00004 - /dev/rbd0

Apr 26 12:32:27 petasan-01 iscsi_service.py[1273]: 1 rbd image-00005 - /dev/rbd1

Apr 26 12:32:27 petasan-01 iscsi_service.py[1273]: 2 rbd image-00006 - /dev/rbd2

Apr 26 12:32:36 petasan-01 iscsi_service.py[1273]: 0 rbd image-00004 - /dev/rbd0

Apr 26 12:32:36 petasan-01 iscsi_service.py[1273]: 1 rbd image-00005 - /dev/rbd1

Apr 26 12:32:36 petasan-01 iscsi_service.py[1273]: 2 rbd image-00006 - /dev/rbd2

Apr 26 12:32:47 petasan-01 iscsi_service.py[1273]: 0 rbd image-00004 - /dev/rbd0

Apr 26 12:32:47 petasan-01 iscsi_service.py[1273]: 1 rbd image-00005 - /dev/rbd1

Apr 26 12:32:47 petasan-01 iscsi_service.py[1273]: 2 rbd image-00006 - /dev/rbd2

Apr 26 12:33:00 petasan-01 iscsi_service.py[1273]: 0 rbd image-00004 - /dev/rbd0

Apr 26 12:33:00 petasan-01 iscsi_service.py[1273]: 1 rbd image-00005 - /dev/rbd1

Apr 26 12:33:00 petasan-01 iscsi_service.py[1273]: 2 rbd image-00006 - /dev/rbd2

Last edited on April 26, 2018, 4:35 am by emanzx · #1

admin
2,930 Posts
April 26, 2018, 5:32 am

This could happen if the node serving the iSCSI disk loses connection to the remaining nodes, more specifically fails to respond to Consul cluster heartbeats, or loses connection to Ceph.

It could be intermittent/flaky network connection on backend 1 subnet or excessive load. I recommend you check backend 1 nics/switches..etc, look at load stats charts for cpu/disk/ram % utilization if they go above 90% then your system may be under-powered  : it may not be able to handle your peak client io + periodic scrubbing + any backend recovery process, you can switch off scrubbing for 1 or 2 days and see if this helps.

You should also look at the log in /opt/petasan/log/PetaSAN.log in will show any connection errors. If you do not solve this via the above then please send the log to contact-us @ petasan.org

Last edited on April 26, 2018, 5:33 am by admin · #2

emanzx
11 Posts
April 26, 2018, 6:06 am

Thanks for the reply.

Im suspecting the network too. I will try to check on that.

but here are some of the info I can grep from the Petasan.log

25/04/2018 08:14:59 ERROR    Error during process.

25/04/2018 08:14:59 ERROR    Cannot get metadata.

Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 98, in start

    self.__process()

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 132, in __process

    while self.__do_process() != True:

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 205, in __do_process

    self.__acquire_path(str(path), self.__paths_consul_unlocked_firstborn.get(path))

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 523, in __acquire_path

    raise e

MetadataException: Cannot get metadata.

25/04/2018 08:15:10 ERROR    Could not acquire path 00001/1

Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 439, in __acquire_path

    all_image_meta = ceph_api.read_image_metadata(image_name)

  File "/usr/lib/python2.7/dist-packages/PetaSAN/core/ceph/api.py", line 182, in read_image_metadata

    raise MetadataException("Cannot get metadata.")

MetadataException: Cannot get metadata.

25/04/2018 08:15:10 ERROR    Error during process.

25/04/2018 08:15:10 ERROR    Cannot get metadata.

Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 98, in start

    self.__process()

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 132, in __process

    while self.__do_process() != True:

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 205, in __do_process

    self.__acquire_path(str(path), self.__paths_consul_unlocked_firstborn.get(path))

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 523, in __acquire_path

    raise e

MetadataException: Cannot get metadata.

25/04/2018 08:15:23 ERROR    Could not acquire path 00001/2

Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 439, in __acquire_path

    all_image_meta = ceph_api.read_image_metadata(image_name)

  File "/usr/lib/python2.7/dist-packages/PetaSAN/core/ceph/api.py", line 182, in read_image_metadata

    raise MetadataException("Cannot get metadata.")

MetadataException: Cannot get metadata.

25/04/2018 08:15:23 ERROR    Error during process.

25/04/2018 08:15:23 ERROR    Cannot get metadata.

Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 98, in start

    self.__process()

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 132, in __process

    while self.__do_process() != True:

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 205, in __do_process

    self.__acquire_path(str(path), self.__paths_consul_unlocked_firstborn.get(path))

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 523, in __acquire_path

    raise e

MetadataException: Cannot get metadata.

25/04/2018 08:15:38 ERROR    Could not acquire path 00001/1

Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 439, in __acquire_path

    all_image_meta = ceph_api.read_image_metadata(image_name)

  File "/usr/lib/python2.7/dist-packages/PetaSAN/core/ceph/api.py", line 182, in read_image_metadata

    raise MetadataException("Cannot get metadata.")

MetadataException: Cannot get metadata.

25/04/2018 08:15:38 ERROR    Error during process.

25/04/2018 08:15:38 ERROR    Cannot get metadata.

Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 98, in start

    self.__process()

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 132, in __process

    while self.__do_process() != True:

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 205, in __do_process

    self.__acquire_path(str(path), self.__paths_consul_unlocked_firstborn.get(path))

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 523, in __acquire_path

    raise e

MetadataException: Cannot get metadata.

25/04/2018 08:15:56 ERROR    Could not acquire path 00001/1

Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 439, in __acquire_path

    all_image_meta = ceph_api.read_image_metadata(image_name)

  File "/usr/lib/python2.7/dist-packages/PetaSAN/core/ceph/api.py", line 182, in read_image_metadata

    raise MetadataException("Cannot get metadata.")

MetadataException: Cannot get metadata.

25/04/2018 08:15:56 ERROR    Error during process.

25/04/2018 08:15:56 ERROR    Cannot get metadata.

Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 98, in start

    self.__process()

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 132, in __process

    while self.__do_process() != True:

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 205, in __do_process

    self.__acquire_path(str(path), self.__paths_consul_unlocked_firstborn.get(path))

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 523, in __acquire_path

    raise e

MetadataException: Cannot get metadata.

25/04/2018 08:16:15 ERROR    Could not acquire path 00001/1

Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 439, in __acquire_path

    all_image_meta = ceph_api.read_image_metadata(image_name)

  File "/usr/lib/python2.7/dist-packages/PetaSAN/core/ceph/api.py", line 182, in read_image_metadata

    raise MetadataException("Cannot get metadata.")

MetadataException: Cannot get metadata.

25/04/2018 08:16:15 ERROR    Error during process.

25/04/2018 08:16:15 ERROR    Cannot get metadata.

Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 98, in start

    self.__process()

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 132, in __process

    while self.__do_process() != True:

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 205, in __do_process

    self.__acquire_path(str(path), self.__paths_consul_unlocked_firstborn.get(path))

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 523, in __acquire_path

    raise e

MetadataException: Cannot get metadata.

25/04/2018 08:16:36 ERROR    Could not acquire path 00001/1

Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 439, in __acquire_path

    all_image_meta = ceph_api.read_image_metadata(image_name)

  File "/usr/lib/python2.7/dist-packages/PetaSAN/core/ceph/api.py", line 182, in read_image_metadata

    raise MetadataException("Cannot get metadata.")

MetadataException: Cannot get metadata.

25/04/2018 08:16:36 ERROR    Error during process.

25/04/2018 08:16:36 ERROR    Cannot get metadata.

Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 98, in start

    self.__process()

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 132, in __process

    while self.__do_process() != True:

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 205, in __do_process

    self.__acquire_path(str(path), self.__paths_consul_unlocked_firstborn.get(path))

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 523, in __acquire_path

    raise e

root@petasan-01:/opt/petasan/log# tail -n 200 PetaSAN.log.1

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 98, in start

    self.__process()

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 132, in __process

    while self.__do_process() != True:

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 205, in __do_process

    self.__acquire_path(str(path), self.__paths_consul_unlocked_firstborn.get(path))

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 523, in __acquire_path

    raise e

MetadataException: Cannot get metadata.

26/04/2018 13:45:01 ERROR    Could not acquire path 00001/2

Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 439, in __acquire_path

    all_image_meta = ceph_api.read_image_metadata(image_name)

  File "/usr/lib/python2.7/dist-packages/PetaSAN/core/ceph/api.py", line 182, in read_image_metadata

    raise MetadataException("Cannot get metadata.")

MetadataException: Cannot get metadata.

26/04/2018 13:45:01 ERROR    Error during process.

26/04/2018 13:45:01 ERROR    Cannot get metadata.

Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 98, in start

    self.__process()

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 132, in __process

    while self.__do_process() != True:

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 205, in __do_process

    self.__acquire_path(str(path), self.__paths_consul_unlocked_firstborn.get(path))

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 523, in __acquire_path

    raise e

MetadataException: Cannot get metadata.

26/04/2018 13:45:12 ERROR    Could not acquire path 00001/1

Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 439, in __acquire_path

    all_image_meta = ceph_api.read_image_metadata(image_name)

  File "/usr/lib/python2.7/dist-packages/PetaSAN/core/ceph/api.py", line 182, in read_image_metadata

    raise MetadataException("Cannot get metadata.")

MetadataException: Cannot get metadata.

26/04/2018 13:45:12 ERROR    Error during process.

26/04/2018 13:45:12 ERROR    Cannot get metadata.

Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 98, in start

    self.__process()

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 132, in __process

    while self.__do_process() != True:

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 205, in __do_process

    self.__acquire_path(str(path), self.__paths_consul_unlocked_firstborn.get(path))

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 523, in __acquire_path

    raise e

MetadataException: Cannot get metadata.

26/04/2018 13:45:25 ERROR    Could not acquire path 00001/2

Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 439, in __acquire_path

    all_image_meta = ceph_api.read_image_metadata(image_name)

  File "/usr/lib/python2.7/dist-packages/PetaSAN/core/ceph/api.py", line 182, in read_image_metadata

    raise MetadataException("Cannot get metadata.")

MetadataException: Cannot get metadata.

26/04/2018 13:45:25 ERROR    Error during process.

26/04/2018 13:45:25 ERROR    Cannot get metadata.

Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 98, in start

    self.__process()

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 132, in __process

    while self.__do_process() != True:

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 205, in __do_process

    self.__acquire_path(str(path), self.__paths_consul_unlocked_firstborn.get(path))

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 523, in __acquire_path

    raise e

MetadataException: Cannot get metadata.

26/04/2018 13:45:40 ERROR    Could not acquire path 00001/1

Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 439, in __acquire_path

    all_image_meta = ceph_api.read_image_metadata(image_name)

  File "/usr/lib/python2.7/dist-packages/PetaSAN/core/ceph/api.py", line 182, in read_image_metadata

    raise MetadataException("Cannot get metadata.")

MetadataException: Cannot get metadata.

26/04/2018 13:45:40 ERROR    Error during process.

26/04/2018 13:45:40 ERROR    Cannot get metadata.

Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 98, in start

    self.__process()

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 132, in __process

    while self.__do_process() != True:

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 205, in __do_process

    self.__acquire_path(str(path), self.__paths_consul_unlocked_firstborn.get(path))

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 523, in __acquire_path

    raise e

MetadataException: Cannot get metadata.

26/04/2018 13:45:58 ERROR    Could not acquire path 00001/2

Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 439, in __acquire_path

    all_image_meta = ceph_api.read_image_metadata(image_name)

  File "/usr/lib/python2.7/dist-packages/PetaSAN/core/ceph/api.py", line 182, in read_image_metadata

    raise MetadataException("Cannot get metadata.")

MetadataException: Cannot get metadata.

26/04/2018 13:45:58 ERROR    Error during process.

26/04/2018 13:45:58 ERROR    Cannot get metadata.

Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 98, in start

    self.__process()

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 132, in __process

    while self.__do_process() != True:

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 205, in __do_process

    self.__acquire_path(str(path), self.__paths_consul_unlocked_firstborn.get(path))

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 523, in __acquire_path

    raise e

MetadataException: Cannot get metadata.

26/04/2018 13:46:17 ERROR    Could not acquire path 00001/1

Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 439, in __acquire_path

    all_image_meta = ceph_api.read_image_metadata(image_name)

  File "/usr/lib/python2.7/dist-packages/PetaSAN/core/ceph/api.py", line 182, in read_image_metadata

    raise MetadataException("Cannot get metadata.")

MetadataException: Cannot get metadata.

26/04/2018 13:46:17 ERROR    Error during process.

26/04/2018 13:46:17 ERROR    Cannot get metadata.

Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 98, in start

    self.__process()

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 132, in __process

    while self.__do_process() != True:

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 205, in __do_process

    self.__acquire_path(str(path), self.__paths_consul_unlocked_firstborn.get(path))

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 523, in __acquire_path

    raise e

MetadataException: Cannot get metadata.

26/04/2018 13:46:38 ERROR    Could not acquire path 00001/1

Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 439, in __acquire_path

    all_image_meta = ceph_api.read_image_metadata(image_name)

  File "/usr/lib/python2.7/dist-packages/PetaSAN/core/ceph/api.py", line 182, in read_image_metadata

    raise MetadataException("Cannot get metadata.")

MetadataException: Cannot get metadata.

26/04/2018 13:46:38 ERROR    Error during process.

26/04/2018 13:46:38 ERROR    Cannot get metadata.

Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 98, in start

    self.__process()

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 132, in __process

    while self.__do_process() != True:

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 205, in __do_process

    self.__acquire_path(str(path), self.__paths_consul_unlocked_firstborn.get(path))

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 523, in __acquire_path

    raise e

MetadataException: Cannot get metadata.

26/04/2018 13:47:01 ERROR    Could not acquire path 00001/1

Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 439, in __acquire_path

    all_image_meta = ceph_api.read_image_metadata(image_name)

  File "/usr/lib/python2.7/dist-packages/PetaSAN/core/ceph/api.py", line 182, in read_image_metadata

    raise MetadataException("Cannot get metadata.")

MetadataException: Cannot get metadata.

26/04/2018 13:47:01 ERROR    Error during process.

26/04/2018 13:47:01 ERROR    Cannot get metadata.

Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 98, in start

    self.__process()

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 132, in __process

    while self.__do_process() != True:

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 205, in __do_process

    self.__acquire_path(str(path), self.__paths_consul_unlocked_firstborn.get(path))

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 523, in __acquire_path

    raise e

MetadataException: Cannot get metadata.

26/04/2018 13:47:12 WARNING  PetaSAN could not complete process, there are too many exceptions.

26/04/2018 13:47:16 ERROR    Could not acquire path 00001/2

Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 439, in __acquire_path

    all_image_meta = ceph_api.read_image_metadata(image_name)

  File "/usr/lib/python2.7/dist-packages/PetaSAN/core/ceph/api.py", line 182, in read_image_metadata

    raise MetadataException("Cannot get metadata.")

MetadataException: Cannot get metadata.

26/04/2018 13:47:16 ERROR    Error during process.

26/04/2018 13:47:16 ERROR    Cannot get metadata.

Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 98, in start

    self.__process()

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 132, in __process

    while self.__do_process() != True:

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 205, in __do_process

    self.__acquire_path(str(path), self.__paths_consul_unlocked_firstborn.get(path))

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 523, in __acquire_path

    raise e

MetadataException: Cannot get metadata.

26/04/2018 13:47:23 ERROR    Could not acquire path 00001/1

Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 439, in __acquire_path

    all_image_meta = ceph_api.read_image_metadata(image_name)

  File "/usr/lib/python2.7/dist-packages/PetaSAN/core/ceph/api.py", line 182, in read_image_metadata

    raise MetadataException("Cannot get metadata.")

MetadataException: Cannot get metadata.

26/04/2018 13:47:23 ERROR    Error during process.

26/04/2018 13:47:23 ERROR    Cannot get metadata.

Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 98, in start

    self.__process()

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 132, in __process

    while self.__do_process() != True:

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 205, in __do_process

    self.__acquire_path(str(path), self.__paths_consul_unlocked_firstborn.get(path))

  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 523, in __acquire_path

    raise e

MetadataException: Cannot get metadata.

Some stats from Petasan 01

cpu 

Mem

Disk utilize

Net Ulitilize

Petasan 02

cpu

mem

disk utilize

net utilize

#3

admin
2,930 Posts
April 26, 2018, 6:16 am

 File "/usr/lib/python2.7/dist-packages/PetaSAN/core/ceph/api.py", line 182, in read_image_metadata

    raise MetadataException("Cannot get metadata.")

MetadataException: Cannot get metadata.

This means the iSCSI service failed to read data from Ceph (on backend 1), but i also notice node 2 ram utilization is high. cpu and disks are not loaded.

Last edited on April 26, 2018, 6:16 am by admin · #4

emanzx
11 Posts
May 14, 2018, 2:50 am

Hi, sorry for taking so long to update. After upgrading the RAM all the issues has been resolve. I should have been more detail on requirement for the iSCSI. Thanks for the help. This issues is resolved.

#5

admin
2,930 Posts
May 14, 2018, 11:09 am

thanks for the feedback. fyi we do have a recommended hardware guide.

#6