Forums

Home / Forums

You need to log in to create posts and topics. Login · Register

2.6 upgrade and monitor node failure not sure if related


R3LZX
50 Posts
August 4, 2020, 9:53 am
smtp alerts show this node went offline for a short period of time I was wondering if you could take a look at these logs, happened from 01:45-01:56 this is a five server cluster
logs from node 1

04/08/2020 04:45:40 INFO GlusterFS mount attempt 

duplicate line Through 

04/08/2020 01:58:07 INFO GlusterFS mount attempt 

04/08/2020 01:57:52 INFO Cleaning unused ips.

04/08/2020 01:57:51 INFO Cleaning unused rbd images.

04/08/2020 01:57:51 INFO Cleaning all mapped disks

04/08/2020 01:57:51 INFO Cleaning unused configurations. 

04/08/2020 01:57:51 WARNING There are too many exceptions.Service will clean this node.

04/08/2020 01:57:50 WARNING PetaSAN could not complete process, there are too many exceptions.

ConsulException: 500 rpc error making call: invalid session "b9873932-4eb7-1255-2137-757de490fc3b"

raise e

File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 641, in __acquire_path

self.__acquire_path(str(path), self.__paths_consul_unlocked_firstborn.get(path))

File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 269, in __do_process

while self.__do_process() != True:

File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 196, in __process

self.__process()

File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 161, in start

Traceback (most recent call last):

04/08/2020 01:57:39 ERROR 500 rpc error making call: invalid session "b9873932-4eb7-1255-2137-757de490fc3b"

04/08/2020 01:57:39 ERROR Error during process.

ConsulException: 500 rpc error making call: invalid session "b9873932-4eb7-1255-2137-757de490fc3b"

raise ConsulException("%d %s" % (response.code, response.body))

File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 188, in _status

CB._status(response, allow_404=allow_404)

File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 223, in cb

return callback(res)

File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/ps_consul.py", line 83, in put

attempt = Attempt(fn(*args, **kwargs), attempt_number, False)

File "/usr/lib/python2.7/dist-packages/retrying.py", line 200, in call

six.reraise(self.value[0], self.value[1], self.value[2])

File "/usr/lib/python2.7/dist-packages/retrying.py", line 247, in get

raise attempt.get()

File "/usr/lib/python2.7/dist-packages/retrying.py", line 212, in call

return Retrying(*dargs, **dkw).call(f, *args, **kw)

File "/usr/lib/python2.7/dist-packages/retrying.py", line 49, in wrapped_f

CB.json(), '/v1/kv/%s' % key, params=params, data=value)

File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 621, in put

result = consul_obj.kv.put(path, data, None, None, str(session))

File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/api.py", line 91, in lock_disk_path

str(consul_kv.CreateIndex))

File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 563, in __acquire_path

Traceback (most recent call last):

04/08/2020 01:57:39 ERROR Could not acquire path 00001/5

04/08/2020 01:57:39 INFO Cleaning unused ips.

04/08/2020 01:57:39 INFO Cleaning unused rbd images.

04/08/2020 01:57:39 INFO Cleaning all mapped disks

04/08/2020 01:57:39 INFO Cleaning unused configurations. 

04/08/2020 01:57:39 INFO Cleaning all mapped disks from old session

04/08/2020 01:57:39 INFO New session id is f2067b14-90be-1f1b-beb5-701fb0340d32

04/08/2020 01:57:39 INFO Trying to create new session id

04/08/2020 01:57:39 ERROR Session is invalid

ConsulException: 500 rpc error making call: invalid session "b9873932-4eb7-1255-2137-757de490fc3b"

raise ConsulException("%d %s" % (response.code, response.body))

File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 188, in _status

CB._status(response, allow_404=allow_404)

File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 223, in cb

return callback(res)

File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/ps_consul.py", line 83, in put

attempt = Attempt(fn(*args, **kwargs), attempt_number, False)

File "/usr/lib/python2.7/dist-packages/retrying.py", line 200, in call

six.reraise(self.value[0], self.value[1], self.value[2])

File "/usr/lib/python2.7/dist-packages/retrying.py", line 247, in get

raise attempt.get()

File "/usr/lib/python2.7/dist-packages/retrying.py", line 212, in call

return Retrying(*dargs, **dkw).call(f, *args, **kw)

File "/usr/lib/python2.7/dist-packages/retrying.py", line 49, in wrapped_f

CB.json(), '/v1/kv/%s' % key, params=params, data=value)

File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 621, in put

result = consul_obj.kv.put(path, data, None, None, str(session))

File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/api.py", line 91, in lock_disk_path

str(consul_kv.CreateIndex))

File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 563, in __acquire_path

Traceback (most recent call last):

04/08/2020 01:57:39 ERROR 500 rpc error making call: invalid session "b9873932-4eb7-1255-2137-757de490fc3b"

04/08/2020 01:57:39 ERROR 500 rpc error making call: invalid session "b9873932-4eb7-1255-2137-757de490fc3b"

04/08/2020 01:57:39 INFO ---------------------------------

04/08/2020 01:57:37 INFO GlusterFS mount attempt 

04/08/2020 01:57:13 INFO LeaderElectionBase new session id created 9771c3fc-fd55-7c3a-edaa-8e3382303f0f

04/08/2020 01:57:13 INFO ClusterLeader stop action

04/08/2020 01:57:13 ERROR LeaderElectionBase session is invalid

04/08/2020 01:57:08 ERROR 500 rpc error making call: invalid session "3936a9fa-81ae-eec9-35d7-446a0f8c911a"

04/08/2020 01:57:08 ERROR LeaderElectionBase exception

04/08/2020 01:57:07 INFO GlusterFS mount attempt 

04/08/2020 01:56:37 INFO GlusterFS mount attempt 

04/08/2020 01:56:07 INFO GlusterFS mount attempt 

04/08/2020 01:55:37 INFO GlusterFS mount attempt 

04/08/2020 01:55:33 INFO Found pool:rbd for disk:00001 via consul

04/08/2020 01:55:07 INFO GlusterFS mount attempt 

04/08/2020 01:54:37 INFO GlusterFS mount attempt 

RetryConsulException

raise RetryConsulException()

File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/ps_consul.py", line 71, in get

attempt = Attempt(fn(*args, **kwargs), attempt_number, False)

File "/usr/lib/python2.7/dist-packages/retrying.py", line 200, in call

six.reraise(self.value[0], self.value[1], self.value[2])

File "/usr/lib/python2.7/dist-packages/retrying.py", line 247, in get

raise attempt.get()

File "/usr/lib/python2.7/dist-packages/retrying.py", line 212, in call

return Retrying(*dargs, **dkw).call(f, *args, **kw)

File "/usr/lib/python2.7/dist-packages/retrying.py", line 49, in wrapped_f

params=params)

File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 554, in get

index, data = consul_obj.kv.get(key, lock_index, False, timeout)

File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/api.py", line 497, in get_key_blocking

Traceback (most recent call last):

04/08/2020 01:54:33 ERROR 

04/08/2020 01:54:33 ERROR Could not find key resource.

PetaSAN.NodeStats.DL360ONE.disks.throughput.sde_write 0.0 `date +%s`" | nc -q0 10.10.21.42 2003

PetaSAN.NodeStats.DL360ONE.disks.percent_util.sde 0.0 `date +%s`" "

PetaSAN.NodeStats.DL360ONE.disks.iops_all.sda 0.0 `date +%s`" "

PetaSAN.NodeStats.DL360ONE.disks.throughput.sda_read 0.0 `date +%s`" "

PetaSAN.NodeStats.DL360ONE.disks.throughput.sda_write 0.0 `date +%s`" "

PetaSAN.NodeStats.DL360ONE.disks.percent_util.sda 0.0 `date +%s`" "

PetaSAN.NodeStats.DL360ONE.disks.iops_all.sdb 0.0 `date +%s`" "

PetaSAN.NodeStats.DL360ONE.disks.throughput.sdb_read 0.0 `date +%s`" "

PetaSAN.NodeStats.DL360ONE.disks.throughput.sdb_write 0.0 `date +%s`" "

Exception: Error running echo command :echo "PetaSAN.NodeStats.DL360ONE.disks.percent_util.sdb 0.0 `date +%s`" "

raise Exception("Error running echo command :" + cmd)

File "/usr/lib/python2.7/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send

graphite_sender.send(leader_ip)

File "/opt/petasan/scripts/node_stats.py", line 64, in get_stats

get_stats()

File "/opt/petasan/scripts/node_stats.py", line 159, in <module>

Traceback (most recent call last):

PetaSAN.NodeStats.DL360ONE.disks.throughput.sde_write 0.0 `date +%s`" | nc -q0 10.10.21.42 2003

PetaSAN.NodeStats.DL360ONE.disks.percent_util.sde 0.0 `date +%s`" "

PetaSAN.NodeStats.DL360ONE.disks.iops_all.sda 0.0 `date +%s`" "

PetaSAN.NodeStats.DL360ONE.disks.throughput.sda_read 0.0 `date +%s`" "

PetaSAN.NodeStats.DL360ONE.disks.throughput.sda_write 0.0 `date +%s`" "

PetaSAN.NodeStats.DL360ONE.disks.percent_util.sda 0.0 `date +%s`" "

PetaSAN.NodeStats.DL360ONE.disks.iops_all.sdb 0.0 `date +%s`" "

PetaSAN.NodeStats.DL360ONE.disks.throughput.sdb_read 0.0 `date +%s`" "

PetaSAN.NodeStats.DL360ONE.disks.throughput.sdb_write 0.0 `date +%s`" "

04/08/2020 01:54:27 ERROR Error running echo command :echo "PetaSAN.NodeStats.DL360ONE.disks.percent_util.sdb 0.0 `date +%s`" "

04/08/2020 01:54:27 ERROR Node Stats exception.

RetryConsulException

raise RetryConsulException()

File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/ps_consul.py", line 71, in get

attempt = Attempt(fn(*args, **kwargs), attempt_number, False)

File "/usr/lib/python2.7/dist-packages/retrying.py", line 200, in call

six.reraise(self.value[0], self.value[1], self.value[2])

File "/usr/lib/python2.7/dist-packages/retrying.py", line 247, in get

raise attempt.get()

File "/usr/lib/python2.7/dist-packages/retrying.py", line 212, in call

return Retrying(*dargs, **dkw).call(f, *args, **kw)

File "/usr/lib/python2.7/dist-packages/retrying.py", line 49, in wrapped_f

params=params)

File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 554, in get

index, data = cons.kv.get(key)

File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/base.py", line 29, in read_value

_exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"]))

File "/usr/lib/python2.7/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session

return self.session_interface.open_session(self, request)

File "/usr/lib/python2.7/dist-packages/flask/app.py", line 912, in open_session

self.session = self.app.open_session(self.request)

File "/usr/lib/python2.7/dist-packages/flask/ctx.py", line 332, in push

ctx.push()

File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1978, in wsgi_app

return self.wsgi_app(environ, start_response)

File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1997, in __call__

application_iter = app(environ, start_response)

File "/usr/lib/python2.7/dist-packages/werkzeug/serving.py", line 258, in execute

execute(self.server.app)

File "/usr/lib/python2.7/dist-packages/werkzeug/serving.py", line 270, in run_wsgi

Traceback (most recent call last):

04/08/2020 01:54:11 ERROR Error on request:

RetryConsulException

raise e

File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 358, in __read_resources_consul

self.__read_resources_consul()

File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 214, in __do_process

while self.__do_process() != True:

File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 196, in __process

self.__process()

File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 161, in start

Traceback (most recent call last):

04/08/2020 01:54:10 ERROR 

04/08/2020 01:54:10 ERROR Error on consul connection.

RetryConsulException

raise RetryConsulException()

File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/ps_consul.py", line 71, in get

attempt = Attempt(fn(*args, **kwargs), attempt_number, False)

File "/usr/lib/python2.7/dist-packages/retrying.py", line 200, in call

six.reraise(self.value[0], self.value[1], self.value[2])

File "/usr/lib/python2.7/dist-packages/retrying.py", line 247, in get

raise attempt.get()

File "/usr/lib/python2.7/dist-packages/retrying.py", line 212, in call

return Retrying(*dargs, **dkw).call(f, *args, **kw)

File "/usr/lib/python2.7/dist-packages/retrying.py", line 49, in wrapped_f

CB.json(index=True), '/v1/session/list', params=params)

File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 1825, in list

for sess in consul_obj.session.list()[1]:

File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/api.py", line 376, in get_sessions_dict

self.__session_dict = ConsulAPI().get_sessions_dict(ConfigAPI().get_iscsi_service_session_name())

File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/mange_path_assignment.py", line 40, in __init__

reassignments = MangePathAssignment().get_forced_paths()

File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 338, in __read_resources_consul

Traceback (most recent call last):

04/08/2020 01:54:10 ERROR 

04/08/2020 01:54:10 ERROR Could not read consul resources.

04/08/2020 01:54:06 INFO GlusterFS mount attempt 

04/08/2020 01:53:36 INFO GlusterFS mount attempt 

04/08/2020 01:53:06 INFO GlusterFS mount attempt 

RetryConsulException

raise RetryConsulException()

File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/ps_consul.py", line 71, in get

attempt = Attempt(fn(*args, **kwargs), attempt_number, False)

File "/usr/lib/python2.7/dist-packages/retrying.py", line 200, in call

six.reraise(self.value[0], self.value[1], self.value[2])

File "/usr/lib/python2.7/dist-packages/retrying.py", line 247, in get

raise attempt.get()

File "/usr/lib/python2.7/dist-packages/retrying.py", line 212, in call

return Retrying(*dargs, **dkw).call(f, *args, **kw)

File "/usr/lib/python2.7/dist-packages/retrying.py", line 49, in wrapped_f

params=params)

File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 554, in get

index, data = cons.kv.get(key)

File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/base.py", line 29, in read_value

_exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"]))

File "/usr/lib/python2.7/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session

return self.session_interface.open_session(self, request)

File "/usr/lib/python2.7/dist-packages/flask/app.py", line 912, in open_session

self.session = self.app.open_session(self.request)

File "/usr/lib/python2.7/dist-packages/flask/ctx.py", line 332, in push

ctx.push()

File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1978, in wsgi_app

return self.wsgi_app(environ, start_response)

File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1997, in __call__

application_iter = app(environ, start_response)

File "/usr/lib/python2.7/dist-packages/werkzeug/serving.py", line 258, in execute

execute(self.server.app)

File "/usr/lib/python2.7/dist-packages/werkzeug/serving.py", line 270, in run_wsgi

Traceback (most recent call last):

04/08/2020 01:52:56 ERROR Error on request:

RetryConsulException

raise RetryConsulException()

File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/ps_consul.py", line 71, in get

attempt = Attempt(fn(*args, **kwargs), attempt_number, False)

File "/usr/lib/python2.7/dist-packages/retrying.py", line 200, in call

six.reraise(self.value[0], self.value[1], self.value[2])

File "/usr/lib/python2.7/dist-packages/retrying.py", line 247, in get

raise attempt.get()

File "/usr/lib/python2.7/dist-packages/retrying.py", line 212, in call

return Retrying(*dargs, **dkw).call(f, *args, **kw)

File "/usr/lib/python2.7/dist-packages/retrying.py", line 49, in wrapped_f

params=params)

File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 554, in get

return cons.kv.get(key, index=current_index, recurse=True)

File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/base.py", line 49, in watch

index, data = base.watch(self.root_path, current_index)

File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/file_sync_manager.py", line 75, in sync

Traceback (most recent call last):

04/08/2020 01:52:41 ERROR 

04/08/2020 01:52:36 INFO GlusterFS mount attempt 

04/08/2020 01:52:06 INFO GlusterFS mount attempt 

RetryConsulException

raise RetryConsulException()

File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/ps_consul.py", line 82, in put

attempt = Attempt(fn(*args, **kwargs), attempt_number, False)

File "/usr/lib/python2.7/dist-packages/retrying.py", line 200, in call

six.reraise(self.value[0], self.value[1], self.value[2])

File "/usr/lib/python2.7/dist-packages/retrying.py", line 247, in get

raise attempt.get()

File "/usr/lib/python2.7/dist-packages/retrying.py", line 212, in call

return Retrying(*dargs, **dkw).call(f, *args, **kw)

File "/usr/lib/python2.7/dist-packages/retrying.py", line 49, in wrapped_f

CB.json(), '/v1/kv/%s' % key, params=params, data=value)

File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 621, in put

cons.kv.put(key, value)

File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/base.py", line 36, in write_value

consul_base_api.write_value("".join( [consul_session_key,self.sid,"/",key]),str(value))

File "/usr/lib/python2.7/dist-packages/PetaSAN/core/security/session.py", line 53, in __setitem__

self['_permanent'] = bool(value)

File "/usr/lib/python2.7/dist-packages/flask/sessions.py", line 34, in _set_permanent

self.permanent= True

File "/usr/lib/python2.7/dist-packages/PetaSAN/core/security/session.py", line 39, in __init__

return ConsulSession(sid,"")

File "/usr/lib/python2.7/dist-packages/PetaSAN/core/security/session.py", line 134, in open_session

return self.session_interface.open_session(self, request)

File "/usr/lib/python2.7/dist-packages/flask/app.py", line 912, in open_session

self.session = self.app.open_session(self.request)

File "/usr/lib/python2.7/dist-packages/flask/ctx.py", line 332, in push

ctx.push()

File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1978, in wsgi_app

return self.wsgi_app(environ, start_response)

File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1997, in __call__

application_iter = app(environ, start_response)

File "/usr/lib/python2.7/dist-packages/werkzeug/serving.py", line 258, in execute

execute(self.server.app)

File "/usr/lib/python2.7/dist-packages/werkzeug/serving.py", line 270, in run_wsgi

Traceback (most recent call last):

04/08/2020 01:52:04 ERROR Error on request:

04/08/2020 01:51:36 INFO GlusterFS mount attempt 

04/08/2020 01:51:06 INFO GlusterFS mount attempt 

RetryConsulException

raise e

File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/api.py", line 349, in current_index

self.__current_lock_index = consul_api.current_index()

File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 157, in start

Traceback (most recent call last):

04/08/2020 01:50:40 ERROR 

04/08/2020 01:50:40 ERROR Error on consul connection.

04/08/2020 01:50:38 ERROR Error on consul leader.

RetryConsulException

raise RetryConsulException()

File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/ps_consul.py", line 82, in put

attempt = Attempt(fn(*args, **kwargs), attempt_number, False)

File "/usr/lib/python2.7/dist-packages/retrying.py", line 200, in call

six.reraise(self.value[0], self.value[1], self.value[2])

File "/usr/lib/python2.7/dist-packages/retrying.py", line 247, in get

raise attempt.get()

File "/usr/lib/python2.7/dist-packages/retrying.py", line 212, in call

return Retrying(*dargs, **dkw).call(f, *args, **kw)

File "/usr/lib/python2.7/dist-packages/retrying.py", line 49, in wrapped_f

CB.json(), '/v1/kv/%s' % key, params=params, data=value)

File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 621, in put

cons.kv.put(key, value)

File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/base.py", line 36, in write_value

consul_base_api.write_value("".join( [consul_session_key,self.sid,"/",key]),str(value))

File "/usr/lib/python2.7/dist-packages/PetaSAN/core/security/session.py", line 53, in __setitem__

self['_permanent'] = bool(value)

File "/usr/lib/python2.7/dist-packages/flask/sessions.py", line 34, in _set_permanent

self.permanent= True

File "/usr/lib/python2.7/dist-packages/PetaSAN/core/security/session.py", line 39, in __init__

return ConsulSession(sid,"")

File "/usr/lib/python2.7/dist-packages/PetaSAN/core/security/session.py", line 134, in open_session

return self.session_interface.open_session(self, request)

File "/usr/lib/python2.7/dist-packages/flask/app.py", line 912, in open_session

self.session = self.app.open_session(self.request)

File "/usr/lib/python2.7/dist-packages/flask/ctx.py", line 332, in push

ctx.push()

File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1978, in wsgi_app

return self.wsgi_app(environ, start_response)

File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1997, in __call__

application_iter = app(environ, start_response)

File "/usr/lib/python2.7/dist-packages/werkzeug/serving.py", line 258, in execute

execute(self.server.app)

File "/usr/lib/python2.7/dist-packages/werkzeug/serving.py", line 270, in run_wsgi

Traceback (most recent call last):

04/08/2020 01:50:36 ERROR Error on request:

04/08/2020 01:50:36 INFO GlusterFS mount attempt 

duplicate line Through 

04/08/2020 00:00:14 INFO GlusterFS mount attempt 

logs from mon 3 (this is the one I lost contact with)

04/08/2020 01:57:12 INFO LeaderElectionBase new session id created 0181e70b-af55-e8c0-709d-c5f09f71e648

04/08/2020 01:57:12 INFO ClusterLeader stop action

04/08/2020 01:57:12 ERROR LeaderElectionBase session is invalid

04/08/2020 01:57:07 ERROR 500 rpc error making call: invalid session "58753fe0-c1c4-c576-9599-3621c0761066"

04/08/2020 01:57:07 ERROR LeaderElectionBase exception

RetryConsulException

raise RetryConsulException()

File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/ps_consul.py", line 71, in get

attempt = Attempt(fn(*args, **kwargs), attempt_number, False)

File "/usr/lib/python2.7/dist-packages/retrying.py", line 200, in call

six.reraise(self.value[0], self.value[1], self.value[2])

File "/usr/lib/python2.7/dist-packages/retrying.py", line 247, in get

raise attempt.get()

File "/usr/lib/python2.7/dist-packages/retrying.py", line 212, in call

return Retrying(*dargs, **dkw).call(f, *args, **kw)

File "/usr/lib/python2.7/dist-packages/retrying.py", line 49, in wrapped_f

params=params)

File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 554, in get

File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/api.py", line 429, in get_key_blocking

Traceback (most recent call last):

04/08/2020 01:54:32 ERROR 

04/08/2020 01:54:32 ERROR Could not find key resource.

#1

admin
2,930 Posts
August 4, 2020, 11:38 am

it looks more a network connection issue. there are multiple services connection errors from gluster, stats collection, consul. these are unrelated systems.

#2