Error when add osd with dm-writecache
zamasaki158
13 Posts
June 17, 2020, 10:33 amQuote from zamasaki158 on June 17, 2020, 10:33 amHi Support Team,
I got error when add osd with dm-writecache. My cluster running version 2.5.3
The logs:
15/06/2020 22:41:56 INFO script
15/06/2020 22:41:56 INFO /opt/petasan/scripts/admin/node_manage_disks.py add-osd
15/06/2020 22:41:56 INFO params
15/06/2020 22:41:56 INFO -disk_name sdf -cache sdc -cache_type writecache
15/06/2020 22:41:56 INFO Start add osd job for disk sdf.
15/06/2020 22:42:02 INFO Start cleaning disk : sdf
15/06/2020 22:42:02 INFO Starting : vgchange -a n ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.16
15/06/2020 22:42:06 INFO Executing : wipefs --all /dev/sdf1
15/06/2020 22:42:06 INFO Executing : dd if=/dev/zero of=/dev/sdf1 bs=1M count=20 oflag=direct,dsync >/dev/null 2>&1
15/06/2020 22:42:06 INFO Executing : wipefs --all /dev/sdf
15/06/2020 22:42:06 INFO Executing : dd if=/dev/zero of=/dev/sdf bs=1M count=20 oflag=direct,dsync >/dev/null 2>&1
15/06/2020 22:42:06 INFO Executing : parted -s /dev/sdf mklabel gpt
15/06/2020 22:42:07 INFO Executing : partprobe /dev/sdf
15/06/2020 22:42:10 INFO User didn't select a journal for disk sdf, so the journal will be on the same disk.
15/06/2020 22:42:15 INFO User selected cache sdc disk for disk sdf.
15/06/2020 22:42:15 INFO Getting the next OSD ID :
15/06/2020 22:42:15 INFO -------------------------
15/06/2020 22:42:16 INFO Start remove osd.16 from crush map
15/06/2020 22:42:17 ERROR Error executing ceph osd out osd.16
15/06/2020 22:42:18 ERROR Error executing ceph osd crush remove osd.16
15/06/2020 22:42:18 ERROR Error executing ceph auth del osd.16
15/06/2020 22:42:19 WARNING The osd still up you need to stop osd service of osd.16
15/06/2020 22:42:19 WARNING osd.16 is removed from crush map
15/06/2020 22:42:19 INFO next_osd_id = 16
15/06/2020 22:42:19 INFO vg_name = ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.16
15/06/2020 22:42:19 INFO ==================================================
15/06/2020 22:42:19 INFO ===== Building DM-Writecache =====
15/06/2020 22:42:19 INFO ==================================================
15/06/2020 22:42:19 INFO Step 1 : Preparing Slow Disk :
15/06/2020 22:42:19 INFO ------------------------------
15/06/2020 22:42:19 INFO Creating data partition num 1 size 0 on /dev/sdf
15/06/2020 22:42:21 INFO Calling partprobe on sdf device
15/06/2020 22:42:21 INFO Executing partprobe /dev/sdf
15/06/2020 22:42:21 INFO Calling udevadm on sdf device
15/06/2020 22:42:21 INFO Executing udevadm settle --timeout 30
15/06/2020 22:42:24 INFO origin_part_name = /dev/sdf1
15/06/2020 22:42:24 INFO done
15/06/2020 22:42:24 INFO Step 2 : Preparing Fast Disk :
15/06/2020 22:42:24 INFO ------------------------------
15/06/2020 22:42:24 INFO Start cleaning disk : sdc1
15/06/2020 22:42:24 INFO Starting : vgchange -a n ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.16
15/06/2020 22:42:24 INFO Executing : umount /dev/sdc1
15/06/2020 22:42:24 ERROR Error executing : umount /dev/sdc1
15/06/2020 22:42:24 INFO Executing : wipefs --all /dev/sdc1
15/06/2020 22:42:24 INFO Executing : dd if=/dev/zero of=/dev/sdc1 bs=1M count=20 oflag=direct,dsync >/dev/null 2>&1
15/06/2020 22:42:24 INFO Executing : dd bs=1M seek=2324783104 if=/dev/zero of=/dev/sdc1 count=20 oflag=seek_bytes,direct,dsync >/dev/null 2>&1
15/06/2020 22:42:24 INFO cache_part_path = /dev/sdc1
15/06/2020 22:42:24 INFO done
15/06/2020 22:42:24 INFO Step 3 : Creating Physical Volumes :
15/06/2020 22:42:24 INFO ------------------------------------
15/06/2020 22:42:25 INFO done
15/06/2020 22:42:25 INFO Step 4 : Creating Volume Group :
15/06/2020 22:42:25 INFO --------------------------------
15/06/2020 22:42:25 INFO Running command: /sbin/vgs --noheadings --readonly --units=g --separator=";" -o vg_name,pv_count,lv_count,snap_count,vg_attr,vg_size,vg_free,vg_free_count
15/06/2020 22:42:25 INFO stdout ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.14";"2";"1";"0";"wz--n-";"6150.14g";"0g";"0
15/06/2020 22:42:25 INFO stdout ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.15";"2";"1";"0";"wz--n-";"6150.14g";"0g";"0
15/06/2020 22:42:25 INFO Running command: /sbin/vgs --noheadings --readonly --units=g --separator=";" -o vg_name,pv_count,lv_count,snap_count,vg_attr,vg_size,vg_free,vg_free_count
15/06/2020 22:42:25 INFO stdout ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.14";"2";"1";"0";"wz--n-";"6150.14g";"0g";"0
15/06/2020 22:42:25 INFO stdout ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.15";"2";"1";"0";"wz--n-";"6150.14g";"0g";"0
15/06/2020 22:42:25 INFO stdout ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.16";"2";"0";"0";"wz--n-";"6150.14g";"6150.14g";"1574437
15/06/2020 22:42:25 INFO vg_name = ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.16
15/06/2020 22:42:25 INFO done
15/06/2020 22:42:25 INFO Step 5 : Creating Logical Volumes :
15/06/2020 22:42:25 INFO -----------------------------------
15/06/2020 22:42:25 INFO main_lv_name = main
15/06/2020 22:42:25 INFO cache_lv_name = cache
15/06/2020 22:42:25 INFO done
15/06/2020 22:42:25 INFO Step 6 : Building dm-writecache :
15/06/2020 22:42:25 INFO ---------------------------------
15/06/2020 22:42:26 INFO cmd = lvchange -a n ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.16/cache ---> done
15/06/2020 22:42:26 INFO cmd = modprobe dm-writecache ---> done
15/06/2020 22:42:37 INFO LeaderElectionBase dropping old sessions
15/06/2020 22:42:37 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff22613b850>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:42:39 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff22613ba90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:42:43 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff22613bcd0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:42:47 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f89910>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:42:49 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f89b50>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:42:51 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff22613bf10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:42:53 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f89d90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:43:01 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880fa2050>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:43:04 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aab410>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:43:06 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aab650>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:43:07 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff226159190>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:43:10 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aab890>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:43:17 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880fa2250>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:43:18 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aabad0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:43:34 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aabd10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:43:39 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2261593d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:43:41 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff22613bc10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:43:43 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff22613bb90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:43:47 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff226159090>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:43:49 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880fa2490>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:43:51 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f89cd0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:43:53 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f89e90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:43:55 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff226159710>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:43:57 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880fa2490>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:44:05 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880fa2790>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:44:06 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aabf90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:44:08 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aab7d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:44:10 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aabc50>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:44:11 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff226159450>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:44:14 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aab8d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:44:21 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880fa2510>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:44:22 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047ca150>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:44:38 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047ca050>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:44:43 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2261599d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:44:43 ERROR HTTPConnectionPool(host='127.0.0.1', port=8500): Max retries exceeded with url: /v1/kv/PetaSAN/Config/Files?recurse=1 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff226159bd0>: Failed to establish a new connection: [Errno 111] Connection refused',))
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/file_sync_manager.py", line 75, in sync
index, data = base.watch(self.root_path, current_index)
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/base.py", line 49, in watch
return cons.kv.get(key, index=current_index, recurse=True)
File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 554, in get
params=params)
File "/usr/lib/python2.7/dist-packages/retrying.py", line 49, in wrapped_f
return Retrying(*dargs, **dkw).call(f, *args, **kw)
File "/usr/lib/python2.7/dist-packages/retrying.py", line 212, in call
raise attempt.get()
File "/usr/lib/python2.7/dist-packages/retrying.py", line 247, in get
six.reraise(self.value[0], self.value[1], self.value[2])
File "/usr/lib/python2.7/dist-packages/retrying.py", line 200, in call
attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/ps_consul.py", line 66, in get
res = self.response(self.session.get(uri, verify=self.verify))
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 533, in get
return self.request('GET', url, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 520, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 630, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 508, in send
raise ConnectionError(e, request=request)
ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=8500): Max retries exceeded with url: /v1/kv/PetaSAN/Config/Files?recurse=1 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff226159bd0>: Failed to establish a new connection: [Errno 111] Connection refused',))
15/06/2020 22:44:46 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260b3750>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:44:48 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260b3950>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:44:52 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260b3b90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:44:53 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880fa2a90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:44:53 ERROR LeaderElectionBase exception dropping old sessions
15/06/2020 22:45:00 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260b3dd0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:45:03 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f82a90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:45:05 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f82c90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:45:09 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880fa2190>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:45:10 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aab190>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:45:10 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader
15/06/2020 22:45:10 ERROR HTTPConnectionPool(host='127.0.0.1', port=8500): Max retries exceeded with url: /v1/kv/PetaSAN/Services/ClusterLeader (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047ca550>: Failed to establish a new connection: [Errno 111] Connection refused',))
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/api.py", line 446, in get_key
index, data = cons.kv.get(key)
File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 554, in get
params=params)
File "/usr/lib/python2.7/dist-packages/retrying.py", line 49, in wrapped_f
return Retrying(*dargs, **dkw).call(f, *args, **kw)
File "/usr/lib/python2.7/dist-packages/retrying.py", line 212, in call
raise attempt.get()
File "/usr/lib/python2.7/dist-packages/retrying.py", line 247, in get
six.reraise(self.value[0], self.value[1], self.value[2])
File "/usr/lib/python2.7/dist-packages/retrying.py", line 200, in call
attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/ps_consul.py", line 66, in get
res = self.response(self.session.get(uri, verify=self.verify))
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 533, in get
return self.request('GET', url, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 520, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 630, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 508, in send
raise ConnectionError(e, request=request)
ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=8500): Max retries exceeded with url: /v1/kv/PetaSAN/Services/ClusterLeader (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047ca550>: Failed to establish a new connection: [Errno 111] Connection refused',))
15/06/2020 22:45:16 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff226159150>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:45:17 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f898d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:45:33 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f82f90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:45:48 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260b34d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:45:50 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260b3750>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:45:52 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260b3b90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:45:56 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260b3fd0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:46:04 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260b3c10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:46:05 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f3b1d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:46:07 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f82a10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:46:09 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f82f10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:46:10 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047af850>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:46:12 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047afa90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:46:13 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f82dd0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:46:16 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047afcd0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:46:20 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260bb110>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:46:21 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f3b390>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:46:24 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047aff10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:46:37 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f3b250>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:46:40 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c04734190>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:46:52 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260bb450>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:46:52 ERROR HTTPConnectionPool(host='127.0.0.1', port=8500): Max retries exceeded with url: /v1/kv/PetaSAN/Config/Files?recurse=1 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260bb650>: Failed to establish a new connection: [Errno 111] Connection refused',))
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/file_sync_manager.py", line 75, in sync
index, data = base.watch(self.root_path, current_index)
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/base.py", line 49, in watch
return cons.kv.get(key, index=current_index, recurse=True)
File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 554, in get
params=params)
File "/usr/lib/python2.7/dist-packages/retrying.py", line 49, in wrapped_f
return Retrying(*dargs, **dkw).call(f, *args, **kw)
File "/usr/lib/python2.7/dist-packages/retrying.py", line 212, in call
raise attempt.get()
File "/usr/lib/python2.7/dist-packages/retrying.py", line 247, in get
six.reraise(self.value[0], self.value[1], self.value[2])
File "/usr/lib/python2.7/dist-packages/retrying.py", line 200, in call
attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/ps_consul.py", line 66, in get
res = self.response(self.session.get(uri, verify=self.verify))
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 533, in get
return self.request('GET', url, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 520, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 630, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 508, in send
raise ConnectionError(e, request=request)
ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=8500): Max retries exceeded with url: /v1/kv/PetaSAN/Config/Files?recurse=1 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260bb650>: Failed to establish a new connection: [Errno 111] Connection refused',))
15/06/2020 22:46:54 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260bba90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:46:56 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260bbc90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:47:00 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260bbed0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:47:08 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260c3150>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:47:10 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f3b790>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:47:10 ERROR LeaderElectionBase exception dropping old sessions
15/06/2020 22:47:12 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047343d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:47:14 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047afc10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:47:16 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047afb90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:47:20 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f44790>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:47:20 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c04734090>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:47:22 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f44990>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:47:24 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260c3390>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:47:26 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f44bd0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:47:28 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047346d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:47:34 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f44e10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
Hi Support Team,
I got error when add osd with dm-writecache. My cluster running version 2.5.3
The logs:
15/06/2020 22:41:56 INFO script
15/06/2020 22:41:56 INFO /opt/petasan/scripts/admin/node_manage_disks.py add-osd
15/06/2020 22:41:56 INFO params
15/06/2020 22:41:56 INFO -disk_name sdf -cache sdc -cache_type writecache
15/06/2020 22:41:56 INFO Start add osd job for disk sdf.
15/06/2020 22:42:02 INFO Start cleaning disk : sdf
15/06/2020 22:42:02 INFO Starting : vgchange -a n ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.16
15/06/2020 22:42:06 INFO Executing : wipefs --all /dev/sdf1
15/06/2020 22:42:06 INFO Executing : dd if=/dev/zero of=/dev/sdf1 bs=1M count=20 oflag=direct,dsync >/dev/null 2>&1
15/06/2020 22:42:06 INFO Executing : wipefs --all /dev/sdf
15/06/2020 22:42:06 INFO Executing : dd if=/dev/zero of=/dev/sdf bs=1M count=20 oflag=direct,dsync >/dev/null 2>&1
15/06/2020 22:42:06 INFO Executing : parted -s /dev/sdf mklabel gpt
15/06/2020 22:42:07 INFO Executing : partprobe /dev/sdf
15/06/2020 22:42:10 INFO User didn't select a journal for disk sdf, so the journal will be on the same disk.
15/06/2020 22:42:15 INFO User selected cache sdc disk for disk sdf.
15/06/2020 22:42:15 INFO Getting the next OSD ID :
15/06/2020 22:42:15 INFO -------------------------
15/06/2020 22:42:16 INFO Start remove osd.16 from crush map
15/06/2020 22:42:17 ERROR Error executing ceph osd out osd.16
15/06/2020 22:42:18 ERROR Error executing ceph osd crush remove osd.16
15/06/2020 22:42:18 ERROR Error executing ceph auth del osd.16
15/06/2020 22:42:19 WARNING The osd still up you need to stop osd service of osd.16
15/06/2020 22:42:19 WARNING osd.16 is removed from crush map
15/06/2020 22:42:19 INFO next_osd_id = 16
15/06/2020 22:42:19 INFO vg_name = ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.16
15/06/2020 22:42:19 INFO ==================================================
15/06/2020 22:42:19 INFO ===== Building DM-Writecache =====
15/06/2020 22:42:19 INFO ==================================================
15/06/2020 22:42:19 INFO Step 1 : Preparing Slow Disk :
15/06/2020 22:42:19 INFO ------------------------------
15/06/2020 22:42:19 INFO Creating data partition num 1 size 0 on /dev/sdf
15/06/2020 22:42:21 INFO Calling partprobe on sdf device
15/06/2020 22:42:21 INFO Executing partprobe /dev/sdf
15/06/2020 22:42:21 INFO Calling udevadm on sdf device
15/06/2020 22:42:21 INFO Executing udevadm settle --timeout 30
15/06/2020 22:42:24 INFO origin_part_name = /dev/sdf1
15/06/2020 22:42:24 INFO done
15/06/2020 22:42:24 INFO Step 2 : Preparing Fast Disk :
15/06/2020 22:42:24 INFO ------------------------------
15/06/2020 22:42:24 INFO Start cleaning disk : sdc1
15/06/2020 22:42:24 INFO Starting : vgchange -a n ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.16
15/06/2020 22:42:24 INFO Executing : umount /dev/sdc1
15/06/2020 22:42:24 ERROR Error executing : umount /dev/sdc1
15/06/2020 22:42:24 INFO Executing : wipefs --all /dev/sdc1
15/06/2020 22:42:24 INFO Executing : dd if=/dev/zero of=/dev/sdc1 bs=1M count=20 oflag=direct,dsync >/dev/null 2>&1
15/06/2020 22:42:24 INFO Executing : dd bs=1M seek=2324783104 if=/dev/zero of=/dev/sdc1 count=20 oflag=seek_bytes,direct,dsync >/dev/null 2>&1
15/06/2020 22:42:24 INFO cache_part_path = /dev/sdc1
15/06/2020 22:42:24 INFO done
15/06/2020 22:42:24 INFO Step 3 : Creating Physical Volumes :
15/06/2020 22:42:24 INFO ------------------------------------
15/06/2020 22:42:25 INFO done
15/06/2020 22:42:25 INFO Step 4 : Creating Volume Group :
15/06/2020 22:42:25 INFO --------------------------------
15/06/2020 22:42:25 INFO Running command: /sbin/vgs --noheadings --readonly --units=g --separator=";" -o vg_name,pv_count,lv_count,snap_count,vg_attr,vg_size,vg_free,vg_free_count
15/06/2020 22:42:25 INFO stdout ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.14";"2";"1";"0";"wz--n-";"6150.14g";"0g";"0
15/06/2020 22:42:25 INFO stdout ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.15";"2";"1";"0";"wz--n-";"6150.14g";"0g";"0
15/06/2020 22:42:25 INFO Running command: /sbin/vgs --noheadings --readonly --units=g --separator=";" -o vg_name,pv_count,lv_count,snap_count,vg_attr,vg_size,vg_free,vg_free_count
15/06/2020 22:42:25 INFO stdout ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.14";"2";"1";"0";"wz--n-";"6150.14g";"0g";"0
15/06/2020 22:42:25 INFO stdout ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.15";"2";"1";"0";"wz--n-";"6150.14g";"0g";"0
15/06/2020 22:42:25 INFO stdout ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.16";"2";"0";"0";"wz--n-";"6150.14g";"6150.14g";"1574437
15/06/2020 22:42:25 INFO vg_name = ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.16
15/06/2020 22:42:25 INFO done
15/06/2020 22:42:25 INFO Step 5 : Creating Logical Volumes :
15/06/2020 22:42:25 INFO -----------------------------------
15/06/2020 22:42:25 INFO main_lv_name = main
15/06/2020 22:42:25 INFO cache_lv_name = cache
15/06/2020 22:42:25 INFO done
15/06/2020 22:42:25 INFO Step 6 : Building dm-writecache :
15/06/2020 22:42:25 INFO ---------------------------------
15/06/2020 22:42:26 INFO cmd = lvchange -a n ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.16/cache ---> done
15/06/2020 22:42:26 INFO cmd = modprobe dm-writecache ---> done
15/06/2020 22:42:37 INFO LeaderElectionBase dropping old sessions
15/06/2020 22:42:37 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff22613b850>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:42:39 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff22613ba90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:42:43 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff22613bcd0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:42:47 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f89910>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:42:49 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f89b50>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:42:51 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff22613bf10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:42:53 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f89d90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:43:01 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880fa2050>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:43:04 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aab410>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:43:06 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aab650>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:43:07 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff226159190>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:43:10 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aab890>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:43:17 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880fa2250>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:43:18 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aabad0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:43:34 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aabd10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:43:39 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2261593d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:43:41 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff22613bc10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:43:43 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff22613bb90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:43:47 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff226159090>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:43:49 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880fa2490>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:43:51 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f89cd0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:43:53 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f89e90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:43:55 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff226159710>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:43:57 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880fa2490>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:44:05 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880fa2790>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:44:06 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aabf90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:44:08 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aab7d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:44:10 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aabc50>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:44:11 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff226159450>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:44:14 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aab8d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:44:21 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880fa2510>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:44:22 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047ca150>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:44:38 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047ca050>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:44:43 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2261599d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:44:43 ERROR HTTPConnectionPool(host='127.0.0.1', port=8500): Max retries exceeded with url: /v1/kv/PetaSAN/Config/Files?recurse=1 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff226159bd0>: Failed to establish a new connection: [Errno 111] Connection refused',))
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/file_sync_manager.py", line 75, in sync
index, data = base.watch(self.root_path, current_index)
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/base.py", line 49, in watch
return cons.kv.get(key, index=current_index, recurse=True)
File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 554, in get
params=params)
File "/usr/lib/python2.7/dist-packages/retrying.py", line 49, in wrapped_f
return Retrying(*dargs, **dkw).call(f, *args, **kw)
File "/usr/lib/python2.7/dist-packages/retrying.py", line 212, in call
raise attempt.get()
File "/usr/lib/python2.7/dist-packages/retrying.py", line 247, in get
six.reraise(self.value[0], self.value[1], self.value[2])
File "/usr/lib/python2.7/dist-packages/retrying.py", line 200, in call
attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/ps_consul.py", line 66, in get
res = self.response(self.session.get(uri, verify=self.verify))
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 533, in get
return self.request('GET', url, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 520, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 630, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 508, in send
raise ConnectionError(e, request=request)
ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=8500): Max retries exceeded with url: /v1/kv/PetaSAN/Config/Files?recurse=1 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff226159bd0>: Failed to establish a new connection: [Errno 111] Connection refused',))
15/06/2020 22:44:46 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260b3750>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:44:48 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260b3950>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:44:52 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260b3b90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:44:53 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880fa2a90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:44:53 ERROR LeaderElectionBase exception dropping old sessions
15/06/2020 22:45:00 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260b3dd0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:45:03 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f82a90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:45:05 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f82c90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:45:09 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880fa2190>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:45:10 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aab190>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:45:10 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader
15/06/2020 22:45:10 ERROR HTTPConnectionPool(host='127.0.0.1', port=8500): Max retries exceeded with url: /v1/kv/PetaSAN/Services/ClusterLeader (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047ca550>: Failed to establish a new connection: [Errno 111] Connection refused',))
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/api.py", line 446, in get_key
index, data = cons.kv.get(key)
File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 554, in get
params=params)
File "/usr/lib/python2.7/dist-packages/retrying.py", line 49, in wrapped_f
return Retrying(*dargs, **dkw).call(f, *args, **kw)
File "/usr/lib/python2.7/dist-packages/retrying.py", line 212, in call
raise attempt.get()
File "/usr/lib/python2.7/dist-packages/retrying.py", line 247, in get
six.reraise(self.value[0], self.value[1], self.value[2])
File "/usr/lib/python2.7/dist-packages/retrying.py", line 200, in call
attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/ps_consul.py", line 66, in get
res = self.response(self.session.get(uri, verify=self.verify))
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 533, in get
return self.request('GET', url, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 520, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 630, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 508, in send
raise ConnectionError(e, request=request)
ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=8500): Max retries exceeded with url: /v1/kv/PetaSAN/Services/ClusterLeader (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047ca550>: Failed to establish a new connection: [Errno 111] Connection refused',))
15/06/2020 22:45:16 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff226159150>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:45:17 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f898d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:45:33 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f82f90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:45:48 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260b34d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:45:50 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260b3750>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:45:52 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260b3b90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:45:56 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260b3fd0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:46:04 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260b3c10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:46:05 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f3b1d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:46:07 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f82a10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:46:09 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f82f10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:46:10 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047af850>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:46:12 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047afa90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:46:13 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f82dd0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:46:16 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047afcd0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:46:20 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260bb110>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:46:21 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f3b390>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:46:24 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047aff10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:46:37 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f3b250>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:46:40 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c04734190>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:46:52 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260bb450>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:46:52 ERROR HTTPConnectionPool(host='127.0.0.1', port=8500): Max retries exceeded with url: /v1/kv/PetaSAN/Config/Files?recurse=1 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260bb650>: Failed to establish a new connection: [Errno 111] Connection refused',))
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/file_sync_manager.py", line 75, in sync
index, data = base.watch(self.root_path, current_index)
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/base.py", line 49, in watch
return cons.kv.get(key, index=current_index, recurse=True)
File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 554, in get
params=params)
File "/usr/lib/python2.7/dist-packages/retrying.py", line 49, in wrapped_f
return Retrying(*dargs, **dkw).call(f, *args, **kw)
File "/usr/lib/python2.7/dist-packages/retrying.py", line 212, in call
raise attempt.get()
File "/usr/lib/python2.7/dist-packages/retrying.py", line 247, in get
six.reraise(self.value[0], self.value[1], self.value[2])
File "/usr/lib/python2.7/dist-packages/retrying.py", line 200, in call
attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/ps_consul.py", line 66, in get
res = self.response(self.session.get(uri, verify=self.verify))
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 533, in get
return self.request('GET', url, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 520, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 630, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 508, in send
raise ConnectionError(e, request=request)
ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=8500): Max retries exceeded with url: /v1/kv/PetaSAN/Config/Files?recurse=1 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260bb650>: Failed to establish a new connection: [Errno 111] Connection refused',))
15/06/2020 22:46:54 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260bba90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:46:56 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260bbc90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:47:00 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260bbed0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:47:08 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260c3150>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:47:10 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f3b790>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:47:10 ERROR LeaderElectionBase exception dropping old sessions
15/06/2020 22:47:12 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047343d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:47:14 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047afc10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:47:16 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047afb90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:47:20 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f44790>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:47:20 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c04734090>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:47:22 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f44990>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:47:24 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260c3390>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:47:26 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f44bd0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:47:28 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047346d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:47:34 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f44e10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
zamasaki158
13 Posts
June 22, 2020, 5:34 amQuote from zamasaki158 on June 22, 2020, 5:34 amI added more RAM and the problem was solved.
I added more RAM and the problem was solved.
admin
2,930 Posts
June 22, 2020, 9:42 pmQuote from admin on June 22, 2020, 9:42 pmGreat. remember that write cache requites 2% of SSD space in memory. A 100 G SSD partition requires 2G of RAM.
Great. remember that write cache requites 2% of SSD space in memory. A 100 G SSD partition requires 2G of RAM.
Error when add osd with dm-writecache
zamasaki158
13 Posts
June 17, 2020, 10:33 am
Quote from zamasaki158 on June 17, 2020, 10:33 amHi Support Team,I got error when add osd with dm-writecache. My cluster running version 2.5.3The logs:15/06/2020 22:41:56 INFO script15/06/2020 22:41:56 INFO /opt/petasan/scripts/admin/node_manage_disks.py add-osd15/06/2020 22:41:56 INFO params15/06/2020 22:41:56 INFO -disk_name sdf -cache sdc -cache_type writecache15/06/2020 22:41:56 INFO Start add osd job for disk sdf.15/06/2020 22:42:02 INFO Start cleaning disk : sdf15/06/2020 22:42:02 INFO Starting : vgchange -a n ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.1615/06/2020 22:42:06 INFO Executing : wipefs --all /dev/sdf115/06/2020 22:42:06 INFO Executing : dd if=/dev/zero of=/dev/sdf1 bs=1M count=20 oflag=direct,dsync >/dev/null 2>&115/06/2020 22:42:06 INFO Executing : wipefs --all /dev/sdf15/06/2020 22:42:06 INFO Executing : dd if=/dev/zero of=/dev/sdf bs=1M count=20 oflag=direct,dsync >/dev/null 2>&115/06/2020 22:42:06 INFO Executing : parted -s /dev/sdf mklabel gpt15/06/2020 22:42:07 INFO Executing : partprobe /dev/sdf15/06/2020 22:42:10 INFO User didn't select a journal for disk sdf, so the journal will be on the same disk.15/06/2020 22:42:15 INFO User selected cache sdc disk for disk sdf.15/06/2020 22:42:15 INFO Getting the next OSD ID :15/06/2020 22:42:15 INFO -------------------------15/06/2020 22:42:16 INFO Start remove osd.16 from crush map15/06/2020 22:42:17 ERROR Error executing ceph osd out osd.1615/06/2020 22:42:18 ERROR Error executing ceph osd crush remove osd.1615/06/2020 22:42:18 ERROR Error executing ceph auth del osd.1615/06/2020 22:42:19 WARNING The osd still up you need to stop osd service of osd.1615/06/2020 22:42:19 WARNING osd.16 is removed from crush map15/06/2020 22:42:19 INFO next_osd_id = 1615/06/2020 22:42:19 INFO vg_name = ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.1615/06/2020 22:42:19 INFO ==================================================15/06/2020 22:42:19 INFO ===== Building DM-Writecache =====15/06/2020 22:42:19 INFO ==================================================15/06/2020 22:42:19 INFO Step 1 : Preparing Slow Disk :15/06/2020 22:42:19 INFO ------------------------------15/06/2020 22:42:19 INFO Creating data partition num 1 size 0 on /dev/sdf15/06/2020 22:42:21 INFO Calling partprobe on sdf device15/06/2020 22:42:21 INFO Executing partprobe /dev/sdf15/06/2020 22:42:21 INFO Calling udevadm on sdf device15/06/2020 22:42:21 INFO Executing udevadm settle --timeout 3015/06/2020 22:42:24 INFO origin_part_name = /dev/sdf115/06/2020 22:42:24 INFO done15/06/2020 22:42:24 INFO Step 2 : Preparing Fast Disk :15/06/2020 22:42:24 INFO ------------------------------15/06/2020 22:42:24 INFO Start cleaning disk : sdc115/06/2020 22:42:24 INFO Starting : vgchange -a n ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.1615/06/2020 22:42:24 INFO Executing : umount /dev/sdc115/06/2020 22:42:24 ERROR Error executing : umount /dev/sdc115/06/2020 22:42:24 INFO Executing : wipefs --all /dev/sdc115/06/2020 22:42:24 INFO Executing : dd if=/dev/zero of=/dev/sdc1 bs=1M count=20 oflag=direct,dsync >/dev/null 2>&115/06/2020 22:42:24 INFO Executing : dd bs=1M seek=2324783104 if=/dev/zero of=/dev/sdc1 count=20 oflag=seek_bytes,direct,dsync >/dev/null 2>&115/06/2020 22:42:24 INFO cache_part_path = /dev/sdc115/06/2020 22:42:24 INFO done15/06/2020 22:42:24 INFO Step 3 : Creating Physical Volumes :15/06/2020 22:42:24 INFO ------------------------------------15/06/2020 22:42:25 INFO done15/06/2020 22:42:25 INFO Step 4 : Creating Volume Group :15/06/2020 22:42:25 INFO --------------------------------15/06/2020 22:42:25 INFO Running command: /sbin/vgs --noheadings --readonly --units=g --separator=";" -o vg_name,pv_count,lv_count,snap_count,vg_attr,vg_size,vg_free,vg_free_count15/06/2020 22:42:25 INFO stdout ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.14";"2";"1";"0";"wz--n-";"6150.14g";"0g";"015/06/2020 22:42:25 INFO stdout ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.15";"2";"1";"0";"wz--n-";"6150.14g";"0g";"015/06/2020 22:42:25 INFO Running command: /sbin/vgs --noheadings --readonly --units=g --separator=";" -o vg_name,pv_count,lv_count,snap_count,vg_attr,vg_size,vg_free,vg_free_count15/06/2020 22:42:25 INFO stdout ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.14";"2";"1";"0";"wz--n-";"6150.14g";"0g";"015/06/2020 22:42:25 INFO stdout ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.15";"2";"1";"0";"wz--n-";"6150.14g";"0g";"015/06/2020 22:42:25 INFO stdout ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.16";"2";"0";"0";"wz--n-";"6150.14g";"6150.14g";"157443715/06/2020 22:42:25 INFO vg_name = ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.1615/06/2020 22:42:25 INFO done15/06/2020 22:42:25 INFO Step 5 : Creating Logical Volumes :15/06/2020 22:42:25 INFO -----------------------------------15/06/2020 22:42:25 INFO main_lv_name = main15/06/2020 22:42:25 INFO cache_lv_name = cache15/06/2020 22:42:25 INFO done15/06/2020 22:42:25 INFO Step 6 : Building dm-writecache :15/06/2020 22:42:25 INFO ---------------------------------15/06/2020 22:42:26 INFO cmd = lvchange -a n ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.16/cache ---> done15/06/2020 22:42:26 INFO cmd = modprobe dm-writecache ---> done15/06/2020 22:42:37 INFO LeaderElectionBase dropping old sessions15/06/2020 22:42:37 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff22613b850>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=115/06/2020 22:42:39 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff22613ba90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=115/06/2020 22:42:43 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff22613bcd0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=115/06/2020 22:42:47 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f89910>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list15/06/2020 22:42:49 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f89b50>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list15/06/2020 22:42:51 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff22613bf10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=115/06/2020 22:42:53 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f89d90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list15/06/2020 22:43:01 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880fa2050>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list15/06/2020 22:43:04 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aab410>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader15/06/2020 22:43:06 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aab650>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader15/06/2020 22:43:07 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff226159190>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=115/06/2020 22:43:10 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aab890>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader15/06/2020 22:43:17 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880fa2250>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list15/06/2020 22:43:18 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aabad0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader15/06/2020 22:43:34 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aabd10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader15/06/2020 22:43:39 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2261593d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=115/06/2020 22:43:41 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff22613bc10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=115/06/2020 22:43:43 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff22613bb90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=115/06/2020 22:43:47 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff226159090>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=115/06/2020 22:43:49 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880fa2490>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list15/06/2020 22:43:51 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f89cd0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list15/06/2020 22:43:53 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f89e90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list15/06/2020 22:43:55 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff226159710>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=115/06/2020 22:43:57 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880fa2490>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list15/06/2020 22:44:05 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880fa2790>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list15/06/2020 22:44:06 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aabf90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader15/06/2020 22:44:08 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aab7d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader15/06/2020 22:44:10 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aabc50>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader15/06/2020 22:44:11 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff226159450>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=115/06/2020 22:44:14 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aab8d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader15/06/2020 22:44:21 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880fa2510>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list15/06/2020 22:44:22 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047ca150>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader15/06/2020 22:44:38 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047ca050>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader15/06/2020 22:44:43 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2261599d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=115/06/2020 22:44:43 ERROR HTTPConnectionPool(host='127.0.0.1', port=8500): Max retries exceeded with url: /v1/kv/PetaSAN/Config/Files?recurse=1 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff226159bd0>: Failed to establish a new connection: [Errno 111] Connection refused',))Traceback (most recent call last):File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/file_sync_manager.py", line 75, in syncindex, data = base.watch(self.root_path, current_index)File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/base.py", line 49, in watchreturn cons.kv.get(key, index=current_index, recurse=True)File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 554, in getparams=params)File "/usr/lib/python2.7/dist-packages/retrying.py", line 49, in wrapped_freturn Retrying(*dargs, **dkw).call(f, *args, **kw)File "/usr/lib/python2.7/dist-packages/retrying.py", line 212, in callraise attempt.get()File "/usr/lib/python2.7/dist-packages/retrying.py", line 247, in getsix.reraise(self.value[0], self.value[1], self.value[2])File "/usr/lib/python2.7/dist-packages/retrying.py", line 200, in callattempt = Attempt(fn(*args, **kwargs), attempt_number, False)File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/ps_consul.py", line 66, in getres = self.response(self.session.get(uri, verify=self.verify))File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 533, in getreturn self.request('GET', url, **kwargs)File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 520, in requestresp = self.send(prep, **send_kwargs)File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 630, in sendr = adapter.send(request, **kwargs)File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 508, in sendraise ConnectionError(e, request=request)ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=8500): Max retries exceeded with url: /v1/kv/PetaSAN/Config/Files?recurse=1 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff226159bd0>: Failed to establish a new connection: [Errno 111] Connection refused',))15/06/2020 22:44:46 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260b3750>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=115/06/2020 22:44:48 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260b3950>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=115/06/2020 22:44:52 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260b3b90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=115/06/2020 22:44:53 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880fa2a90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list15/06/2020 22:44:53 ERROR LeaderElectionBase exception dropping old sessions15/06/2020 22:45:00 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260b3dd0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=115/06/2020 22:45:03 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f82a90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list15/06/2020 22:45:05 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f82c90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list15/06/2020 22:45:09 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880fa2190>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list15/06/2020 22:45:10 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aab190>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader15/06/2020 22:45:10 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader15/06/2020 22:45:10 ERROR HTTPConnectionPool(host='127.0.0.1', port=8500): Max retries exceeded with url: /v1/kv/PetaSAN/Services/ClusterLeader (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047ca550>: Failed to establish a new connection: [Errno 111] Connection refused',))Traceback (most recent call last):File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/api.py", line 446, in get_keyindex, data = cons.kv.get(key)File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 554, in getparams=params)File "/usr/lib/python2.7/dist-packages/retrying.py", line 49, in wrapped_freturn Retrying(*dargs, **dkw).call(f, *args, **kw)File "/usr/lib/python2.7/dist-packages/retrying.py", line 212, in callraise attempt.get()File "/usr/lib/python2.7/dist-packages/retrying.py", line 247, in getsix.reraise(self.value[0], self.value[1], self.value[2])File "/usr/lib/python2.7/dist-packages/retrying.py", line 200, in callattempt = Attempt(fn(*args, **kwargs), attempt_number, False)File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/ps_consul.py", line 66, in getres = self.response(self.session.get(uri, verify=self.verify))File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 533, in getreturn self.request('GET', url, **kwargs)File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 520, in requestresp = self.send(prep, **send_kwargs)File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 630, in sendr = adapter.send(request, **kwargs)File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 508, in sendraise ConnectionError(e, request=request)ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=8500): Max retries exceeded with url: /v1/kv/PetaSAN/Services/ClusterLeader (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047ca550>: Failed to establish a new connection: [Errno 111] Connection refused',))15/06/2020 22:45:16 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff226159150>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=115/06/2020 22:45:17 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f898d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list15/06/2020 22:45:33 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f82f90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list15/06/2020 22:45:48 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260b34d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=115/06/2020 22:45:50 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260b3750>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=115/06/2020 22:45:52 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260b3b90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=115/06/2020 22:45:56 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260b3fd0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=115/06/2020 22:46:04 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260b3c10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=115/06/2020 22:46:05 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f3b1d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list15/06/2020 22:46:07 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f82a10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list15/06/2020 22:46:09 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f82f10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list15/06/2020 22:46:10 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047af850>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader15/06/2020 22:46:12 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047afa90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader15/06/2020 22:46:13 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f82dd0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list15/06/2020 22:46:16 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047afcd0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader15/06/2020 22:46:20 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260bb110>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=115/06/2020 22:46:21 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f3b390>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list15/06/2020 22:46:24 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047aff10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader15/06/2020 22:46:37 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f3b250>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list15/06/2020 22:46:40 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c04734190>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader15/06/2020 22:46:52 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260bb450>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=115/06/2020 22:46:52 ERROR HTTPConnectionPool(host='127.0.0.1', port=8500): Max retries exceeded with url: /v1/kv/PetaSAN/Config/Files?recurse=1 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260bb650>: Failed to establish a new connection: [Errno 111] Connection refused',))Traceback (most recent call last):File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/file_sync_manager.py", line 75, in syncindex, data = base.watch(self.root_path, current_index)File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/base.py", line 49, in watchreturn cons.kv.get(key, index=current_index, recurse=True)File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 554, in getparams=params)File "/usr/lib/python2.7/dist-packages/retrying.py", line 49, in wrapped_freturn Retrying(*dargs, **dkw).call(f, *args, **kw)File "/usr/lib/python2.7/dist-packages/retrying.py", line 212, in callraise attempt.get()File "/usr/lib/python2.7/dist-packages/retrying.py", line 247, in getsix.reraise(self.value[0], self.value[1], self.value[2])File "/usr/lib/python2.7/dist-packages/retrying.py", line 200, in callattempt = Attempt(fn(*args, **kwargs), attempt_number, False)File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/ps_consul.py", line 66, in getres = self.response(self.session.get(uri, verify=self.verify))File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 533, in getreturn self.request('GET', url, **kwargs)File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 520, in requestresp = self.send(prep, **send_kwargs)File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 630, in sendr = adapter.send(request, **kwargs)File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 508, in sendraise ConnectionError(e, request=request)ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=8500): Max retries exceeded with url: /v1/kv/PetaSAN/Config/Files?recurse=1 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260bb650>: Failed to establish a new connection: [Errno 111] Connection refused',))15/06/2020 22:46:54 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260bba90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=115/06/2020 22:46:56 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260bbc90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=115/06/2020 22:47:00 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260bbed0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=115/06/2020 22:47:08 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260c3150>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=115/06/2020 22:47:10 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f3b790>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list15/06/2020 22:47:10 ERROR LeaderElectionBase exception dropping old sessions15/06/2020 22:47:12 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047343d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader15/06/2020 22:47:14 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047afc10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader15/06/2020 22:47:16 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047afb90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader15/06/2020 22:47:20 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f44790>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list15/06/2020 22:47:20 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c04734090>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader15/06/2020 22:47:22 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f44990>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list15/06/2020 22:47:24 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260c3390>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=115/06/2020 22:47:26 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f44bd0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list15/06/2020 22:47:28 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047346d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader15/06/2020 22:47:34 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f44e10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
Hi Support Team,
I got error when add osd with dm-writecache. My cluster running version 2.5.3
The logs:
15/06/2020 22:41:56 INFO script
15/06/2020 22:41:56 INFO /opt/petasan/scripts/admin/node_manage_disks.py add-osd
15/06/2020 22:41:56 INFO params
15/06/2020 22:41:56 INFO -disk_name sdf -cache sdc -cache_type writecache
15/06/2020 22:41:56 INFO Start add osd job for disk sdf.
15/06/2020 22:42:02 INFO Start cleaning disk : sdf
15/06/2020 22:42:02 INFO Starting : vgchange -a n ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.16
15/06/2020 22:42:06 INFO Executing : wipefs --all /dev/sdf1
15/06/2020 22:42:06 INFO Executing : dd if=/dev/zero of=/dev/sdf1 bs=1M count=20 oflag=direct,dsync >/dev/null 2>&1
15/06/2020 22:42:06 INFO Executing : wipefs --all /dev/sdf
15/06/2020 22:42:06 INFO Executing : dd if=/dev/zero of=/dev/sdf bs=1M count=20 oflag=direct,dsync >/dev/null 2>&1
15/06/2020 22:42:06 INFO Executing : parted -s /dev/sdf mklabel gpt
15/06/2020 22:42:07 INFO Executing : partprobe /dev/sdf
15/06/2020 22:42:10 INFO User didn't select a journal for disk sdf, so the journal will be on the same disk.
15/06/2020 22:42:15 INFO User selected cache sdc disk for disk sdf.
15/06/2020 22:42:15 INFO Getting the next OSD ID :
15/06/2020 22:42:15 INFO -------------------------
15/06/2020 22:42:16 INFO Start remove osd.16 from crush map
15/06/2020 22:42:17 ERROR Error executing ceph osd out osd.16
15/06/2020 22:42:18 ERROR Error executing ceph osd crush remove osd.16
15/06/2020 22:42:18 ERROR Error executing ceph auth del osd.16
15/06/2020 22:42:19 WARNING The osd still up you need to stop osd service of osd.16
15/06/2020 22:42:19 WARNING osd.16 is removed from crush map
15/06/2020 22:42:19 INFO next_osd_id = 16
15/06/2020 22:42:19 INFO vg_name = ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.16
15/06/2020 22:42:19 INFO ==================================================
15/06/2020 22:42:19 INFO ===== Building DM-Writecache =====
15/06/2020 22:42:19 INFO ==================================================
15/06/2020 22:42:19 INFO Step 1 : Preparing Slow Disk :
15/06/2020 22:42:19 INFO ------------------------------
15/06/2020 22:42:19 INFO Creating data partition num 1 size 0 on /dev/sdf
15/06/2020 22:42:21 INFO Calling partprobe on sdf device
15/06/2020 22:42:21 INFO Executing partprobe /dev/sdf
15/06/2020 22:42:21 INFO Calling udevadm on sdf device
15/06/2020 22:42:21 INFO Executing udevadm settle --timeout 30
15/06/2020 22:42:24 INFO origin_part_name = /dev/sdf1
15/06/2020 22:42:24 INFO done
15/06/2020 22:42:24 INFO Step 2 : Preparing Fast Disk :
15/06/2020 22:42:24 INFO ------------------------------
15/06/2020 22:42:24 INFO Start cleaning disk : sdc1
15/06/2020 22:42:24 INFO Starting : vgchange -a n ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.16
15/06/2020 22:42:24 INFO Executing : umount /dev/sdc1
15/06/2020 22:42:24 ERROR Error executing : umount /dev/sdc1
15/06/2020 22:42:24 INFO Executing : wipefs --all /dev/sdc1
15/06/2020 22:42:24 INFO Executing : dd if=/dev/zero of=/dev/sdc1 bs=1M count=20 oflag=direct,dsync >/dev/null 2>&1
15/06/2020 22:42:24 INFO Executing : dd bs=1M seek=2324783104 if=/dev/zero of=/dev/sdc1 count=20 oflag=seek_bytes,direct,dsync >/dev/null 2>&1
15/06/2020 22:42:24 INFO cache_part_path = /dev/sdc1
15/06/2020 22:42:24 INFO done
15/06/2020 22:42:24 INFO Step 3 : Creating Physical Volumes :
15/06/2020 22:42:24 INFO ------------------------------------
15/06/2020 22:42:25 INFO done
15/06/2020 22:42:25 INFO Step 4 : Creating Volume Group :
15/06/2020 22:42:25 INFO --------------------------------
15/06/2020 22:42:25 INFO Running command: /sbin/vgs --noheadings --readonly --units=g --separator=";" -o vg_name,pv_count,lv_count,snap_count,vg_attr,vg_size,vg_free,vg_free_count
15/06/2020 22:42:25 INFO stdout ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.14";"2";"1";"0";"wz--n-";"6150.14g";"0g";"0
15/06/2020 22:42:25 INFO stdout ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.15";"2";"1";"0";"wz--n-";"6150.14g";"0g";"0
15/06/2020 22:42:25 INFO Running command: /sbin/vgs --noheadings --readonly --units=g --separator=";" -o vg_name,pv_count,lv_count,snap_count,vg_attr,vg_size,vg_free,vg_free_count
15/06/2020 22:42:25 INFO stdout ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.14";"2";"1";"0";"wz--n-";"6150.14g";"0g";"0
15/06/2020 22:42:25 INFO stdout ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.15";"2";"1";"0";"wz--n-";"6150.14g";"0g";"0
15/06/2020 22:42:25 INFO stdout ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.16";"2";"0";"0";"wz--n-";"6150.14g";"6150.14g";"1574437
15/06/2020 22:42:25 INFO vg_name = ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.16
15/06/2020 22:42:25 INFO done
15/06/2020 22:42:25 INFO Step 5 : Creating Logical Volumes :
15/06/2020 22:42:25 INFO -----------------------------------
15/06/2020 22:42:25 INFO main_lv_name = main
15/06/2020 22:42:25 INFO cache_lv_name = cache
15/06/2020 22:42:25 INFO done
15/06/2020 22:42:25 INFO Step 6 : Building dm-writecache :
15/06/2020 22:42:25 INFO ---------------------------------
15/06/2020 22:42:26 INFO cmd = lvchange -a n ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.16/cache ---> done
15/06/2020 22:42:26 INFO cmd = modprobe dm-writecache ---> done
15/06/2020 22:42:37 INFO LeaderElectionBase dropping old sessions
15/06/2020 22:42:37 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff22613b850>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:42:39 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff22613ba90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:42:43 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff22613bcd0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:42:47 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f89910>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:42:49 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f89b50>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:42:51 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff22613bf10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:42:53 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f89d90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:43:01 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880fa2050>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:43:04 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aab410>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:43:06 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aab650>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:43:07 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff226159190>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:43:10 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aab890>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:43:17 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880fa2250>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:43:18 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aabad0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:43:34 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aabd10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:43:39 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2261593d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:43:41 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff22613bc10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:43:43 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff22613bb90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:43:47 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff226159090>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:43:49 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880fa2490>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:43:51 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f89cd0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:43:53 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f89e90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:43:55 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff226159710>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:43:57 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880fa2490>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:44:05 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880fa2790>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:44:06 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aabf90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:44:08 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aab7d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:44:10 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aabc50>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:44:11 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff226159450>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:44:14 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aab8d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:44:21 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880fa2510>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:44:22 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047ca150>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:44:38 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047ca050>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:44:43 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2261599d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:44:43 ERROR HTTPConnectionPool(host='127.0.0.1', port=8500): Max retries exceeded with url: /v1/kv/PetaSAN/Config/Files?recurse=1 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff226159bd0>: Failed to establish a new connection: [Errno 111] Connection refused',))
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/file_sync_manager.py", line 75, in sync
index, data = base.watch(self.root_path, current_index)
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/base.py", line 49, in watch
return cons.kv.get(key, index=current_index, recurse=True)
File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 554, in get
params=params)
File "/usr/lib/python2.7/dist-packages/retrying.py", line 49, in wrapped_f
return Retrying(*dargs, **dkw).call(f, *args, **kw)
File "/usr/lib/python2.7/dist-packages/retrying.py", line 212, in call
raise attempt.get()
File "/usr/lib/python2.7/dist-packages/retrying.py", line 247, in get
six.reraise(self.value[0], self.value[1], self.value[2])
File "/usr/lib/python2.7/dist-packages/retrying.py", line 200, in call
attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/ps_consul.py", line 66, in get
res = self.response(self.session.get(uri, verify=self.verify))
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 533, in get
return self.request('GET', url, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 520, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 630, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 508, in send
raise ConnectionError(e, request=request)
ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=8500): Max retries exceeded with url: /v1/kv/PetaSAN/Config/Files?recurse=1 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff226159bd0>: Failed to establish a new connection: [Errno 111] Connection refused',))
15/06/2020 22:44:46 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260b3750>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:44:48 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260b3950>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:44:52 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260b3b90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:44:53 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880fa2a90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:44:53 ERROR LeaderElectionBase exception dropping old sessions
15/06/2020 22:45:00 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260b3dd0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:45:03 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f82a90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:45:05 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f82c90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:45:09 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880fa2190>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:45:10 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aab190>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:45:10 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader
15/06/2020 22:45:10 ERROR HTTPConnectionPool(host='127.0.0.1', port=8500): Max retries exceeded with url: /v1/kv/PetaSAN/Services/ClusterLeader (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047ca550>: Failed to establish a new connection: [Errno 111] Connection refused',))
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/api.py", line 446, in get_key
index, data = cons.kv.get(key)
File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 554, in get
params=params)
File "/usr/lib/python2.7/dist-packages/retrying.py", line 49, in wrapped_f
return Retrying(*dargs, **dkw).call(f, *args, **kw)
File "/usr/lib/python2.7/dist-packages/retrying.py", line 212, in call
raise attempt.get()
File "/usr/lib/python2.7/dist-packages/retrying.py", line 247, in get
six.reraise(self.value[0], self.value[1], self.value[2])
File "/usr/lib/python2.7/dist-packages/retrying.py", line 200, in call
attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/ps_consul.py", line 66, in get
res = self.response(self.session.get(uri, verify=self.verify))
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 533, in get
return self.request('GET', url, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 520, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 630, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 508, in send
raise ConnectionError(e, request=request)
ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=8500): Max retries exceeded with url: /v1/kv/PetaSAN/Services/ClusterLeader (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047ca550>: Failed to establish a new connection: [Errno 111] Connection refused',))
15/06/2020 22:45:16 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff226159150>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:45:17 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f898d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:45:33 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f82f90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:45:48 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260b34d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:45:50 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260b3750>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:45:52 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260b3b90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:45:56 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260b3fd0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:46:04 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260b3c10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:46:05 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f3b1d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:46:07 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f82a10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:46:09 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f82f10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:46:10 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047af850>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:46:12 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047afa90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:46:13 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f82dd0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:46:16 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047afcd0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:46:20 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260bb110>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:46:21 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f3b390>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:46:24 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047aff10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:46:37 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f3b250>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:46:40 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c04734190>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:46:52 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260bb450>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:46:52 ERROR HTTPConnectionPool(host='127.0.0.1', port=8500): Max retries exceeded with url: /v1/kv/PetaSAN/Config/Files?recurse=1 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260bb650>: Failed to establish a new connection: [Errno 111] Connection refused',))
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/file_sync_manager.py", line 75, in sync
index, data = base.watch(self.root_path, current_index)
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/base.py", line 49, in watch
return cons.kv.get(key, index=current_index, recurse=True)
File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 554, in get
params=params)
File "/usr/lib/python2.7/dist-packages/retrying.py", line 49, in wrapped_f
return Retrying(*dargs, **dkw).call(f, *args, **kw)
File "/usr/lib/python2.7/dist-packages/retrying.py", line 212, in call
raise attempt.get()
File "/usr/lib/python2.7/dist-packages/retrying.py", line 247, in get
six.reraise(self.value[0], self.value[1], self.value[2])
File "/usr/lib/python2.7/dist-packages/retrying.py", line 200, in call
attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/ps_consul.py", line 66, in get
res = self.response(self.session.get(uri, verify=self.verify))
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 533, in get
return self.request('GET', url, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 520, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 630, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 508, in send
raise ConnectionError(e, request=request)
ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=8500): Max retries exceeded with url: /v1/kv/PetaSAN/Config/Files?recurse=1 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260bb650>: Failed to establish a new connection: [Errno 111] Connection refused',))
15/06/2020 22:46:54 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260bba90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:46:56 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260bbc90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:47:00 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260bbed0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:47:08 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260c3150>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:47:10 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f3b790>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:47:10 ERROR LeaderElectionBase exception dropping old sessions
15/06/2020 22:47:12 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047343d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:47:14 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047afc10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:47:16 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047afb90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:47:20 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f44790>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:47:20 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c04734090>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:47:22 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f44990>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:47:24 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260c3390>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
15/06/2020 22:47:26 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f44bd0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
15/06/2020 22:47:28 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047346d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
15/06/2020 22:47:34 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f44e10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
zamasaki158
13 Posts
June 22, 2020, 5:34 am
Quote from zamasaki158 on June 22, 2020, 5:34 amI added more RAM and the problem was solved.
I added more RAM and the problem was solved.
admin
2,930 Posts
June 22, 2020, 9:42 pm
Quote from admin on June 22, 2020, 9:42 pmGreat. remember that write cache requites 2% of SSD space in memory. A 100 G SSD partition requires 2G of RAM.
Great. remember that write cache requites 2% of SSD space in memory. A 100 G SSD partition requires 2G of RAM.