Forums

Home / Forums

You need to log in to create posts and topics. Login · Register

Error when add osd with dm-writecache


zamasaki158
13 Posts
June 17, 2020, 10:33 am
Hi Support Team,
I got error when add osd with dm-writecache. My cluster running version 2.5.3
The logs:
 15/06/2020 22:41:56 INFO script
 15/06/2020 22:41:56 INFO /opt/petasan/scripts/admin/node_manage_disks.py add-osd
 15/06/2020 22:41:56 INFO params
 15/06/2020 22:41:56 INFO -disk_name sdf -cache sdc -cache_type writecache
 15/06/2020 22:41:56 INFO Start add osd job for disk sdf.
 15/06/2020 22:42:02 INFO Start cleaning disk : sdf
 15/06/2020 22:42:02 INFO Starting : vgchange -a n ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.16
 15/06/2020 22:42:06 INFO Executing : wipefs --all /dev/sdf1
 15/06/2020 22:42:06 INFO Executing : dd if=/dev/zero of=/dev/sdf1 bs=1M count=20 oflag=direct,dsync >/dev/null 2>&1
 15/06/2020 22:42:06 INFO Executing : wipefs --all /dev/sdf
 15/06/2020 22:42:06 INFO Executing : dd if=/dev/zero of=/dev/sdf bs=1M count=20 oflag=direct,dsync >/dev/null 2>&1
 15/06/2020 22:42:06 INFO Executing : parted -s /dev/sdf mklabel gpt
 15/06/2020 22:42:07 INFO Executing : partprobe /dev/sdf
 15/06/2020 22:42:10 INFO User didn't select a journal for disk sdf, so the journal will be on the same disk.
 15/06/2020 22:42:15 INFO User selected cache sdc disk for disk sdf.
 15/06/2020 22:42:15 INFO Getting the next OSD ID :
 15/06/2020 22:42:15 INFO -------------------------
 15/06/2020 22:42:16 INFO Start remove osd.16 from crush map
 15/06/2020 22:42:17 ERROR Error executing ceph osd out osd.16
 15/06/2020 22:42:18 ERROR Error executing ceph osd crush remove osd.16
 15/06/2020 22:42:18 ERROR Error executing ceph auth del osd.16
 15/06/2020 22:42:19 WARNING The osd still up you need to stop osd service of osd.16
 15/06/2020 22:42:19 WARNING osd.16 is removed from crush map
 15/06/2020 22:42:19 INFO next_osd_id = 16
 15/06/2020 22:42:19 INFO vg_name = ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.16
 15/06/2020 22:42:19 INFO ==================================================
 15/06/2020 22:42:19 INFO ===== Building DM-Writecache =====
 15/06/2020 22:42:19 INFO ==================================================
 15/06/2020 22:42:19 INFO Step 1 : Preparing Slow Disk :
 15/06/2020 22:42:19 INFO ------------------------------
 15/06/2020 22:42:19 INFO Creating data partition num 1 size 0 on /dev/sdf
 15/06/2020 22:42:21 INFO Calling partprobe on sdf device
 15/06/2020 22:42:21 INFO Executing partprobe /dev/sdf
 15/06/2020 22:42:21 INFO Calling udevadm on sdf device
 15/06/2020 22:42:21 INFO Executing udevadm settle --timeout 30
 15/06/2020 22:42:24 INFO origin_part_name = /dev/sdf1
 15/06/2020 22:42:24 INFO done
 15/06/2020 22:42:24 INFO Step 2 : Preparing Fast Disk :
 15/06/2020 22:42:24 INFO ------------------------------
 15/06/2020 22:42:24 INFO Start cleaning disk : sdc1
 15/06/2020 22:42:24 INFO Starting : vgchange -a n ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.16
 15/06/2020 22:42:24 INFO Executing : umount /dev/sdc1
 15/06/2020 22:42:24 ERROR Error executing : umount /dev/sdc1
 15/06/2020 22:42:24 INFO Executing : wipefs --all /dev/sdc1
 15/06/2020 22:42:24 INFO Executing : dd if=/dev/zero of=/dev/sdc1 bs=1M count=20 oflag=direct,dsync >/dev/null 2>&1
 15/06/2020 22:42:24 INFO Executing : dd bs=1M seek=2324783104 if=/dev/zero of=/dev/sdc1 count=20 oflag=seek_bytes,direct,dsync >/dev/null 2>&1
 15/06/2020 22:42:24 INFO cache_part_path = /dev/sdc1
 15/06/2020 22:42:24 INFO done
 15/06/2020 22:42:24 INFO Step 3 : Creating Physical Volumes :
 15/06/2020 22:42:24 INFO ------------------------------------
 15/06/2020 22:42:25 INFO done
 15/06/2020 22:42:25 INFO Step 4 : Creating Volume Group :
 15/06/2020 22:42:25 INFO --------------------------------
 15/06/2020 22:42:25 INFO Running command: /sbin/vgs --noheadings --readonly --units=g --separator=";" -o vg_name,pv_count,lv_count,snap_count,vg_attr,vg_size,vg_free,vg_free_count
 15/06/2020 22:42:25 INFO stdout ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.14";"2";"1";"0";"wz--n-";"6150.14g";"0g";"0
 15/06/2020 22:42:25 INFO stdout ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.15";"2";"1";"0";"wz--n-";"6150.14g";"0g";"0
 15/06/2020 22:42:25 INFO Running command: /sbin/vgs --noheadings --readonly --units=g --separator=";" -o vg_name,pv_count,lv_count,snap_count,vg_attr,vg_size,vg_free,vg_free_count
 15/06/2020 22:42:25 INFO stdout ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.14";"2";"1";"0";"wz--n-";"6150.14g";"0g";"0
 15/06/2020 22:42:25 INFO stdout ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.15";"2";"1";"0";"wz--n-";"6150.14g";"0g";"0
 15/06/2020 22:42:25 INFO stdout ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.16";"2";"0";"0";"wz--n-";"6150.14g";"6150.14g";"1574437
 15/06/2020 22:42:25 INFO vg_name = ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.16
 15/06/2020 22:42:25 INFO done
 15/06/2020 22:42:25 INFO Step 5 : Creating Logical Volumes :
 15/06/2020 22:42:25 INFO -----------------------------------
 15/06/2020 22:42:25 INFO main_lv_name = main
 15/06/2020 22:42:25 INFO cache_lv_name = cache
 15/06/2020 22:42:25 INFO done
 15/06/2020 22:42:25 INFO Step 6 : Building dm-writecache :
 15/06/2020 22:42:25 INFO ---------------------------------
 15/06/2020 22:42:26 INFO cmd = lvchange -a n ps-55c7bcf4-46dd-4bf0-9674-cbc3f9779a39-wc-osd.16/cache ---> done
 15/06/2020 22:42:26 INFO cmd = modprobe dm-writecache ---> done
 15/06/2020 22:42:37 INFO LeaderElectionBase dropping old sessions
 15/06/2020 22:42:37 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff22613b850>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
 15/06/2020 22:42:39 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff22613ba90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
 15/06/2020 22:42:43 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff22613bcd0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
 15/06/2020 22:42:47 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f89910>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
 15/06/2020 22:42:49 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f89b50>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
 15/06/2020 22:42:51 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff22613bf10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
 15/06/2020 22:42:53 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f89d90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
 15/06/2020 22:43:01 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880fa2050>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
 15/06/2020 22:43:04 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aab410>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
 15/06/2020 22:43:06 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aab650>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
 15/06/2020 22:43:07 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff226159190>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
 15/06/2020 22:43:10 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aab890>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
 15/06/2020 22:43:17 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880fa2250>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
 15/06/2020 22:43:18 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aabad0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
 15/06/2020 22:43:34 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aabd10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
 15/06/2020 22:43:39 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2261593d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
 15/06/2020 22:43:41 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff22613bc10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
 15/06/2020 22:43:43 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff22613bb90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
 15/06/2020 22:43:47 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff226159090>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
 15/06/2020 22:43:49 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880fa2490>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
 15/06/2020 22:43:51 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f89cd0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
 15/06/2020 22:43:53 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f89e90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
 15/06/2020 22:43:55 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff226159710>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
 15/06/2020 22:43:57 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880fa2490>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
 15/06/2020 22:44:05 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880fa2790>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
 15/06/2020 22:44:06 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aabf90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
 15/06/2020 22:44:08 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aab7d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
 15/06/2020 22:44:10 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aabc50>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
 15/06/2020 22:44:11 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff226159450>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
 15/06/2020 22:44:14 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aab8d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
 15/06/2020 22:44:21 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880fa2510>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
 15/06/2020 22:44:22 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047ca150>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
 15/06/2020 22:44:38 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047ca050>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
 15/06/2020 22:44:43 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2261599d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
 15/06/2020 22:44:43 ERROR HTTPConnectionPool(host='127.0.0.1', port=8500): Max retries exceeded with url: /v1/kv/PetaSAN/Config/Files?recurse=1 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff226159bd0>: Failed to establish a new connection: [Errno 111] Connection refused',))
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/file_sync_manager.py", line 75, in sync
    index, data = base.watch(self.root_path, current_index)
  File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/base.py", line 49, in watch
    return cons.kv.get(key, index=current_index, recurse=True)
  File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 554, in get
    params=params)
  File "/usr/lib/python2.7/dist-packages/retrying.py", line 49, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/retrying.py", line 212, in call
    raise attempt.get()
  File "/usr/lib/python2.7/dist-packages/retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "/usr/lib/python2.7/dist-packages/retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/ps_consul.py", line 66, in get
    res = self.response(self.session.get(uri, verify=self.verify))
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 533, in get
    return self.request('GET', url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 520, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 630, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 508, in send
    raise ConnectionError(e, request=request)
ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=8500): Max retries exceeded with url: /v1/kv/PetaSAN/Config/Files?recurse=1 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff226159bd0>: Failed to establish a new connection: [Errno 111] Connection refused',))
 15/06/2020 22:44:46 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260b3750>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
 15/06/2020 22:44:48 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260b3950>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
 15/06/2020 22:44:52 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260b3b90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
 15/06/2020 22:44:53 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880fa2a90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
 15/06/2020 22:44:53 ERROR LeaderElectionBase exception dropping old sessions
 15/06/2020 22:45:00 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260b3dd0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
 15/06/2020 22:45:03 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f82a90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
 15/06/2020 22:45:05 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f82c90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
 15/06/2020 22:45:09 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880fa2190>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
 15/06/2020 22:45:10 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c14aab190>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
 15/06/2020 22:45:10 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader
 15/06/2020 22:45:10 ERROR HTTPConnectionPool(host='127.0.0.1', port=8500): Max retries exceeded with url: /v1/kv/PetaSAN/Services/ClusterLeader (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047ca550>: Failed to establish a new connection: [Errno 111] Connection refused',))
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/api.py", line 446, in get_key
    index, data = cons.kv.get(key)
  File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 554, in get
    params=params)
  File "/usr/lib/python2.7/dist-packages/retrying.py", line 49, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/retrying.py", line 212, in call
    raise attempt.get()
  File "/usr/lib/python2.7/dist-packages/retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "/usr/lib/python2.7/dist-packages/retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/ps_consul.py", line 66, in get
    res = self.response(self.session.get(uri, verify=self.verify))
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 533, in get
    return self.request('GET', url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 520, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 630, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 508, in send
    raise ConnectionError(e, request=request)
ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=8500): Max retries exceeded with url: /v1/kv/PetaSAN/Services/ClusterLeader (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047ca550>: Failed to establish a new connection: [Errno 111] Connection refused',))
 15/06/2020 22:45:16 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff226159150>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
 15/06/2020 22:45:17 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f898d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
 15/06/2020 22:45:33 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f82f90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
 15/06/2020 22:45:48 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260b34d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
 15/06/2020 22:45:50 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260b3750>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
 15/06/2020 22:45:52 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260b3b90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
 15/06/2020 22:45:56 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260b3fd0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
 15/06/2020 22:46:04 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260b3c10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
 15/06/2020 22:46:05 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f3b1d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
 15/06/2020 22:46:07 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f82a10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
 15/06/2020 22:46:09 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f82f10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
 15/06/2020 22:46:10 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047af850>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
 15/06/2020 22:46:12 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047afa90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
 15/06/2020 22:46:13 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f82dd0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
 15/06/2020 22:46:16 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047afcd0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
 15/06/2020 22:46:20 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260bb110>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
 15/06/2020 22:46:21 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f3b390>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
 15/06/2020 22:46:24 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047aff10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
 15/06/2020 22:46:37 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f3b250>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
 15/06/2020 22:46:40 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c04734190>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
 15/06/2020 22:46:52 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260bb450>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
 15/06/2020 22:46:52 ERROR HTTPConnectionPool(host='127.0.0.1', port=8500): Max retries exceeded with url: /v1/kv/PetaSAN/Config/Files?recurse=1 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260bb650>: Failed to establish a new connection: [Errno 111] Connection refused',))
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/file_sync_manager.py", line 75, in sync
    index, data = base.watch(self.root_path, current_index)
  File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/base.py", line 49, in watch
    return cons.kv.get(key, index=current_index, recurse=True)
  File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 554, in get
    params=params)
  File "/usr/lib/python2.7/dist-packages/retrying.py", line 49, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/retrying.py", line 212, in call
    raise attempt.get()
  File "/usr/lib/python2.7/dist-packages/retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "/usr/lib/python2.7/dist-packages/retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/ps_consul.py", line 66, in get
    res = self.response(self.session.get(uri, verify=self.verify))
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 533, in get
    return self.request('GET', url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 520, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 630, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 508, in send
    raise ConnectionError(e, request=request)
ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=8500): Max retries exceeded with url: /v1/kv/PetaSAN/Config/Files?recurse=1 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260bb650>: Failed to establish a new connection: [Errno 111] Connection refused',))
 15/06/2020 22:46:54 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260bba90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
 15/06/2020 22:46:56 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260bbc90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
 15/06/2020 22:47:00 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260bbed0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
 15/06/2020 22:47:08 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260c3150>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
 15/06/2020 22:47:10 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f3b790>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
 15/06/2020 22:47:10 ERROR LeaderElectionBase exception dropping old sessions
 15/06/2020 22:47:12 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047343d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
 15/06/2020 22:47:14 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047afc10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
 15/06/2020 22:47:16 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047afb90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
 15/06/2020 22:47:20 WARNING Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f44790>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
 15/06/2020 22:47:20 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c04734090>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
 15/06/2020 22:47:22 WARNING Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f44990>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
 15/06/2020 22:47:24 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff2260c3390>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Config/Files?recurse=1
 15/06/2020 22:47:26 WARNING Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f44bd0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
 15/06/2020 22:47:28 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c047346d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/kv/PetaSAN/Services/ClusterLeader
 15/06/2020 22:47:34 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff880f44e10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v1/session/list
#1

zamasaki158
13 Posts
June 22, 2020, 5:34 am

I added more RAM and the problem was solved.

#2

admin
2,918 Posts
June 22, 2020, 9:42 pm

Great.  remember that write cache requites 2% of SSD space in memory. A 100 G SSD partition requires 2G of RAM.

#3