Striping not working
xConnection
11 Posts
December 5, 2019, 11:40 amQuote from xConnection on December 5, 2019, 11:40 amHello,
If I create image with custom striping settings, iSCSI path assignment is not working (for example, rbd create test-image --size 10240 --image-feature striping --image-feature layering --stripe-unit 32768 --stripe-count 16).
After attaching image, iSCSI IP addresses are not assigned to any nodes.
If I create standard image without custom striping, everything is okay.
Does PetaSAN support striping?
Thanks.
Hello,
If I create image with custom striping settings, iSCSI path assignment is not working (for example, rbd create test-image --size 10240 --image-feature striping --image-feature layering --stripe-unit 32768 --stripe-count 16).
After attaching image, iSCSI IP addresses are not assigned to any nodes.
If I create standard image without custom striping, everything is okay.
Does PetaSAN support striping?
Thanks.
xConnection
11 Posts
December 5, 2019, 11:58 amQuote from xConnection on December 5, 2019, 11:58 amLog:
05/12/2019 16:53:43 INFO include_wwn_fsid_tag() is true
05/12/2019 16:53:43 INFO add disk wwn is e0ae0c4f00021
05/12/2019 16:53:45 INFO Successfully created key 00021 for new disk.
05/12/2019 16:53:45 INFO Successfully created key /00021/1 for new disk.
05/12/2019 16:53:45 INFO Successfully created key /00021/2 for new disk.
05/12/2019 16:53:45 INFO Successfully created key /00021/3 for new disk.
05/12/2019 16:53:45 INFO Successfully created key /00021/4 for new disk.
05/12/2019 16:53:45 INFO Successfully created key /00021/5 for new disk.
05/12/2019 16:53:45 INFO Successfully created key /00021/6 for new disk.
05/12/2019 16:53:50 INFO Found pool:rbd for disk:00021 via ceph
05/12/2019 16:53:50 ERROR Could not acquire path 00021/4
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 474, in __acquire_path
status = ceph_api.map_iamge(image_name,pool)
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/ceph/api.py", line 769, in map_iamge
logger.errord("map error is {}".format(err))
AttributeError: 'RootLogger' object has no attribute 'errord'
05/12/2019 16:53:50 ERROR Error during process.
05/12/2019 16:53:50 ERROR 'RootLogger' object has no attribute 'errord'
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 98, in start
self.__process()
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 132, in __process
while self.__do_process() != True:
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 205, in __do_process
self.__acquire_path(str(path), self.__paths_consul_unlocked_firstborn.get(path))
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 535, in __acquire_path
raise e
AttributeError: 'RootLogger' object has no attribute 'errord'
05/12/2019 16:53:53 INFO Unlock path 00021/4
05/12/2019 16:53:53 INFO PetaSAN unlocked any consul locks not configured in this node.
05/12/2019 16:53:56 INFO Found pool:rbd for disk:00021 via ceph
05/12/2019 16:53:56 ERROR Could not acquire path 00021/3
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 474, in __acquire_path
status = ceph_api.map_iamge(image_name,pool)
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/ceph/api.py", line 769, in map_iamge
logger.errord("map error is {}".format(err))
AttributeError: 'RootLogger' object has no attribute 'errord'
05/12/2019 16:53:56 ERROR Error during process.
05/12/2019 16:53:56 ERROR 'RootLogger' object has no attribute 'errord'
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 98, in start
self.__process()
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 132, in __process
while self.__do_process() != True:
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 205, in __do_process
self.__acquire_path(str(path), self.__paths_consul_unlocked_firstborn.get(path))
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 535, in __acquire_path
raise e
AttributeError: 'RootLogger' object has no attribute 'errord'
05/12/2019 16:54:01 INFO Unlock path 00021/3
05/12/2019 16:54:01 INFO PetaSAN unlocked any consul locks not configured in this node.
05/12/2019 16:54:04 INFO Found pool:rbd for disk:00021 via ceph
05/12/2019 16:54:11 ERROR Could not acquire path 00021/3
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 474, in __acquire_path
status = ceph_api.map_iamge(image_name,pool)
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/ceph/api.py", line 769, in map_iamge
logger.errord("map error is {}".format(err))
AttributeError: 'RootLogger' object has no attribute 'errord'
05/12/2019 16:54:11 ERROR Error during process.
05/12/2019 16:54:11 ERROR 'RootLogger' object has no attribute 'errord'
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 98, in start
self.__process()
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 132, in __process
while self.__do_process() != True:
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 205, in __do_process
self.__acquire_path(str(path), self.__paths_consul_unlocked_firstborn.get(path))
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 535, in __acquire_path
raise e
AttributeError: 'RootLogger' object has no attribute 'errord'
05/12/2019 16:54:18 INFO Unlock path 00021/3
05/12/2019 16:54:18 INFO PetaSAN unlocked any consul locks not configured in this node.
Log:
05/12/2019 16:53:43 INFO include_wwn_fsid_tag() is true
05/12/2019 16:53:43 INFO add disk wwn is e0ae0c4f00021
05/12/2019 16:53:45 INFO Successfully created key 00021 for new disk.
05/12/2019 16:53:45 INFO Successfully created key /00021/1 for new disk.
05/12/2019 16:53:45 INFO Successfully created key /00021/2 for new disk.
05/12/2019 16:53:45 INFO Successfully created key /00021/3 for new disk.
05/12/2019 16:53:45 INFO Successfully created key /00021/4 for new disk.
05/12/2019 16:53:45 INFO Successfully created key /00021/5 for new disk.
05/12/2019 16:53:45 INFO Successfully created key /00021/6 for new disk.
05/12/2019 16:53:50 INFO Found pool:rbd for disk:00021 via ceph
05/12/2019 16:53:50 ERROR Could not acquire path 00021/4
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 474, in __acquire_path
status = ceph_api.map_iamge(image_name,pool)
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/ceph/api.py", line 769, in map_iamge
logger.errord("map error is {}".format(err))
AttributeError: 'RootLogger' object has no attribute 'errord'
05/12/2019 16:53:50 ERROR Error during process.
05/12/2019 16:53:50 ERROR 'RootLogger' object has no attribute 'errord'
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 98, in start
self.__process()
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 132, in __process
while self.__do_process() != True:
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 205, in __do_process
self.__acquire_path(str(path), self.__paths_consul_unlocked_firstborn.get(path))
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 535, in __acquire_path
raise e
AttributeError: 'RootLogger' object has no attribute 'errord'
05/12/2019 16:53:53 INFO Unlock path 00021/4
05/12/2019 16:53:53 INFO PetaSAN unlocked any consul locks not configured in this node.
05/12/2019 16:53:56 INFO Found pool:rbd for disk:00021 via ceph
05/12/2019 16:53:56 ERROR Could not acquire path 00021/3
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 474, in __acquire_path
status = ceph_api.map_iamge(image_name,pool)
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/ceph/api.py", line 769, in map_iamge
logger.errord("map error is {}".format(err))
AttributeError: 'RootLogger' object has no attribute 'errord'
05/12/2019 16:53:56 ERROR Error during process.
05/12/2019 16:53:56 ERROR 'RootLogger' object has no attribute 'errord'
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 98, in start
self.__process()
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 132, in __process
while self.__do_process() != True:
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 205, in __do_process
self.__acquire_path(str(path), self.__paths_consul_unlocked_firstborn.get(path))
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 535, in __acquire_path
raise e
AttributeError: 'RootLogger' object has no attribute 'errord'
05/12/2019 16:54:01 INFO Unlock path 00021/3
05/12/2019 16:54:01 INFO PetaSAN unlocked any consul locks not configured in this node.
05/12/2019 16:54:04 INFO Found pool:rbd for disk:00021 via ceph
05/12/2019 16:54:11 ERROR Could not acquire path 00021/3
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 474, in __acquire_path
status = ceph_api.map_iamge(image_name,pool)
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/ceph/api.py", line 769, in map_iamge
logger.errord("map error is {}".format(err))
AttributeError: 'RootLogger' object has no attribute 'errord'
05/12/2019 16:54:11 ERROR Error during process.
05/12/2019 16:54:11 ERROR 'RootLogger' object has no attribute 'errord'
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 98, in start
self.__process()
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 132, in __process
while self.__do_process() != True:
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 205, in __do_process
self.__acquire_path(str(path), self.__paths_consul_unlocked_firstborn.get(path))
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 535, in __acquire_path
raise e
AttributeError: 'RootLogger' object has no attribute 'errord'
05/12/2019 16:54:18 INFO Unlock path 00021/3
05/12/2019 16:54:18 INFO PetaSAN unlocked any consul locks not configured in this node.
Last edited on December 5, 2019, 12:00 pm by xConnection · #2
xConnection
11 Posts
December 5, 2019, 12:08 pmQuote from xConnection on December 5, 2019, 12:08 pmrbd image 'image-00021':
size 10 GiB in 2560 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: 479a3d80ca1d10
block_name_prefix: rbd_data.479a3d80ca1d10
format: 2
features: layering, striping
op_features:
flags:
create_timestamp: Thu Dec 5 16:53:17 2019
access_timestamp: Thu Dec 5 16:53:17 2019
modify_timestamp: Thu Dec 5 16:53:17 2019
stripe unit: 32 KiB
stripe count: 16
rbd image 'image-00021':
size 10 GiB in 2560 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: 479a3d80ca1d10
block_name_prefix: rbd_data.479a3d80ca1d10
format: 2
features: layering, striping
op_features:
flags:
create_timestamp: Thu Dec 5 16:53:17 2019
access_timestamp: Thu Dec 5 16:53:17 2019
modify_timestamp: Thu Dec 5 16:53:17 2019
stripe unit: 32 KiB
stripe count: 16
xConnection
11 Posts
December 5, 2019, 12:45 pmQuote from xConnection on December 5, 2019, 12:45 pmroot@ceph2:~# rbd map test-image
rbd: sysfs write failed
In some cases useful info is found in syslog - try "dmesg | tail".
rbd: map failed: (22) Invalid argument
root@ceph2:~# dmesg | tail
[260461.139776] libceph: mon4 100.64.1.2:3300 socket closed (con state CONNECTING)
[260462.164212] libceph: mon5 100.64.1.2:6789 session established
[260462.164331] libceph: client4719962 fsid e0ae0c4f-8db8-47e2-9dcd-d04b4529acf0
[260462.179484] rbd: rbd0: capacity 10737418240 features 0x1
[261000.775712] libceph: mon5 100.64.1.2:6789 session established
[261000.775868] libceph: client4723145 fsid e0ae0c4f-8db8-47e2-9dcd-d04b4529acf0
[261000.792583] rbd: image test-image: unsupported stripe unit (got 32768 want 4194304)
[261092.552202] libceph: mon3 100.64.1.1:6789 session established
[261092.552536] libceph: client4731676 fsid e0ae0c4f-8db8-47e2-9dcd-d04b4529acf0
[261092.557507] rbd: image test-image: unsupported stripe unit (got 32768 want 4194304)
root@ceph2:~# rbd map test-image
rbd: sysfs write failed
In some cases useful info is found in syslog - try "dmesg | tail".
rbd: map failed: (22) Invalid argument
root@ceph2:~# dmesg | tail
[260461.139776] libceph: mon4 100.64.1.2:3300 socket closed (con state CONNECTING)
[260462.164212] libceph: mon5 100.64.1.2:6789 session established
[260462.164331] libceph: client4719962 fsid e0ae0c4f-8db8-47e2-9dcd-d04b4529acf0
[260462.179484] rbd: rbd0: capacity 10737418240 features 0x1
[261000.775712] libceph: mon5 100.64.1.2:6789 session established
[261000.775868] libceph: client4723145 fsid e0ae0c4f-8db8-47e2-9dcd-d04b4529acf0
[261000.792583] rbd: image test-image: unsupported stripe unit (got 32768 want 4194304)
[261092.552202] libceph: mon3 100.64.1.1:6789 session established
[261092.552536] libceph: client4731676 fsid e0ae0c4f-8db8-47e2-9dcd-d04b4529acf0
[261092.557507] rbd: image test-image: unsupported stripe unit (got 32768 want 4194304)
admin
2,930 Posts
December 5, 2019, 1:06 pmQuote from admin on December 5, 2019, 1:06 pmWe internally support this prior to v 2.3.1, we no longer support as we saw it leads to degraded performance for most workloads. we did not support it in ui. if you want to experiment, you can test with 2.3.0, but again we do not recommend it in production. The cases where it could be of use is when you have many more disks / osds than your current client io depth, else it just multiplies your latencies.
We internally support this prior to v 2.3.1, we no longer support as we saw it leads to degraded performance for most workloads. we did not support it in ui. if you want to experiment, you can test with 2.3.0, but again we do not recommend it in production. The cases where it could be of use is when you have many more disks / osds than your current client io depth, else it just multiplies your latencies.
xConnection
11 Posts
December 5, 2019, 1:50 pmQuote from xConnection on December 5, 2019, 1:50 pmThanks for the quick answer!
We are using VMware ESXi and we wanted to test striping in v 2.3.1.
There is no "easy" way to enable striping support by ourselves in v 2.3.1 or later?
Why not to keep this option so that users can decide whether they need striping in their particular situation (considering their workload, number of OSDs, etc.), especially regarding that striping support can be enabled only via cli?
Thanks for the quick answer!
We are using VMware ESXi and we wanted to test striping in v 2.3.1.
There is no "easy" way to enable striping support by ourselves in v 2.3.1 or later?
Why not to keep this option so that users can decide whether they need striping in their particular situation (considering their workload, number of OSDs, etc.), especially regarding that striping support can be enabled only via cli?
admin
2,930 Posts
December 5, 2019, 2:06 pmQuote from admin on December 5, 2019, 2:06 pmWe are using the kernel client and kernel based on SUSE SLE 15 SP1 which is 4.12.x based. kernel support for stripping was added in 4.17 and would be available in future SLE 15 SP2 kernel. As indicated earlier, we did add striping support in our earlier kernels ourselves on top of the SUSE kernel but stopped in 2.3.1 as it is not only something we do not support but also did not see performance gains to justify this.
You can test performance if striping via rbd bench tool, which uses user space rather than kernel io, and see if it really makes a difference.
We are using the kernel client and kernel based on SUSE SLE 15 SP1 which is 4.12.x based. kernel support for stripping was added in 4.17 and would be available in future SLE 15 SP2 kernel. As indicated earlier, we did add striping support in our earlier kernels ourselves on top of the SUSE kernel but stopped in 2.3.1 as it is not only something we do not support but also did not see performance gains to justify this.
You can test performance if striping via rbd bench tool, which uses user space rather than kernel io, and see if it really makes a difference.
xConnection
11 Posts
December 23, 2019, 6:43 amQuote from xConnection on December 23, 2019, 6:43 amThank you for information.
Now it is clear that it is not PetaSAN problem, but SUSE kernel.
Thank you for information.
Now it is clear that it is not PetaSAN problem, but SUSE kernel.
admin
2,930 Posts
December 23, 2019, 4:57 pmQuote from admin on December 23, 2019, 4:57 pmI would not say so, we are responsible for our own kernel. The issue is as stated, we had added internal (non ui) support for this ourselves till version 2.3.1, we did not see it was justified as we saw no performance advantages. You can test striping features using rbd bench cli tool as it uses userspace librbd rather than kernel, if you see any benefit lets us know and we can re-add it as an internal feature.
I would not say so, we are responsible for our own kernel. The issue is as stated, we had added internal (non ui) support for this ourselves till version 2.3.1, we did not see it was justified as we saw no performance advantages. You can test striping features using rbd bench cli tool as it uses userspace librbd rather than kernel, if you see any benefit lets us know and we can re-add it as an internal feature.
Striping not working
xConnection
11 Posts
Quote from xConnection on December 5, 2019, 11:40 amHello,
If I create image with custom striping settings, iSCSI path assignment is not working (for example, rbd create test-image --size 10240 --image-feature striping --image-feature layering --stripe-unit 32768 --stripe-count 16).
After attaching image, iSCSI IP addresses are not assigned to any nodes.
If I create standard image without custom striping, everything is okay.
Does PetaSAN support striping?
Thanks.
Hello,
If I create image with custom striping settings, iSCSI path assignment is not working (for example, rbd create test-image --size 10240 --image-feature striping --image-feature layering --stripe-unit 32768 --stripe-count 16).
After attaching image, iSCSI IP addresses are not assigned to any nodes.
If I create standard image without custom striping, everything is okay.
Does PetaSAN support striping?
Thanks.
xConnection
11 Posts
Quote from xConnection on December 5, 2019, 11:58 amLog:
05/12/2019 16:53:43 INFO include_wwn_fsid_tag() is true
05/12/2019 16:53:43 INFO add disk wwn is e0ae0c4f00021
05/12/2019 16:53:45 INFO Successfully created key 00021 for new disk.
05/12/2019 16:53:45 INFO Successfully created key /00021/1 for new disk.
05/12/2019 16:53:45 INFO Successfully created key /00021/2 for new disk.
05/12/2019 16:53:45 INFO Successfully created key /00021/3 for new disk.
05/12/2019 16:53:45 INFO Successfully created key /00021/4 for new disk.
05/12/2019 16:53:45 INFO Successfully created key /00021/5 for new disk.
05/12/2019 16:53:45 INFO Successfully created key /00021/6 for new disk.
05/12/2019 16:53:50 INFO Found pool:rbd for disk:00021 via ceph
05/12/2019 16:53:50 ERROR Could not acquire path 00021/4
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 474, in __acquire_path
status = ceph_api.map_iamge(image_name,pool)
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/ceph/api.py", line 769, in map_iamge
logger.errord("map error is {}".format(err))
AttributeError: 'RootLogger' object has no attribute 'errord'
05/12/2019 16:53:50 ERROR Error during process.
05/12/2019 16:53:50 ERROR 'RootLogger' object has no attribute 'errord'
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 98, in start
self.__process()
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 132, in __process
while self.__do_process() != True:
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 205, in __do_process
self.__acquire_path(str(path), self.__paths_consul_unlocked_firstborn.get(path))
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 535, in __acquire_path
raise e
AttributeError: 'RootLogger' object has no attribute 'errord'
05/12/2019 16:53:53 INFO Unlock path 00021/4
05/12/2019 16:53:53 INFO PetaSAN unlocked any consul locks not configured in this node.
05/12/2019 16:53:56 INFO Found pool:rbd for disk:00021 via ceph
05/12/2019 16:53:56 ERROR Could not acquire path 00021/3
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 474, in __acquire_path
status = ceph_api.map_iamge(image_name,pool)
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/ceph/api.py", line 769, in map_iamge
logger.errord("map error is {}".format(err))
AttributeError: 'RootLogger' object has no attribute 'errord'
05/12/2019 16:53:56 ERROR Error during process.
05/12/2019 16:53:56 ERROR 'RootLogger' object has no attribute 'errord'
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 98, in start
self.__process()
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 132, in __process
while self.__do_process() != True:
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 205, in __do_process
self.__acquire_path(str(path), self.__paths_consul_unlocked_firstborn.get(path))
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 535, in __acquire_path
raise e
AttributeError: 'RootLogger' object has no attribute 'errord'
05/12/2019 16:54:01 INFO Unlock path 00021/3
05/12/2019 16:54:01 INFO PetaSAN unlocked any consul locks not configured in this node.
05/12/2019 16:54:04 INFO Found pool:rbd for disk:00021 via ceph
05/12/2019 16:54:11 ERROR Could not acquire path 00021/3
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 474, in __acquire_path
status = ceph_api.map_iamge(image_name,pool)
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/ceph/api.py", line 769, in map_iamge
logger.errord("map error is {}".format(err))
AttributeError: 'RootLogger' object has no attribute 'errord'
05/12/2019 16:54:11 ERROR Error during process.
05/12/2019 16:54:11 ERROR 'RootLogger' object has no attribute 'errord'
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 98, in start
self.__process()
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 132, in __process
while self.__do_process() != True:
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 205, in __do_process
self.__acquire_path(str(path), self.__paths_consul_unlocked_firstborn.get(path))
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 535, in __acquire_path
raise e
AttributeError: 'RootLogger' object has no attribute 'errord'
05/12/2019 16:54:18 INFO Unlock path 00021/3
05/12/2019 16:54:18 INFO PetaSAN unlocked any consul locks not configured in this node.
Log:
05/12/2019 16:53:43 INFO include_wwn_fsid_tag() is true
05/12/2019 16:53:43 INFO add disk wwn is e0ae0c4f00021
05/12/2019 16:53:45 INFO Successfully created key 00021 for new disk.
05/12/2019 16:53:45 INFO Successfully created key /00021/1 for new disk.
05/12/2019 16:53:45 INFO Successfully created key /00021/2 for new disk.
05/12/2019 16:53:45 INFO Successfully created key /00021/3 for new disk.
05/12/2019 16:53:45 INFO Successfully created key /00021/4 for new disk.
05/12/2019 16:53:45 INFO Successfully created key /00021/5 for new disk.
05/12/2019 16:53:45 INFO Successfully created key /00021/6 for new disk.
05/12/2019 16:53:50 INFO Found pool:rbd for disk:00021 via ceph
05/12/2019 16:53:50 ERROR Could not acquire path 00021/4
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 474, in __acquire_path
status = ceph_api.map_iamge(image_name,pool)
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/ceph/api.py", line 769, in map_iamge
logger.errord("map error is {}".format(err))
AttributeError: 'RootLogger' object has no attribute 'errord'
05/12/2019 16:53:50 ERROR Error during process.
05/12/2019 16:53:50 ERROR 'RootLogger' object has no attribute 'errord'
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 98, in start
self.__process()
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 132, in __process
while self.__do_process() != True:
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 205, in __do_process
self.__acquire_path(str(path), self.__paths_consul_unlocked_firstborn.get(path))
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 535, in __acquire_path
raise e
AttributeError: 'RootLogger' object has no attribute 'errord'
05/12/2019 16:53:53 INFO Unlock path 00021/4
05/12/2019 16:53:53 INFO PetaSAN unlocked any consul locks not configured in this node.
05/12/2019 16:53:56 INFO Found pool:rbd for disk:00021 via ceph
05/12/2019 16:53:56 ERROR Could not acquire path 00021/3
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 474, in __acquire_path
status = ceph_api.map_iamge(image_name,pool)
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/ceph/api.py", line 769, in map_iamge
logger.errord("map error is {}".format(err))
AttributeError: 'RootLogger' object has no attribute 'errord'
05/12/2019 16:53:56 ERROR Error during process.
05/12/2019 16:53:56 ERROR 'RootLogger' object has no attribute 'errord'
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 98, in start
self.__process()
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 132, in __process
while self.__do_process() != True:
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 205, in __do_process
self.__acquire_path(str(path), self.__paths_consul_unlocked_firstborn.get(path))
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 535, in __acquire_path
raise e
AttributeError: 'RootLogger' object has no attribute 'errord'
05/12/2019 16:54:01 INFO Unlock path 00021/3
05/12/2019 16:54:01 INFO PetaSAN unlocked any consul locks not configured in this node.
05/12/2019 16:54:04 INFO Found pool:rbd for disk:00021 via ceph
05/12/2019 16:54:11 ERROR Could not acquire path 00021/3
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 474, in __acquire_path
status = ceph_api.map_iamge(image_name,pool)
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/ceph/api.py", line 769, in map_iamge
logger.errord("map error is {}".format(err))
AttributeError: 'RootLogger' object has no attribute 'errord'
05/12/2019 16:54:11 ERROR Error during process.
05/12/2019 16:54:11 ERROR 'RootLogger' object has no attribute 'errord'
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 98, in start
self.__process()
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 132, in __process
while self.__do_process() != True:
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 205, in __do_process
self.__acquire_path(str(path), self.__paths_consul_unlocked_firstborn.get(path))
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/iscsi_service.py", line 535, in __acquire_path
raise e
AttributeError: 'RootLogger' object has no attribute 'errord'
05/12/2019 16:54:18 INFO Unlock path 00021/3
05/12/2019 16:54:18 INFO PetaSAN unlocked any consul locks not configured in this node.
xConnection
11 Posts
Quote from xConnection on December 5, 2019, 12:08 pmrbd image 'image-00021':
size 10 GiB in 2560 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: 479a3d80ca1d10
block_name_prefix: rbd_data.479a3d80ca1d10
format: 2
features: layering, striping
op_features:
flags:
create_timestamp: Thu Dec 5 16:53:17 2019
access_timestamp: Thu Dec 5 16:53:17 2019
modify_timestamp: Thu Dec 5 16:53:17 2019
stripe unit: 32 KiB
stripe count: 16
rbd image 'image-00021':
size 10 GiB in 2560 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: 479a3d80ca1d10
block_name_prefix: rbd_data.479a3d80ca1d10
format: 2
features: layering, striping
op_features:
flags:
create_timestamp: Thu Dec 5 16:53:17 2019
access_timestamp: Thu Dec 5 16:53:17 2019
modify_timestamp: Thu Dec 5 16:53:17 2019
stripe unit: 32 KiB
stripe count: 16
xConnection
11 Posts
Quote from xConnection on December 5, 2019, 12:45 pmroot@ceph2:~# rbd map test-image
rbd: sysfs write failed
In some cases useful info is found in syslog - try "dmesg | tail".
rbd: map failed: (22) Invalid argumentroot@ceph2:~# dmesg | tail
[260461.139776] libceph: mon4 100.64.1.2:3300 socket closed (con state CONNECTING)
[260462.164212] libceph: mon5 100.64.1.2:6789 session established
[260462.164331] libceph: client4719962 fsid e0ae0c4f-8db8-47e2-9dcd-d04b4529acf0
[260462.179484] rbd: rbd0: capacity 10737418240 features 0x1
[261000.775712] libceph: mon5 100.64.1.2:6789 session established
[261000.775868] libceph: client4723145 fsid e0ae0c4f-8db8-47e2-9dcd-d04b4529acf0
[261000.792583] rbd: image test-image: unsupported stripe unit (got 32768 want 4194304)
[261092.552202] libceph: mon3 100.64.1.1:6789 session established
[261092.552536] libceph: client4731676 fsid e0ae0c4f-8db8-47e2-9dcd-d04b4529acf0
[261092.557507] rbd: image test-image: unsupported stripe unit (got 32768 want 4194304)
root@ceph2:~# rbd map test-image
rbd: sysfs write failed
In some cases useful info is found in syslog - try "dmesg | tail".
rbd: map failed: (22) Invalid argument
root@ceph2:~# dmesg | tail
[260461.139776] libceph: mon4 100.64.1.2:3300 socket closed (con state CONNECTING)
[260462.164212] libceph: mon5 100.64.1.2:6789 session established
[260462.164331] libceph: client4719962 fsid e0ae0c4f-8db8-47e2-9dcd-d04b4529acf0
[260462.179484] rbd: rbd0: capacity 10737418240 features 0x1
[261000.775712] libceph: mon5 100.64.1.2:6789 session established
[261000.775868] libceph: client4723145 fsid e0ae0c4f-8db8-47e2-9dcd-d04b4529acf0
[261000.792583] rbd: image test-image: unsupported stripe unit (got 32768 want 4194304)
[261092.552202] libceph: mon3 100.64.1.1:6789 session established
[261092.552536] libceph: client4731676 fsid e0ae0c4f-8db8-47e2-9dcd-d04b4529acf0
[261092.557507] rbd: image test-image: unsupported stripe unit (got 32768 want 4194304)
admin
2,930 Posts
Quote from admin on December 5, 2019, 1:06 pmWe internally support this prior to v 2.3.1, we no longer support as we saw it leads to degraded performance for most workloads. we did not support it in ui. if you want to experiment, you can test with 2.3.0, but again we do not recommend it in production. The cases where it could be of use is when you have many more disks / osds than your current client io depth, else it just multiplies your latencies.
We internally support this prior to v 2.3.1, we no longer support as we saw it leads to degraded performance for most workloads. we did not support it in ui. if you want to experiment, you can test with 2.3.0, but again we do not recommend it in production. The cases where it could be of use is when you have many more disks / osds than your current client io depth, else it just multiplies your latencies.
xConnection
11 Posts
Quote from xConnection on December 5, 2019, 1:50 pmThanks for the quick answer!
We are using VMware ESXi and we wanted to test striping in v 2.3.1.
There is no "easy" way to enable striping support by ourselves in v 2.3.1 or later?
Why not to keep this option so that users can decide whether they need striping in their particular situation (considering their workload, number of OSDs, etc.), especially regarding that striping support can be enabled only via cli?
Thanks for the quick answer!
We are using VMware ESXi and we wanted to test striping in v 2.3.1.
There is no "easy" way to enable striping support by ourselves in v 2.3.1 or later?
Why not to keep this option so that users can decide whether they need striping in their particular situation (considering their workload, number of OSDs, etc.), especially regarding that striping support can be enabled only via cli?
admin
2,930 Posts
Quote from admin on December 5, 2019, 2:06 pmWe are using the kernel client and kernel based on SUSE SLE 15 SP1 which is 4.12.x based. kernel support for stripping was added in 4.17 and would be available in future SLE 15 SP2 kernel. As indicated earlier, we did add striping support in our earlier kernels ourselves on top of the SUSE kernel but stopped in 2.3.1 as it is not only something we do not support but also did not see performance gains to justify this.
You can test performance if striping via rbd bench tool, which uses user space rather than kernel io, and see if it really makes a difference.
We are using the kernel client and kernel based on SUSE SLE 15 SP1 which is 4.12.x based. kernel support for stripping was added in 4.17 and would be available in future SLE 15 SP2 kernel. As indicated earlier, we did add striping support in our earlier kernels ourselves on top of the SUSE kernel but stopped in 2.3.1 as it is not only something we do not support but also did not see performance gains to justify this.
You can test performance if striping via rbd bench tool, which uses user space rather than kernel io, and see if it really makes a difference.
xConnection
11 Posts
Quote from xConnection on December 23, 2019, 6:43 amThank you for information.
Now it is clear that it is not PetaSAN problem, but SUSE kernel.
Thank you for information.
Now it is clear that it is not PetaSAN problem, but SUSE kernel.
admin
2,930 Posts
Quote from admin on December 23, 2019, 4:57 pmI would not say so, we are responsible for our own kernel. The issue is as stated, we had added internal (non ui) support for this ourselves till version 2.3.1, we did not see it was justified as we saw no performance advantages. You can test striping features using rbd bench cli tool as it uses userspace librbd rather than kernel, if you see any benefit lets us know and we can re-add it as an internal feature.
I would not say so, we are responsible for our own kernel. The issue is as stated, we had added internal (non ui) support for this ourselves till version 2.3.1, we did not see it was justified as we saw no performance advantages. You can test striping features using rbd bench cli tool as it uses userspace librbd rather than kernel, if you see any benefit lets us know and we can re-add it as an internal feature.