About stripe!
Aliaksei Nazarenka
9 Posts
June 27, 2017, 9:27 amQuote from Aliaksei Nazarenka on June 27, 2017, 9:27 amGood afternoon!
I would like to ask how you managed to set the value --stripe-count 1 --stripe-unit 4194304, because when I try to do this I get the message: "stripe unit is not a factor of the object size". I shall be very grateful for the answer.
Good afternoon!
I would like to ask how you managed to set the value --stripe-count 1 --stripe-unit 4194304, because when I try to do this I get the message: "stripe unit is not a factor of the object size". I shall be very grateful for the answer.
admin
2,930 Posts
June 27, 2017, 12:55 pmQuote from admin on June 27, 2017, 12:55 pmCurrently we do not support advanced striping. To enable striping you need to specify striping "feature" when adding an rbd image, else it will use the default of 4MB stripe size which is what you see.
We do have it in our roadmap but we delayed it because:
- We did not see any benchmarks that show it makes performance better. The only use case that is said to take advantage of this is when you have low concurrency load (more cores/disks than the number of concurrent ios ) and have small sequential io block sizes ( most sequential io block size tend to be large, small io tend to be random ) .
- Currently advanced striping feature is supported in user space librbd library, we use the kernel rbd access to make it faster working with LIO kernel target..it will be too much effort for us to do it right now.
Currently we do not support advanced striping. To enable striping you need to specify striping "feature" when adding an rbd image, else it will use the default of 4MB stripe size which is what you see.
We do have it in our roadmap but we delayed it because:
- We did not see any benchmarks that show it makes performance better. The only use case that is said to take advantage of this is when you have low concurrency load (more cores/disks than the number of concurrent ios ) and have small sequential io block sizes ( most sequential io block size tend to be large, small io tend to be random ) .
- Currently advanced striping feature is supported in user space librbd library, we use the kernel rbd access to make it faster working with LIO kernel target..it will be too much effort for us to do it right now.
Aliaksei Nazarenka
9 Posts
June 27, 2017, 1:02 pmQuote from Aliaksei Nazarenka on June 27, 2017, 1:02 pmHave you conducted any tests on the optimal size of objects? What can you advise?
Have you conducted any tests on the optimal size of objects? What can you advise?
Aliaksei Nazarenka
9 Posts
June 27, 2017, 1:07 pmQuote from Aliaksei Nazarenka on June 27, 2017, 1:07 pmWhen I try to enable the function: "rbd create lun_3 --size 1024 --pool rbd --image-format=2 --image-feature striping", he gives me: "must specify both of stripe-unit and stripe-count when specify striping features"
When I try to enable the function: "rbd create lun_3 --size 1024 --pool rbd --image-format=2 --image-feature striping", he gives me: "must specify both of stripe-unit and stripe-count when specify striping features"
admin
2,930 Posts
June 27, 2017, 2:06 pmQuote from admin on June 27, 2017, 2:06 pmHi,
Actually as i mentioned above, the default stripe size of 4M (this is a Ceph rbd default value) is the only value that works in PetaSAN. Changing this will not work ! This is because it is a features supported by user space librbd, so a user space application can use it, but is not supported in the Linux kernel.
I also know it is only useful in very limited cases as i described earlier.
If however you just need it for your own testing outside of PetaSAN, you can for example create images using
rbd create lun-test1 --size 1024 --pool rbd --image-format=2 --image-feature striping --stripe-unit 1048576 --stripe-count 4 --cluster demo
rbd create lun-test2 --size 1024 --pool rbd --image-format=2 --image-feature striping --stripe-unit 65536 --stripe-count 4 --cluster demo
note if you will test performance it yourself set --rbd_cache=false
Hi,
Actually as i mentioned above, the default stripe size of 4M (this is a Ceph rbd default value) is the only value that works in PetaSAN. Changing this will not work ! This is because it is a features supported by user space librbd, so a user space application can use it, but is not supported in the Linux kernel.
I also know it is only useful in very limited cases as i described earlier.
If however you just need it for your own testing outside of PetaSAN, you can for example create images using
rbd create lun-test1 --size 1024 --pool rbd --image-format=2 --image-feature striping --stripe-unit 1048576 --stripe-count 4 --cluster demo
rbd create lun-test2 --size 1024 --pool rbd --image-format=2 --image-feature striping --stripe-unit 65536 --stripe-count 4 --cluster demo
note if you will test performance it yourself set --rbd_cache=false
Aliaksei Nazarenka
9 Posts
June 28, 2017, 5:34 amQuote from Aliaksei Nazarenka on June 28, 2017, 5:34 amWhen I try map a disk created by your recommendations, I get this:
[60616.921073] rbd: image lun_3: unsupported stripe unit (got 1048576 want 4194304)
[60670.175467] rbd: image lun_3: unsupported stripe unit (got 65536 want 4194304)
If i try this: rbd create lun_4 --size 1024 --pool rbd --image-format=2 --image-feature striping --stripe-unit 4194304 --stripe-count 1
I get: stripe unit is not a factor of the object size
Therefore, if it is easy to bring please exactly the version of the team that you have in petasan.
When I try map a disk created by your recommendations, I get this:
[60616.921073] rbd: image lun_3: unsupported stripe unit (got 1048576 want 4194304)
[60670.175467] rbd: image lun_3: unsupported stripe unit (got 65536 want 4194304)
If i try this: rbd create lun_4 --size 1024 --pool rbd --image-format=2 --image-feature striping --stripe-unit 4194304 --stripe-count 1
I get: stripe unit is not a factor of the object size
Therefore, if it is easy to bring please exactly the version of the team that you have in petasan.
Last edited on June 28, 2017, 5:37 am · #6
admin
2,930 Posts
June 28, 2017, 2:37 pmQuote from admin on June 28, 2017, 2:37 pmHi there, what i was trying to say is that you cannot change the stripe policy unless you use user level applications that talks directly to ceph using librbd user space library. It is not supported in the kernel. You cannot map your striped image to a kernel block device since it is using non default stripe size.
Hi there, what i was trying to say is that you cannot change the stripe policy unless you use user level applications that talks directly to ceph using librbd user space library. It is not supported in the kernel. You cannot map your striped image to a kernel block device since it is using non default stripe size.
Last edited on June 28, 2017, 2:37 pm · #7
About stripe!
Aliaksei Nazarenka
9 Posts
Quote from Aliaksei Nazarenka on June 27, 2017, 9:27 amGood afternoon!
I would like to ask how you managed to set the value --stripe-count 1 --stripe-unit 4194304, because when I try to do this I get the message: "stripe unit is not a factor of the object size". I shall be very grateful for the answer.
Good afternoon!
I would like to ask how you managed to set the value --stripe-count 1 --stripe-unit 4194304, because when I try to do this I get the message: "stripe unit is not a factor of the object size". I shall be very grateful for the answer.
admin
2,930 Posts
Quote from admin on June 27, 2017, 12:55 pmCurrently we do not support advanced striping. To enable striping you need to specify striping "feature" when adding an rbd image, else it will use the default of 4MB stripe size which is what you see.
We do have it in our roadmap but we delayed it because:
- We did not see any benchmarks that show it makes performance better. The only use case that is said to take advantage of this is when you have low concurrency load (more cores/disks than the number of concurrent ios ) and have small sequential io block sizes ( most sequential io block size tend to be large, small io tend to be random ) .
- Currently advanced striping feature is supported in user space librbd library, we use the kernel rbd access to make it faster working with LIO kernel target..it will be too much effort for us to do it right now.
Currently we do not support advanced striping. To enable striping you need to specify striping "feature" when adding an rbd image, else it will use the default of 4MB stripe size which is what you see.
We do have it in our roadmap but we delayed it because:
- We did not see any benchmarks that show it makes performance better. The only use case that is said to take advantage of this is when you have low concurrency load (more cores/disks than the number of concurrent ios ) and have small sequential io block sizes ( most sequential io block size tend to be large, small io tend to be random ) .
- Currently advanced striping feature is supported in user space librbd library, we use the kernel rbd access to make it faster working with LIO kernel target..it will be too much effort for us to do it right now.
Aliaksei Nazarenka
9 Posts
Quote from Aliaksei Nazarenka on June 27, 2017, 1:02 pmHave you conducted any tests on the optimal size of objects? What can you advise?
Have you conducted any tests on the optimal size of objects? What can you advise?
Aliaksei Nazarenka
9 Posts
Quote from Aliaksei Nazarenka on June 27, 2017, 1:07 pmWhen I try to enable the function: "rbd create lun_3 --size 1024 --pool rbd --image-format=2 --image-feature striping", he gives me: "must specify both of stripe-unit and stripe-count when specify striping features"
When I try to enable the function: "rbd create lun_3 --size 1024 --pool rbd --image-format=2 --image-feature striping", he gives me: "must specify both of stripe-unit and stripe-count when specify striping features"
admin
2,930 Posts
Quote from admin on June 27, 2017, 2:06 pmHi,
Actually as i mentioned above, the default stripe size of 4M (this is a Ceph rbd default value) is the only value that works in PetaSAN. Changing this will not work ! This is because it is a features supported by user space librbd, so a user space application can use it, but is not supported in the Linux kernel.
I also know it is only useful in very limited cases as i described earlier.
If however you just need it for your own testing outside of PetaSAN, you can for example create images using
rbd create lun-test1 --size 1024 --pool rbd --image-format=2 --image-feature striping --stripe-unit 1048576 --stripe-count 4 --cluster demo
rbd create lun-test2 --size 1024 --pool rbd --image-format=2 --image-feature striping --stripe-unit 65536 --stripe-count 4 --cluster demo
note if you will test performance it yourself set --rbd_cache=false
Hi,
Actually as i mentioned above, the default stripe size of 4M (this is a Ceph rbd default value) is the only value that works in PetaSAN. Changing this will not work ! This is because it is a features supported by user space librbd, so a user space application can use it, but is not supported in the Linux kernel.
I also know it is only useful in very limited cases as i described earlier.
If however you just need it for your own testing outside of PetaSAN, you can for example create images using
rbd create lun-test1 --size 1024 --pool rbd --image-format=2 --image-feature striping --stripe-unit 1048576 --stripe-count 4 --cluster demo
rbd create lun-test2 --size 1024 --pool rbd --image-format=2 --image-feature striping --stripe-unit 65536 --stripe-count 4 --cluster demo
note if you will test performance it yourself set --rbd_cache=false
Aliaksei Nazarenka
9 Posts
Quote from Aliaksei Nazarenka on June 28, 2017, 5:34 amWhen I try map a disk created by your recommendations, I get this:
[60616.921073] rbd: image lun_3: unsupported stripe unit (got 1048576 want 4194304)
[60670.175467] rbd: image lun_3: unsupported stripe unit (got 65536 want 4194304)
If i try this: rbd create lun_4 --size 1024 --pool rbd --image-format=2 --image-feature striping --stripe-unit 4194304 --stripe-count 1
I get: stripe unit is not a factor of the object size
Therefore, if it is easy to bring please exactly the version of the team that you have in petasan.
When I try map a disk created by your recommendations, I get this:
[60616.921073] rbd: image lun_3: unsupported stripe unit (got 1048576 want 4194304)
[60670.175467] rbd: image lun_3: unsupported stripe unit (got 65536 want 4194304)
If i try this: rbd create lun_4 --size 1024 --pool rbd --image-format=2 --image-feature striping --stripe-unit 4194304 --stripe-count 1
I get: stripe unit is not a factor of the object size
Therefore, if it is easy to bring please exactly the version of the team that you have in petasan.
admin
2,930 Posts
Quote from admin on June 28, 2017, 2:37 pmHi there, what i was trying to say is that you cannot change the stripe policy unless you use user level applications that talks directly to ceph using librbd user space library. It is not supported in the kernel. You cannot map your striped image to a kernel block device since it is using non default stripe size.
Hi there, what i was trying to say is that you cannot change the stripe policy unless you use user level applications that talks directly to ceph using librbd user space library. It is not supported in the kernel. You cannot map your striped image to a kernel block device since it is using non default stripe size.