how to remove big ISCSI disk
Pages: 1 2
elwan
14 Posts
June 26, 2019, 11:24 amQuote from elwan on June 26, 2019, 11:24 amHello ,
I have create some iscsi disk with petasan but when i tried to remove it with web interface it take some many time.So i do some research with how to delete big image with ceph.Solution i found is there and i am asking if this way is a best practice.
rados -p rbd rm rbd_id.image-00002 --cluster ClusterBKO
rados -p rbd rm rbd_header.81d12ae8944a --cluster ClusterBKO
rados --cluster ClusterBKO -p rbd ls | grep '^rbd_data.247d2a6b8b4567.' | xargs -n 200 rados --cluster ClusterBKO -p rbd rm
rbd rm rbdbigsize --cluster ClusterBKO
Hello ,
I have create some iscsi disk with petasan but when i tried to remove it with web interface it take some many time.So i do some research with how to delete big image with ceph.Solution i found is there and i am asking if this way is a best practice.
rados -p rbd rm rbd_id.image-00002 --cluster ClusterBKO
rados -p rbd rm rbd_header.81d12ae8944a --cluster ClusterBKO
rados --cluster ClusterBKO -p rbd ls | grep '^rbd_data.247d2a6b8b4567.' | xargs -n 200 rados --cluster ClusterBKO -p rbd rm
rbd rm rbdbigsize --cluster ClusterBKO
admin
2,930 Posts
June 26, 2019, 11:51 amQuote from admin on June 26, 2019, 11:51 amBig images (tens of TBs) will take some time, maybe 10 sec per TB for a small slow cluster. It deletes at a rate of read_iops x 4 MB per sec so a 25K iops cluster will do 10 sec per TB.
The commands you ran are not recommended, they iterate all objects found in the rados database (rocksdb) to match the particular image you want, this will work well if you have a small number of images but if you have a larger number of images with filled data, you will be iterating on a huge number of objects, the performance will be terrible.
There is work to improve the delete speed for large images by supporting rbd object map in the kernel.
Big images (tens of TBs) will take some time, maybe 10 sec per TB for a small slow cluster. It deletes at a rate of read_iops x 4 MB per sec so a 25K iops cluster will do 10 sec per TB.
The commands you ran are not recommended, they iterate all objects found in the rados database (rocksdb) to match the particular image you want, this will work well if you have a small number of images but if you have a larger number of images with filled data, you will be iterating on a huge number of objects, the performance will be terrible.
There is work to improve the delete speed for large images by supporting rbd object map in the kernel.
elwan
14 Posts
June 27, 2019, 11:50 amQuote from elwan on June 27, 2019, 11:50 amThanks for your reply obviously your right about my previous method to remove big image iscsi.After command was finish i got this error about another disk that was not deleted it was at Detached Status and when i tried to attach it again it i got this alert "This disk has not been created by PetaSAN. The Ceph rbd image will be renamed to match the PetaSAN naming standard".
Another error i have is "Error disk attached but could not read metadata" when i try to save attach process.
Thanks
Thanks for your reply obviously your right about my previous method to remove big image iscsi.After command was finish i got this error about another disk that was not deleted it was at Detached Status and when i tried to attach it again it i got this alert "This disk has not been created by PetaSAN. The Ceph rbd image will be renamed to match the PetaSAN naming standard".
Another error i have is "Error disk attached but could not read metadata" when i try to save attach process.
Thanks
admin
2,930 Posts
June 27, 2019, 12:32 pmQuote from admin on June 27, 2019, 12:32 pmIf you try a new image you create manually then attach it to PetaSAN, do you get an error ?
Can you do an rbd info on the disk that gives error attaching.
If you try a new image you create manually then attach it to PetaSAN, do you get an error ?
Can you do an rbd info on the disk that gives error attaching.
elwan
14 Posts
June 27, 2019, 1:18 pmQuote from elwan on June 27, 2019, 1:18 pmi create new image and i dont have this errros.
rbd info of the disk
root@NODEBKO-01:~# rbd info image-00006 --cluster ClusterBKO
rbd image 'image-00006':
size 30720 GB in 7864320 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.1bf3496b8b4567
format: 2
features: layering
flags:
create_timestamp: Wed Apr 17 14:22:15 2019
i create new image and i dont have this errros.
rbd info of the disk
root@NODEBKO-01:~# rbd info image-00006 --cluster ClusterBKO
rbd image 'image-00006':
size 30720 GB in 7864320 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.1bf3496b8b4567
format: 2
features: layering
flags:
create_timestamp: Wed Apr 17 14:22:15 2019
admin
2,930 Posts
June 27, 2019, 2:45 pmQuote from admin on June 27, 2019, 2:45 pmCould you give more detail how to reproduce this. It is not clear why was the disk detached, did you created it manually or was this a result of the failed delete step ? Or was it a PetaSAN created disk and you detached it ? image-00006 is the disk failing to be attached ? anything special or manual you did on this disk will help.
Could you give more detail how to reproduce this. It is not clear why was the disk detached, did you created it manually or was this a result of the failed delete step ? Or was it a PetaSAN created disk and you detached it ? image-00006 is the disk failing to be attached ? anything special or manual you did on this disk will help.
Last edited on June 27, 2019, 2:47 pm by admin · #6
elwan
14 Posts
June 27, 2019, 4:11 pmQuote from elwan on June 27, 2019, 4:11 pmThis error occur when i delete image-00009 with procedure discribe on my fist ticket. image-00006 is the disk failling to be attached it was create through petasan (webinterface). I was automatically detached when delete step was finish.
This error occur when i delete image-00009 with procedure discribe on my fist ticket. image-00006 is the disk failling to be attached it was create through petasan (webinterface). I was automatically detached when delete step was finish.
admin
2,930 Posts
June 27, 2019, 6:35 pmQuote from admin on June 27, 2019, 6:35 pmif this occurred after the delete steps you posted, maybe the object header for image-00006 was accidentally deleted. we use the header object to attach iscsi metadata, if the header gets deleted the image will be switched from attached to detached and cannot be re-attched.
if this occurred after the delete steps you posted, maybe the object header for image-00006 was accidentally deleted. we use the header object to attach iscsi metadata, if the header gets deleted the image will be switched from attached to detached and cannot be re-attched.
Last edited on June 27, 2019, 6:38 pm by admin · #8
elwan
14 Posts
June 28, 2019, 11:03 pmQuote from elwan on June 28, 2019, 11:03 pmhow can i do to recreate hearder for image-00006 please ?
how can i do to recreate hearder for image-00006 please ?
elwan
14 Posts
June 29, 2019, 12:19 pmQuote from elwan on June 29, 2019, 12:19 pmi do a grep commande and i see the header was here details in below:
root@NODEBKO-03:~# rados --cluster ClusterCCTVBKO -p rbd ls | grep -v '^rbd_data.1bf3496b8b4567.'
rbd_header.447c7c6b8b4567
rbd_directory
rbd_info
rbd_id.image-00006
rbd_header.447c7c6b8b456
rbd_header.1bf3496b8b4567
rbd_id.image-00011
root@NODEBKO-03:~# rbd info image-00006 --cluster ClusterCCTVBKO
rbd image 'image-00006':
size 30720 GB in 7864320 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.1bf3496b8b4567
format: 2
features: layering
flags:
create_timestamp: Wed Apr 17 14:22:15 2019
i do a grep commande and i see the header was here details in below:
root@NODEBKO-03:~# rados --cluster ClusterCCTVBKO -p rbd ls | grep -v '^rbd_data.1bf3496b8b4567.'
rbd_header.447c7c6b8b4567
rbd_directory
rbd_info
rbd_id.image-00006
rbd_header.447c7c6b8b456
rbd_header.1bf3496b8b4567
rbd_id.image-00011
root@NODEBKO-03:~# rbd info image-00006 --cluster ClusterCCTVBKO
rbd image 'image-00006':
size 30720 GB in 7864320 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.1bf3496b8b4567
format: 2
features: layering
flags:
create_timestamp: Wed Apr 17 14:22:15 2019
Pages: 1 2
how to remove big ISCSI disk
elwan
14 Posts
Quote from elwan on June 26, 2019, 11:24 amHello ,
I have create some iscsi disk with petasan but when i tried to remove it with web interface it take some many time.So i do some research with how to delete big image with ceph.Solution i found is there and i am asking if this way is a best practice.
rados -p rbd rm rbd_id.image-00002 --cluster ClusterBKOrados -p rbd rm rbd_header.81d12ae8944a --cluster ClusterBKOrados --cluster ClusterBKO -p rbd ls | grep '^rbd_data.247d2a6b8b4567.' | xargs -n 200 rados --cluster ClusterBKO -p rbd rmrbd rm rbdbigsize --cluster ClusterBKO
Hello ,
I have create some iscsi disk with petasan but when i tried to remove it with web interface it take some many time.So i do some research with how to delete big image with ceph.Solution i found is there and i am asking if this way is a best practice.
rados -p rbd rm rbd_id.image-00002 --cluster ClusterBKOrados -p rbd rm rbd_header.81d12ae8944a --cluster ClusterBKOrados --cluster ClusterBKO -p rbd ls | grep '^rbd_data.247d2a6b8b4567.' | xargs -n 200 rados --cluster ClusterBKO -p rbd rmrbd rm rbdbigsize --cluster ClusterBKO
admin
2,930 Posts
Quote from admin on June 26, 2019, 11:51 amBig images (tens of TBs) will take some time, maybe 10 sec per TB for a small slow cluster. It deletes at a rate of read_iops x 4 MB per sec so a 25K iops cluster will do 10 sec per TB.
The commands you ran are not recommended, they iterate all objects found in the rados database (rocksdb) to match the particular image you want, this will work well if you have a small number of images but if you have a larger number of images with filled data, you will be iterating on a huge number of objects, the performance will be terrible.
There is work to improve the delete speed for large images by supporting rbd object map in the kernel.
Big images (tens of TBs) will take some time, maybe 10 sec per TB for a small slow cluster. It deletes at a rate of read_iops x 4 MB per sec so a 25K iops cluster will do 10 sec per TB.
The commands you ran are not recommended, they iterate all objects found in the rados database (rocksdb) to match the particular image you want, this will work well if you have a small number of images but if you have a larger number of images with filled data, you will be iterating on a huge number of objects, the performance will be terrible.
There is work to improve the delete speed for large images by supporting rbd object map in the kernel.
elwan
14 Posts
Quote from elwan on June 27, 2019, 11:50 amThanks for your reply obviously your right about my previous method to remove big image iscsi.After command was finish i got this error about another disk that was not deleted it was at Detached Status and when i tried to attach it again it i got this alert "This disk has not been created by PetaSAN. The Ceph rbd image will be renamed to match the PetaSAN naming standard".
Another error i have is "Error disk attached but could not read metadata" when i try to save attach process.
Thanks
Thanks for your reply obviously your right about my previous method to remove big image iscsi.After command was finish i got this error about another disk that was not deleted it was at Detached Status and when i tried to attach it again it i got this alert "This disk has not been created by PetaSAN. The Ceph rbd image will be renamed to match the PetaSAN naming standard".
Another error i have is "Error disk attached but could not read metadata" when i try to save attach process.
Thanks
admin
2,930 Posts
Quote from admin on June 27, 2019, 12:32 pmIf you try a new image you create manually then attach it to PetaSAN, do you get an error ?
Can you do an rbd info on the disk that gives error attaching.
If you try a new image you create manually then attach it to PetaSAN, do you get an error ?
Can you do an rbd info on the disk that gives error attaching.
elwan
14 Posts
Quote from elwan on June 27, 2019, 1:18 pmi create new image and i dont have this errros.
rbd info of the disk
root@NODEBKO-01:~# rbd info image-00006 --cluster ClusterBKO
rbd image 'image-00006':
size 30720 GB in 7864320 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.1bf3496b8b4567
format: 2
features: layering
flags:
create_timestamp: Wed Apr 17 14:22:15 2019
i create new image and i dont have this errros.
rbd info of the disk
root@NODEBKO-01:~# rbd info image-00006 --cluster ClusterBKO
rbd image 'image-00006':
size 30720 GB in 7864320 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.1bf3496b8b4567
format: 2
features: layering
flags:
create_timestamp: Wed Apr 17 14:22:15 2019
admin
2,930 Posts
Quote from admin on June 27, 2019, 2:45 pmCould you give more detail how to reproduce this. It is not clear why was the disk detached, did you created it manually or was this a result of the failed delete step ? Or was it a PetaSAN created disk and you detached it ? image-00006 is the disk failing to be attached ? anything special or manual you did on this disk will help.
Could you give more detail how to reproduce this. It is not clear why was the disk detached, did you created it manually or was this a result of the failed delete step ? Or was it a PetaSAN created disk and you detached it ? image-00006 is the disk failing to be attached ? anything special or manual you did on this disk will help.
elwan
14 Posts
Quote from elwan on June 27, 2019, 4:11 pmThis error occur when i delete image-00009 with procedure discribe on my fist ticket. image-00006 is the disk failling to be attached it was create through petasan (webinterface). I was automatically detached when delete step was finish.
This error occur when i delete image-00009 with procedure discribe on my fist ticket. image-00006 is the disk failling to be attached it was create through petasan (webinterface). I was automatically detached when delete step was finish.
admin
2,930 Posts
Quote from admin on June 27, 2019, 6:35 pmif this occurred after the delete steps you posted, maybe the object header for image-00006 was accidentally deleted. we use the header object to attach iscsi metadata, if the header gets deleted the image will be switched from attached to detached and cannot be re-attched.
if this occurred after the delete steps you posted, maybe the object header for image-00006 was accidentally deleted. we use the header object to attach iscsi metadata, if the header gets deleted the image will be switched from attached to detached and cannot be re-attched.
elwan
14 Posts
Quote from elwan on June 28, 2019, 11:03 pmhow can i do to recreate hearder for image-00006 please ?
how can i do to recreate hearder for image-00006 please ?
elwan
14 Posts
Quote from elwan on June 29, 2019, 12:19 pmi do a grep commande and i see the header was here details in below:
root@NODEBKO-03:~# rados --cluster ClusterCCTVBKO -p rbd ls | grep -v '^rbd_data.1bf3496b8b4567.'
rbd_header.447c7c6b8b4567
rbd_directory
rbd_info
rbd_id.image-00006
rbd_header.447c7c6b8b456
rbd_header.1bf3496b8b4567
rbd_id.image-00011
root@NODEBKO-03:~# rbd info image-00006 --cluster ClusterCCTVBKOrbd image 'image-00006':
size 30720 GB in 7864320 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.1bf3496b8b4567
format: 2
features: layering
flags:
create_timestamp: Wed Apr 17 14:22:15 2019
i do a grep commande and i see the header was here details in below:
root@NODEBKO-03:~# rados --cluster ClusterCCTVBKO -p rbd ls | grep -v '^rbd_data.1bf3496b8b4567.'
rbd_header.447c7c6b8b4567
rbd_directory
rbd_info
rbd_id.image-00006
rbd_header.447c7c6b8b456
rbd_header.1bf3496b8b4567
rbd_id.image-00011
root@NODEBKO-03:~# rbd info image-00006 --cluster ClusterCCTVBKOrbd image 'image-00006':
size 30720 GB in 7864320 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.1bf3496b8b4567
format: 2
features: layering
flags:
create_timestamp: Wed Apr 17 14:22:15 2019