Forums

Home / Forums

You need to log in to create posts and topics. Login · Register

Not able to delete a stopped disk


Bala
13 Posts
September 3, 2024, 9:23 am

When i try to delete a stopped disk, It went to deleting process, but after few minutes again it came back to stopped state. I am not able to remove/delete the stopped disk.

In logs ,I am getting the below error:

root@mys-dc-gc1-mon01:/opt/petasan/log# cat PetaSAN.log | grep -i "error" | grep -v "ServiceChecker"
02/09/2024 19:49:19 ERROR Delete disk image-00248 error
02/09/2024 19:49:19 ERROR [errno 16] RBD image is busy (error removing image)
rbd.ImageBusy: [errno 16] RBD image is busy (error removing image)
02/09/2024 20:05:14 ERROR Delete disk image-00248 error
02/09/2024 20:05:14 ERROR [errno 16] RBD image is busy (error removing image)
rbd.ImageBusy: [errno 16] RBD image is busy (error removing image)
02/09/2024 20:11:12 ERROR Delete disk image-00248 error
02/09/2024 20:11:12 ERROR [errno 16] RBD image is busy (error removing image)
rbd.ImageBusy: [errno 16] RBD image is busy (error removing image)
02/09/2024 20:15:09 ERROR Delete disk image-00248 error
02/09/2024 20:15:09 ERROR [errno 16] RBD image is busy (error removing image)
rbd.ImageBusy: [errno 16] RBD image is busy (error removing image)
02/09/2024 20:48:38 ERROR Delete disk image-00248 error
02/09/2024 20:48:38 ERROR [errno 16] RBD image is busy (error removing image)

Even-though I stopped the disk in PetaSAN, the image is used by some process,

root@mys-dc-gc1-mon01:~# rbd -p RBD_SSD rm image-00249
2024-09-03T21:25:54.045+0530 7f0bab7fe700 -1 librbd::image::PreRemoveRequest: 0x55cb4cda6370 check_image_watchers: image has watchers - not removing
Removing image: 0% complete...failed.
rbd: error: image still has watchers
This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout.

root@mys-dc-gc1-mon01:~# rbd status image-00249 -p RBD_SSD
Watchers:
watcher=10.25.12.27:0/3040519787 client.195355634 cookie=18446462598732841031

How to solve this issue?Your assistance will be more helpful to solve

Last edited on September 3, 2024, 3:56 pm by Bala · #1

admin
2,930 Posts
September 3, 2024, 5:36 pm

There is a client 10.25.12.27 connected to the image

 

#2

Bala
13 Posts
September 4, 2024, 9:58 am

When i create a disk, Basically It is assigned to two nodes

Active Paths

IP Interface Assigned Node
10.25.16.184 eth4 mys-dc-gc1-osd02(10.25.12.25)
10.25.32.184 eth6 mys-dc-gc1-osd03(10.25.12.26)

But when i grep the status of the images, it is mapped to three nodes.

root@mys-dc-gc1-mon01:~# rbd status image-00249 -p RBD_SSD
Watchers:
watcher=10.25.12.27:0/3040519787 client.195355634 cookie=18446462598732841031
watcher=10.25.12.26:0/674991386 client.195272289 cookie=18446462598732841032
watcher=10.25.12.25:0/201138659 client.195331298 cookie=18446462598732841040

After i stopped the disk in petasan, still the image is used by some other node, which is not assigned

root@mys-dc-gc1-mon01:~# rbd status image-00249 -p RBD_SSD
Watchers:
watcher=10.25.12.27:0/3040519787 client.195355634 cookie=18446462598732841031

When i try to manually unmap the image,  it show the device busy error

root@mys-dc-gc1-osd04:~# rbd unmap RBD_SSD/image-00249
rbd: sysfs write failed
rbd: unmap failed: (16) Device or resource busy

How to remove the watchers or unmap the image?So that once I unmapped the image, I think  able to delete the disk

#3

admin
2,930 Posts
September 4, 2024, 11:09 am

it could be some other process mapping the image outside of iSCSI. can you run

ps aux | grep rbd

and see if this is the case, you can kill it. It could be a process to delete the image that is either taking a long time or stuck. Large images on HDD can take a long time to delete. You can also reboot the node.

what version of PetaSAN are you using ?

 

 

#4