Unable to remove iscsi disk
Pages: 1 2
khopkins
96 Posts
June 1, 2018, 12:58 pmQuote from khopkins on June 1, 2018, 12:58 pmExcellent! it stopped the disk. Now the disk appears in the "disk list" as stopped, tried to detach and it said "detached", but the disk will not delete. Red banner on top of page "Alert, error deleting disk".
Excellent! it stopped the disk. Now the disk appears in the "disk list" as stopped, tried to detach and it said "detached", but the disk will not delete. Red banner on top of page "Alert, error deleting disk".
admin
2,930 Posts
June 1, 2018, 1:56 pmQuote from admin on June 1, 2018, 1:56 pm
delete the image via
rbd rm image-00002 --cluster xx
if this fails check the status via
rbd status image-00002 --cluster xx
delete the image via
rbd rm image-00002 --cluster xx
if this fails check the status via
rbd status image-00002 --cluster xx
khopkins
96 Posts
June 1, 2018, 3:24 pmQuote from khopkins on June 1, 2018, 3:24 pmLooks like the watcher need to stop.
rbd rm image-00002 --cluster XenStorage
2018-06-01 10:20:50.587325 7f4a30ff9700 -1 librbd::image::RemoveRequest: 0x561603b343e0 check_image_watchers: image has watchers - not removing
Removing image: 0% complete...failed.
rbd: error: image still has watchers
This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout.
rbd status image-00002 --cluster XenStorage
Watchers:
watcher=10.0.4.32:0/3341537800 client.79425 cookie=18446462598732840961
Looks like the watcher need to stop.
rbd rm image-00002 --cluster XenStorage
2018-06-01 10:20:50.587325 7f4a30ff9700 -1 librbd::image::RemoveRequest: 0x561603b343e0 check_image_watchers: image has watchers - not removing
Removing image: 0% complete...failed.
rbd: error: image still has watchers
This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout.
rbd status image-00002 --cluster XenStorage
Watchers:
watcher=10.0.4.32:0/3341537800 client.79425 cookie=18446462598732840961
admin
2,930 Posts
June 1, 2018, 3:40 pmQuote from admin on June 1, 2018, 3:40 pmif you can reboot 10.0.4.32 then do so, else i will list you some commands to remove it. wait 15 min before you delete it
if you can reboot 10.0.4.32 then do so, else i will list you some commands to remove it. wait 15 min before you delete it
Last edited on June 1, 2018, 3:45 pm by admin · #14
khopkins
96 Posts
June 1, 2018, 4:14 pmQuote from khopkins on June 1, 2018, 4:14 pmRebooted the server (x.32) and lost connectivity to my Xenservers SR. Lost the path assignments, they are blank. Xen is unable to see the virtual IP to connect to it. Thought it would fail over to the other 2 nodes, that's the purpose of this cluster.
Rebooted the server (x.32) and lost connectivity to my Xenservers SR. Lost the path assignments, they are blank. Xen is unable to see the virtual IP to connect to it. Thought it would fail over to the other 2 nodes, that's the purpose of this cluster.
admin
2,930 Posts
June 1, 2018, 4:43 pmQuote from admin on June 1, 2018, 4:43 pmYes it should failover, How many paths does disk 00001 have ? if you go to the disk list does it show the disk as running ? if you click on the paths link..does it show any assigned nodes ? Before you booted were all paths assigned to the node that rebooted ?
Also can you please check the status of both ceph and consul
ceph status --cluster xx
consul members
Yes it should failover, How many paths does disk 00001 have ? if you go to the disk list does it show the disk as running ? if you click on the paths link..does it show any assigned nodes ? Before you booted were all paths assigned to the node that rebooted ?
Also can you please check the status of both ceph and consul
ceph status --cluster xx
consul members
khopkins
96 Posts
June 1, 2018, 6:11 pmQuote from khopkins on June 1, 2018, 6:11 pmThe "Path Assignment List" is blank, no paths listed. Went to the "Iscsi disk list" and the service was stopped, restarted and path's came back online along with the Xen cluster. But why did it stop? Also the disk can be deleted and cleared. Very much appreciate the insight on how this thinks, very good information. So now, why did the cluster service stop itself during a failover?
Commands given before starting the cluster.
/etc/ceph# ceph status --cluster XenStorage
cluster:
id: b473ff79-febb-48c9-ba18-50b02b6ecb86
health: HEALTH_OK
services:
mon: 3 daemons, quorum PS-Node-1,PS-Node-2,PS-Node-3
mgr: PS-Node-1(active), standbys: PS-Node-2, PS-Node-3
osd: 3 osds: 3 up, 3 in
data:
pools: 1 pools, 256 pgs
objects: 247k objects, 982 GB
usage: 1954 GB used, 9220 GB / 11174 GB avail
pgs: 256 active+clean
consul members
Node Address Status Type Build Protocol DC
PS-Node-1 10.0.4.30:8301 alive server 0.7.3 2 petasan
PS-Node-2 10.0.4.31:8301 alive server 0.7.3 2 petasan
PS-Node-3 10.0.4.32:8301 alive server 0.7.3 2 petasan
The "Path Assignment List" is blank, no paths listed. Went to the "Iscsi disk list" and the service was stopped, restarted and path's came back online along with the Xen cluster. But why did it stop? Also the disk can be deleted and cleared. Very much appreciate the insight on how this thinks, very good information. So now, why did the cluster service stop itself during a failover?
Commands given before starting the cluster.
/etc/ceph# ceph status --cluster XenStorage
cluster:
id: b473ff79-febb-48c9-ba18-50b02b6ecb86
health: HEALTH_OK
services:
mon: 3 daemons, quorum PS-Node-1,PS-Node-2,PS-Node-3
mgr: PS-Node-1(active), standbys: PS-Node-2, PS-Node-3
osd: 3 osds: 3 up, 3 in
data:
pools: 1 pools, 256 pgs
objects: 247k objects, 982 GB
usage: 1954 GB used, 9220 GB / 11174 GB avail
pgs: 256 active+clean
consul members
Node Address Status Type Build Protocol DC
PS-Node-1 10.0.4.30:8301 alive server 0.7.3 2 petasan
PS-Node-2 10.0.4.31:8301 alive server 0.7.3 2 petasan
PS-Node-3 10.0.4.32:8301 alive server 0.7.3 2 petasan
admin
2,930 Posts
June 1, 2018, 8:13 pmQuote from admin on June 1, 2018, 8:13 pmThe disk should not have stopped during failover, something is seriously wrong. it is actually the first time i hear this happened. Do you have enough hardware resources such as ram ? if this is a test/poc environment i would recommend you re-install the cluster, checking your configuration, make sure network connectivity is good. If the problem still occurs then we can go over more detail and look at the logs.
The disk should not have stopped during failover, something is seriously wrong. it is actually the first time i hear this happened. Do you have enough hardware resources such as ram ? if this is a test/poc environment i would recommend you re-install the cluster, checking your configuration, make sure network connectivity is good. If the problem still occurs then we can go over more detail and look at the logs.
khopkins
96 Posts
June 1, 2018, 8:28 pmQuote from khopkins on June 1, 2018, 8:28 pmHello,
Ok, I'll look at it. Also want to thank you for your help and patients. You have provide some excellent commands that will certainly be using in the future.
Hello,
Ok, I'll look at it. Also want to thank you for your help and patients. You have provide some excellent commands that will certainly be using in the future.
Pages: 1 2
Unable to remove iscsi disk
khopkins
96 Posts
Quote from khopkins on June 1, 2018, 12:58 pmExcellent! it stopped the disk. Now the disk appears in the "disk list" as stopped, tried to detach and it said "detached", but the disk will not delete. Red banner on top of page "Alert, error deleting disk".
Excellent! it stopped the disk. Now the disk appears in the "disk list" as stopped, tried to detach and it said "detached", but the disk will not delete. Red banner on top of page "Alert, error deleting disk".
admin
2,930 Posts
Quote from admin on June 1, 2018, 1:56 pm
delete the image via
rbd rm image-00002 --cluster xx
if this fails check the status via
rbd status image-00002 --cluster xx
delete the image via
rbd rm image-00002 --cluster xx
if this fails check the status via
rbd status image-00002 --cluster xx
khopkins
96 Posts
Quote from khopkins on June 1, 2018, 3:24 pmLooks like the watcher need to stop.
rbd rm image-00002 --cluster XenStorage
2018-06-01 10:20:50.587325 7f4a30ff9700 -1 librbd::image::RemoveRequest: 0x561603b343e0 check_image_watchers: image has watchers - not removing
Removing image: 0% complete...failed.
rbd: error: image still has watchers
This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout.
rbd status image-00002 --cluster XenStorage
Watchers:
watcher=10.0.4.32:0/3341537800 client.79425 cookie=18446462598732840961
Looks like the watcher need to stop.
rbd rm image-00002 --cluster XenStorage
2018-06-01 10:20:50.587325 7f4a30ff9700 -1 librbd::image::RemoveRequest: 0x561603b343e0 check_image_watchers: image has watchers - not removing
Removing image: 0% complete...failed.
rbd: error: image still has watchers
This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout.
rbd status image-00002 --cluster XenStorage
Watchers:
watcher=10.0.4.32:0/3341537800 client.79425 cookie=18446462598732840961
admin
2,930 Posts
Quote from admin on June 1, 2018, 3:40 pmif you can reboot 10.0.4.32 then do so, else i will list you some commands to remove it. wait 15 min before you delete it
if you can reboot 10.0.4.32 then do so, else i will list you some commands to remove it. wait 15 min before you delete it
khopkins
96 Posts
Quote from khopkins on June 1, 2018, 4:14 pmRebooted the server (x.32) and lost connectivity to my Xenservers SR. Lost the path assignments, they are blank. Xen is unable to see the virtual IP to connect to it. Thought it would fail over to the other 2 nodes, that's the purpose of this cluster.
Rebooted the server (x.32) and lost connectivity to my Xenservers SR. Lost the path assignments, they are blank. Xen is unable to see the virtual IP to connect to it. Thought it would fail over to the other 2 nodes, that's the purpose of this cluster.
admin
2,930 Posts
Quote from admin on June 1, 2018, 4:43 pmYes it should failover, How many paths does disk 00001 have ? if you go to the disk list does it show the disk as running ? if you click on the paths link..does it show any assigned nodes ? Before you booted were all paths assigned to the node that rebooted ?
Also can you please check the status of both ceph and consul
ceph status --cluster xx
consul members
Yes it should failover, How many paths does disk 00001 have ? if you go to the disk list does it show the disk as running ? if you click on the paths link..does it show any assigned nodes ? Before you booted were all paths assigned to the node that rebooted ?
Also can you please check the status of both ceph and consul
ceph status --cluster xx
consul members
khopkins
96 Posts
Quote from khopkins on June 1, 2018, 6:11 pmThe "Path Assignment List" is blank, no paths listed. Went to the "Iscsi disk list" and the service was stopped, restarted and path's came back online along with the Xen cluster. But why did it stop? Also the disk can be deleted and cleared. Very much appreciate the insight on how this thinks, very good information. So now, why did the cluster service stop itself during a failover?Commands given before starting the cluster./etc/ceph# ceph status --cluster XenStorage
cluster:
id: b473ff79-febb-48c9-ba18-50b02b6ecb86
health: HEALTH_OKservices:
mon: 3 daemons, quorum PS-Node-1,PS-Node-2,PS-Node-3
mgr: PS-Node-1(active), standbys: PS-Node-2, PS-Node-3
osd: 3 osds: 3 up, 3 indata:
pools: 1 pools, 256 pgs
objects: 247k objects, 982 GB
usage: 1954 GB used, 9220 GB / 11174 GB avail
pgs: 256 active+cleanconsul members
Node Address Status Type Build Protocol DC
PS-Node-1 10.0.4.30:8301 alive server 0.7.3 2 petasan
PS-Node-2 10.0.4.31:8301 alive server 0.7.3 2 petasan
PS-Node-3 10.0.4.32:8301 alive server 0.7.3 2 petasan
cluster:
id: b473ff79-febb-48c9-ba18-50b02b6ecb86
health: HEALTH_OK
mon: 3 daemons, quorum PS-Node-1,PS-Node-2,PS-Node-3
mgr: PS-Node-1(active), standbys: PS-Node-2, PS-Node-3
osd: 3 osds: 3 up, 3 in
pools: 1 pools, 256 pgs
objects: 247k objects, 982 GB
usage: 1954 GB used, 9220 GB / 11174 GB avail
pgs: 256 active+clean
Node Address Status Type Build Protocol DC
PS-Node-1 10.0.4.30:8301 alive server 0.7.3 2 petasan
PS-Node-2 10.0.4.31:8301 alive server 0.7.3 2 petasan
PS-Node-3 10.0.4.32:8301 alive server 0.7.3 2 petasan
admin
2,930 Posts
Quote from admin on June 1, 2018, 8:13 pmThe disk should not have stopped during failover, something is seriously wrong. it is actually the first time i hear this happened. Do you have enough hardware resources such as ram ? if this is a test/poc environment i would recommend you re-install the cluster, checking your configuration, make sure network connectivity is good. If the problem still occurs then we can go over more detail and look at the logs.
The disk should not have stopped during failover, something is seriously wrong. it is actually the first time i hear this happened. Do you have enough hardware resources such as ram ? if this is a test/poc environment i would recommend you re-install the cluster, checking your configuration, make sure network connectivity is good. If the problem still occurs then we can go over more detail and look at the logs.
khopkins
96 Posts
Quote from khopkins on June 1, 2018, 8:28 pmHello,
Ok, I'll look at it. Also want to thank you for your help and patients. You have provide some excellent commands that will certainly be using in the future.
Hello,
Ok, I'll look at it. Also want to thank you for your help and patients. You have provide some excellent commands that will certainly be using in the future.