msalem
87 Posts
Quote from msalem on June 17, 2018, 2:24 am
Hello Admin.
I have setup a cluster of 4 nodes, I have added an ISCSI with 50TB, I realized I used the wrong IP's so I tried to remove the ISCSI, after 15 mins. the Disk was no longer there.
Now when I try to add any new disks I get this message.
https://ibb.co/iOav0J
Thanks
Hello Admin.
I have setup a cluster of 4 nodes, I have added an ISCSI with 50TB, I realized I used the wrong IP's so I tried to remove the ISCSI, after 15 mins. the Disk was no longer there.
Now when I try to add any new disks I get this message.
Thanks
admin
2,930 Posts
Quote from admin on June 17, 2018, 10:06 am
This happens if for some reason the removed disk was not cleaned from consul system, you can manually list disks and remove disks for example disk 00001 :
consul kv get -recurse PetaSAN/Disks
consul kv delete -recurse PetaSAN/Disks/00001
Once fixed, if you can try to reproduce this it will help us. The 15 min deletion time is not a problem and could happen for very large disks (tens of TB) or for cases when a node with connections goes down uncleanly, Ceph will keep a connection watch open for that long but in both cases this should not lead to this issue.
This happens if for some reason the removed disk was not cleaned from consul system, you can manually list disks and remove disks for example disk 00001 :
consul kv get -recurse PetaSAN/Disks
consul kv delete -recurse PetaSAN/Disks/00001
Once fixed, if you can try to reproduce this it will help us. The 15 min deletion time is not a problem and could happen for very large disks (tens of TB) or for cases when a node with connections goes down uncleanly, Ceph will keep a connection watch open for that long but in both cases this should not lead to this issue.
msalem
87 Posts
Quote from msalem on June 18, 2018, 12:55 am
Hey Admin,
Yes that did the trick with me, I created a 50TB disk, and tried to delete it. I think as you said it takes time for it to be deleted.
Hey Admin,
Yes that did the trick with me, I created a 50TB disk, and tried to delete it. I think as you said it takes time for it to be deleted.
msalem
87 Posts
Quote from msalem on June 19, 2018, 12:17 am
Hey Admin,
After I did the commands you have recommended, I was able to create new LUN's however they seems to be stuck in "Starting" I left it for more than 24 hours.
Can you please tell me the best way to troubleshoot this issues and what would cause this to happen.
https://ibb.co/irs6Sy
This is what I have now in the CLI:
root@srocceph2:~# consul kv get -recurse PetaSAN/Disks
PetaSAN/Disks/00001:disk
PetaSAN/Disks/00001/1:
PetaSAN/Disks/00001/2:
PetaSAN/Disks/00002:disk
PetaSAN/Disks/00002/1:
PetaSAN/Disks/00002/2:
root@srocceph2:~#
Thanks
Hey Admin,
After I did the commands you have recommended, I was able to create new LUN's however they seems to be stuck in "Starting" I left it for more than 24 hours.
Can you please tell me the best way to troubleshoot this issues and what would cause this to happen.
This is what I have now in the CLI:
root@srocceph2:~# consul kv get -recurse PetaSAN/Disks
PetaSAN/Disks/00001:disk
PetaSAN/Disks/00001/1:
PetaSAN/Disks/00001/2:
PetaSAN/Disks/00002:disk
PetaSAN/Disks/00002/1:
PetaSAN/Disks/00002/2:
root@srocceph2:~#
Thanks
admin
2,930 Posts
Quote from admin on June 19, 2018, 11:24 am
Before closing the initial "error adding disk" issue which got fixed via the cli commands, we meed to know if you can reproduce it and if so what steps so we can try it.
The current issue: this indicates the disks were assigned to the nodes correctly by consul system, but the nodes are not able to configure the iSCSI targets: most likely it is an error in mapping the ceph rbd image or a lio setup issue: can you do a simple rbd list check on the nodes
rbd ls --cluster xx
If this is ok: can you send me the PetaSAN.log file at contact-us @ petasan.org
Have you done any changes to the system or added / upgraded any software ?
Before closing the initial "error adding disk" issue which got fixed via the cli commands, we meed to know if you can reproduce it and if so what steps so we can try it.
The current issue: this indicates the disks were assigned to the nodes correctly by consul system, but the nodes are not able to configure the iSCSI targets: most likely it is an error in mapping the ceph rbd image or a lio setup issue: can you do a simple rbd list check on the nodes
rbd ls --cluster xx
If this is ok: can you send me the PetaSAN.log file at contact-us @ petasan.org
Have you done any changes to the system or added / upgraded any software ?
msalem
87 Posts
Quote from msalem on June 19, 2018, 2:08 pm
Hello Admin.
Thanks for the reply.
I have not changed anything, these systems are all default installation, and has no internet access. I just send you the logs.
root@srocceph1:~# rbd ls --cluster ROC
image-00001
image-00002
image-00003
Hello Admin.
Thanks for the reply.
I have not changed anything, these systems are all default installation, and has no internet access. I just send you the logs.
root@srocceph1:~# rbd ls --cluster ROC
image-00001
image-00002
image-00003
msalem
87 Posts
Quote from msalem on June 19, 2018, 2:13 pm
Quote from admin on June 19, 2018, 11:24 am
Before closing the initial "error adding disk" issue which got fixed via the cli commands, we meed to know if you can reproduce it and if so what steps so we can try it.
The current issue: this indicates the disks were assigned to the nodes correctly by consul system, but the nodes are not able to configure the iSCSI targets: most likely it is an error in mapping the ceph rbd image or a lio setup issue: can you do a simple rbd list check on the nodes
rbd ls --cluster xx
If this is ok: can you send me the PetaSAN.log file at contact-us @ petasan.org
Have you done any changes to the system or added / upgraded any software ?
To reproduce the issue, I did not do much. Its a fresh install I have created a 50TB ISCSI disk. then I realized that the ISCSI IP's were off. So I tried to stop it, here are the steps, as I can remember.
1 - Created 50TB Disk. = OK, it took like 1 min
2- Stopped the disk = OK, it took like 3 mins.
3 - Delete the DISK, this here were it got stuck.
4 - went to ISCSI settings and changed my IP settings.
5 - I was able to delete the Disk after changing the settings . tried to create a new Disk I got first error that the Disk already exists.
hope this help
Quote from admin on June 19, 2018, 11:24 am
Before closing the initial "error adding disk" issue which got fixed via the cli commands, we meed to know if you can reproduce it and if so what steps so we can try it.
The current issue: this indicates the disks were assigned to the nodes correctly by consul system, but the nodes are not able to configure the iSCSI targets: most likely it is an error in mapping the ceph rbd image or a lio setup issue: can you do a simple rbd list check on the nodes
rbd ls --cluster xx
If this is ok: can you send me the PetaSAN.log file at contact-us @ petasan.org
Have you done any changes to the system or added / upgraded any software ?
To reproduce the issue, I did not do much. Its a fresh install I have created a 50TB ISCSI disk. then I realized that the ISCSI IP's were off. So I tried to stop it, here are the steps, as I can remember.
1 - Created 50TB Disk. = OK, it took like 1 min
2- Stopped the disk = OK, it took like 3 mins.
3 - Delete the DISK, this here were it got stuck.
4 - went to ISCSI settings and changed my IP settings.
5 - I was able to delete the Disk after changing the settings . tried to create a new Disk I got first error that the Disk already exists.
hope this help
admin
2,930 Posts
Quote from admin on June 19, 2018, 2:36 pm
I looked at the logs, it is full of errors from the start of cluster creation. since this is a fresh cluster i would recommend you re-install. If the problem persists, we can go more deeper.
I looked at the logs, it is full of errors from the start of cluster creation. since this is a fresh cluster i would recommend you re-install. If the problem persists, we can go more deeper.
msalem
87 Posts
Quote from msalem on June 19, 2018, 3:07 pm
I think that is because previous installations, I noticed that when you re-install the OS it does not over-write or format the drives, I ad some files and they were still there.
I will try to format the disk and do it again.
I will keep you posted.
I think that is because previous installations, I noticed that when you re-install the OS it does not over-write or format the drives, I ad some files and they were still there.
I will try to format the disk and do it again.
I will keep you posted.
admin
2,930 Posts
Quote from admin on June 19, 2018, 3:36 pm
That could be the issue 🙂
Actually it does format the disks, so if this did not happen it is an issue.
That could be the issue 🙂
Actually it does format the disks, so if this did not happen it is an issue.
