Operations in the web GUI timeout or are very slow
dbutti
33 Posts
March 27, 2025, 2:37 pmQuote from dbutti on March 27, 2025, 2:37 pmGood afternoon; we are running a Petasan cluster with 4 nodes, 2x10Gbps network interfaces each, 20 OSDs and ~120TB total raw capacity.
The cluster runs smoothly and it is used to support iSCSI disks for virtual machines, NFS/CIFS filesystems and S3 applications with good performance.
However, operations in the web GUI are painfully slow, and they sometimes time out. For example, opening the "iSCSI disks" page takes minutes, and reassigning the iSCSI paths automatically is impossible, because the operation never completes. It also takes ages to simply display the page with the current iSCSI path assignments.
We only have 10 iSCSI disks defined, however we have also created 4-5 additional Ceph pools, and created some RBD images there, which are directly consumed by RBD clients. Perhaps these additional images are slowing down the operations somewhat, but anyway I'm not expecting it to be a critical situation if a cluster contains around 100 RBD images, should I?
Is there a way I can restrict the GUI to operate only on the "plain" rbd pool? And, is there way I can reassign iSCSI paths and manage iSCSI disks on the CLI, without relying on the web GUI?
Thank-you very much in advance,
Good afternoon; we are running a Petasan cluster with 4 nodes, 2x10Gbps network interfaces each, 20 OSDs and ~120TB total raw capacity.
The cluster runs smoothly and it is used to support iSCSI disks for virtual machines, NFS/CIFS filesystems and S3 applications with good performance.
However, operations in the web GUI are painfully slow, and they sometimes time out. For example, opening the "iSCSI disks" page takes minutes, and reassigning the iSCSI paths automatically is impossible, because the operation never completes. It also takes ages to simply display the page with the current iSCSI path assignments.
We only have 10 iSCSI disks defined, however we have also created 4-5 additional Ceph pools, and created some RBD images there, which are directly consumed by RBD clients. Perhaps these additional images are slowing down the operations somewhat, but anyway I'm not expecting it to be a critical situation if a cluster contains around 100 RBD images, should I?
Is there a way I can restrict the GUI to operate only on the "plain" rbd pool? And, is there way I can reassign iSCSI paths and manage iSCSI disks on the CLI, without relying on the web GUI?
Thank-you very much in advance,
admin
2,957 Posts
March 28, 2025, 5:21 pmQuote from admin on March 28, 2025, 5:21 pmWe do test with 500 images, so it should work.
Do you see any errors in PetaSAN.log ?
Are your subnets set up correctly with no overlap in ip addresses between them.
Are you using pure HDD without SSD journals ?
Do you see high % util/busy for cpu or disk in the Node Statistics charts
Do you think the UI is slow just or iSCSI? is it fast when you open other pages like NFS/S3/Configuration menus ?
We do test with 500 images, so it should work.
Do you see any errors in PetaSAN.log ?
Are your subnets set up correctly with no overlap in ip addresses between them.
Are you using pure HDD without SSD journals ?
Do you see high % util/busy for cpu or disk in the Node Statistics charts
Do you think the UI is slow just or iSCSI? is it fast when you open other pages like NFS/S3/Configuration menus ?
dbutti
33 Posts
March 28, 2025, 7:40 pmQuote from dbutti on March 28, 2025, 7:40 pmWell the GUI overall is never very „snappy“, but yes, I‘d say that it‘s just the iSCSI section which is impossibly slow; NFS, S3 and other menus are acceptable.
Every HDD OSD uses a fast SSD for both cache and journal. I have 2 separate subnets for iSCSI and 2 separate subnets for backend connections, all on different VLANs without routing.
The CPU node statistics show an average utilization between 35% and 40%, and quite similarly for the disks.
The PetaSAN.log does not show anything interesting either.
I would really appreciate it anyway if there were some commands to manipulate these important settings via the CLI, because this would make things faster and safer under many circumstances.
Thank-you in advance,
Well the GUI overall is never very „snappy“, but yes, I‘d say that it‘s just the iSCSI section which is impossibly slow; NFS, S3 and other menus are acceptable.
Every HDD OSD uses a fast SSD for both cache and journal. I have 2 separate subnets for iSCSI and 2 separate subnets for backend connections, all on different VLANs without routing.
The CPU node statistics show an average utilization between 35% and 40%, and quite similarly for the disks.
The PetaSAN.log does not show anything interesting either.
I would really appreciate it anyway if there were some commands to manipulate these important settings via the CLI, because this would make things faster and safer under many circumstances.
Thank-you in advance,
admin
2,957 Posts
March 30, 2025, 7:24 pmQuote from admin on March 30, 2025, 7:24 pmHow many images do you have (including non-iSCSI images ) ?
We will look into this. cannot commit to anytime soon. Again we do test with 500 drives and the speed is good.
Currently we store the iSCSI info as metadata per image, each image stores its own iSCSI info. From a design point of view this looks clean and robust, but it does incur a performance hit, as we have to read the metadata of each image to refresh the display. It is possible we store the metadata of all disks in a single external db location/object, so we need to perform a single read to get the data, the drawback is that it is not as robust as the image depends on external entity for its iSCSI data. Reading metadata from each rbd image should not be slow when using SSD journal (rocksdb on SSD), but maybe it is slow under load in some setups. We will evaluate if storing iSCSI data outside of image metadata would be a better approach.
How many images do you have (including non-iSCSI images ) ?
We will look into this. cannot commit to anytime soon. Again we do test with 500 drives and the speed is good.
Currently we store the iSCSI info as metadata per image, each image stores its own iSCSI info. From a design point of view this looks clean and robust, but it does incur a performance hit, as we have to read the metadata of each image to refresh the display. It is possible we store the metadata of all disks in a single external db location/object, so we need to perform a single read to get the data, the drawback is that it is not as robust as the image depends on external entity for its iSCSI data. Reading metadata from each rbd image should not be slow when using SSD journal (rocksdb on SSD), but maybe it is slow under load in some setups. We will evaluate if storing iSCSI data outside of image metadata would be a better approach.
Operations in the web GUI timeout or are very slow
dbutti
33 Posts
Quote from dbutti on March 27, 2025, 2:37 pmGood afternoon; we are running a Petasan cluster with 4 nodes, 2x10Gbps network interfaces each, 20 OSDs and ~120TB total raw capacity.
The cluster runs smoothly and it is used to support iSCSI disks for virtual machines, NFS/CIFS filesystems and S3 applications with good performance.
However, operations in the web GUI are painfully slow, and they sometimes time out. For example, opening the "iSCSI disks" page takes minutes, and reassigning the iSCSI paths automatically is impossible, because the operation never completes. It also takes ages to simply display the page with the current iSCSI path assignments.
We only have 10 iSCSI disks defined, however we have also created 4-5 additional Ceph pools, and created some RBD images there, which are directly consumed by RBD clients. Perhaps these additional images are slowing down the operations somewhat, but anyway I'm not expecting it to be a critical situation if a cluster contains around 100 RBD images, should I?
Is there a way I can restrict the GUI to operate only on the "plain" rbd pool? And, is there way I can reassign iSCSI paths and manage iSCSI disks on the CLI, without relying on the web GUI?
Thank-you very much in advance,
Good afternoon; we are running a Petasan cluster with 4 nodes, 2x10Gbps network interfaces each, 20 OSDs and ~120TB total raw capacity.
The cluster runs smoothly and it is used to support iSCSI disks for virtual machines, NFS/CIFS filesystems and S3 applications with good performance.
However, operations in the web GUI are painfully slow, and they sometimes time out. For example, opening the "iSCSI disks" page takes minutes, and reassigning the iSCSI paths automatically is impossible, because the operation never completes. It also takes ages to simply display the page with the current iSCSI path assignments.
We only have 10 iSCSI disks defined, however we have also created 4-5 additional Ceph pools, and created some RBD images there, which are directly consumed by RBD clients. Perhaps these additional images are slowing down the operations somewhat, but anyway I'm not expecting it to be a critical situation if a cluster contains around 100 RBD images, should I?
Is there a way I can restrict the GUI to operate only on the "plain" rbd pool? And, is there way I can reassign iSCSI paths and manage iSCSI disks on the CLI, without relying on the web GUI?
Thank-you very much in advance,
admin
2,957 Posts
Quote from admin on March 28, 2025, 5:21 pmWe do test with 500 images, so it should work.
Do you see any errors in PetaSAN.log ?
Are your subnets set up correctly with no overlap in ip addresses between them.
Are you using pure HDD without SSD journals ?
Do you see high % util/busy for cpu or disk in the Node Statistics charts
Do you think the UI is slow just or iSCSI? is it fast when you open other pages like NFS/S3/Configuration menus ?
We do test with 500 images, so it should work.
Do you see any errors in PetaSAN.log ?
Are your subnets set up correctly with no overlap in ip addresses between them.
Are you using pure HDD without SSD journals ?
Do you see high % util/busy for cpu or disk in the Node Statistics charts
Do you think the UI is slow just or iSCSI? is it fast when you open other pages like NFS/S3/Configuration menus ?
dbutti
33 Posts
Quote from dbutti on March 28, 2025, 7:40 pmWell the GUI overall is never very „snappy“, but yes, I‘d say that it‘s just the iSCSI section which is impossibly slow; NFS, S3 and other menus are acceptable.
Every HDD OSD uses a fast SSD for both cache and journal. I have 2 separate subnets for iSCSI and 2 separate subnets for backend connections, all on different VLANs without routing.
The CPU node statistics show an average utilization between 35% and 40%, and quite similarly for the disks.
The PetaSAN.log does not show anything interesting either.
I would really appreciate it anyway if there were some commands to manipulate these important settings via the CLI, because this would make things faster and safer under many circumstances.
Thank-you in advance,
Well the GUI overall is never very „snappy“, but yes, I‘d say that it‘s just the iSCSI section which is impossibly slow; NFS, S3 and other menus are acceptable.
Every HDD OSD uses a fast SSD for both cache and journal. I have 2 separate subnets for iSCSI and 2 separate subnets for backend connections, all on different VLANs without routing.
The CPU node statistics show an average utilization between 35% and 40%, and quite similarly for the disks.
The PetaSAN.log does not show anything interesting either.
I would really appreciate it anyway if there were some commands to manipulate these important settings via the CLI, because this would make things faster and safer under many circumstances.
Thank-you in advance,
admin
2,957 Posts
Quote from admin on March 30, 2025, 7:24 pmHow many images do you have (including non-iSCSI images ) ?
We will look into this. cannot commit to anytime soon. Again we do test with 500 drives and the speed is good.
Currently we store the iSCSI info as metadata per image, each image stores its own iSCSI info. From a design point of view this looks clean and robust, but it does incur a performance hit, as we have to read the metadata of each image to refresh the display. It is possible we store the metadata of all disks in a single external db location/object, so we need to perform a single read to get the data, the drawback is that it is not as robust as the image depends on external entity for its iSCSI data. Reading metadata from each rbd image should not be slow when using SSD journal (rocksdb on SSD), but maybe it is slow under load in some setups. We will evaluate if storing iSCSI data outside of image metadata would be a better approach.
How many images do you have (including non-iSCSI images ) ?
We will look into this. cannot commit to anytime soon. Again we do test with 500 drives and the speed is good.
Currently we store the iSCSI info as metadata per image, each image stores its own iSCSI info. From a design point of view this looks clean and robust, but it does incur a performance hit, as we have to read the metadata of each image to refresh the display. It is possible we store the metadata of all disks in a single external db location/object, so we need to perform a single read to get the data, the drawback is that it is not as robust as the image depends on external entity for its iSCSI data. Reading metadata from each rbd image should not be slow when using SSD journal (rocksdb on SSD), but maybe it is slow under load in some setups. We will evaluate if storing iSCSI data outside of image metadata would be a better approach.