PetaSAN in our 2 datacenters for production
Syscon
23 Posts
January 21, 2020, 12:36 pmQuote from Syscon on January 21, 2020, 12:36 pmThank you, a reboot of NODE03 helped to get the Ceph status healthy again.
Now we started to test snapshotting.
We have two PetaSAN storages, both with 3 NODES and same hardware (see also start of this tread), per NODE:
Dell Power Edge R620 (R620 chassis + mainboard v2)
Server chassis has space for 10 2.5" SAS/SSD disks
CPU: 2x 1.70GHz/Ten Core/QPi 7.20/Cache 25 MB/TDP 70W/64-bit XEON E5-2650L v2
128 GB memory
Dell PERC H310 Mini Mono
1x KST-SEDC500M/1920G SSD for OS
3x KST-SEDC500M/1920G SSD ODS's (second PetaSAN setup has 8 OSD’s per NODE)
2x 10GB+2x1GB network (iSCSI+Heartbeat are combined via a BOND on both 10GB ports; Management owns 1x1GB adapter)
We are testing snapshotting in both our 2 VMWare (5.5) environments, using iSCSI (one PetaSAN operating in each environment).
In both our PetaSAN/VMWare setups a snapshot causes the VM to freeze for a few seconds or more (depending on the size of the VM and the amount of snapshots) in case we:
- create a snapshot
- delete a snapshot
- delete all snapshots
Furthermore, the time to remove an existing snapshot takes minutes.
One of the setups has only 20% storage (the other 50%) in use.
Tested VM isn’t big (40GB) and in rest (no workload), HOST has 10GB network interface and not a lot of workload. Snapshot is made without the VM’s memory.
Checking Disk and Network Throughput, IOP’s, Latency, CPU, Memory during a snapshot doesn’t show any alarming values.
When running the same VM’s on our NAS4Free storages (in the same VMWare environments) removing a snapshot takes seconds instead of minutes with PetaSAN.
Are there things I need to adjust and/or tune in PetaSAN (or VMWare) to avoid this problem? Thanks in advance!
Thank you, a reboot of NODE03 helped to get the Ceph status healthy again.
Now we started to test snapshotting.
We have two PetaSAN storages, both with 3 NODES and same hardware (see also start of this tread), per NODE:
Dell Power Edge R620 (R620 chassis + mainboard v2)
Server chassis has space for 10 2.5" SAS/SSD disks
CPU: 2x 1.70GHz/Ten Core/QPi 7.20/Cache 25 MB/TDP 70W/64-bit XEON E5-2650L v2
128 GB memory
Dell PERC H310 Mini Mono
1x KST-SEDC500M/1920G SSD for OS
3x KST-SEDC500M/1920G SSD ODS's (second PetaSAN setup has 8 OSD’s per NODE)
2x 10GB+2x1GB network (iSCSI+Heartbeat are combined via a BOND on both 10GB ports; Management owns 1x1GB adapter)
We are testing snapshotting in both our 2 VMWare (5.5) environments, using iSCSI (one PetaSAN operating in each environment).
In both our PetaSAN/VMWare setups a snapshot causes the VM to freeze for a few seconds or more (depending on the size of the VM and the amount of snapshots) in case we:
- create a snapshot
- delete a snapshot
- delete all snapshots
Furthermore, the time to remove an existing snapshot takes minutes.
One of the setups has only 20% storage (the other 50%) in use.
Tested VM isn’t big (40GB) and in rest (no workload), HOST has 10GB network interface and not a lot of workload. Snapshot is made without the VM’s memory.
Checking Disk and Network Throughput, IOP’s, Latency, CPU, Memory during a snapshot doesn’t show any alarming values.
When running the same VM’s on our NAS4Free storages (in the same VMWare environments) removing a snapshot takes seconds instead of minutes with PetaSAN.
Are there things I need to adjust and/or tune in PetaSAN (or VMWare) to avoid this problem? Thanks in advance!
Last edited on January 21, 2020, 12:36 pm by Syscon · #41
admin
2,930 Posts
January 21, 2020, 8:27 pmQuote from admin on January 21, 2020, 8:27 pmThe vmware freeze of a few seconds during snapshot creation is due to flushing all pending writes.
when you measure deletion, is this using snapshot commands via cli commands or do you are you referring to PetaSAN's disk replication ?
The vmware freeze of a few seconds during snapshot creation is due to flushing all pending writes.
when you measure deletion, is this using snapshot commands via cli commands or do you are you referring to PetaSAN's disk replication ?
Last edited on January 21, 2020, 8:27 pm by admin · #42
Syscon
23 Posts
January 22, 2020, 9:14 amQuote from Syscon on January 22, 2020, 9:14 amSorry for not being clear, I am referring to creating (or removing) a snapshot using vCenter on a specific VM. This VM then freezes much longer and snapshot removal takes much longer (minutes instead of seconds) on PetaSAN in comparism to our NAS4Free storages.
Sorry for not being clear, I am referring to creating (or removing) a snapshot using vCenter on a specific VM. This VM then freezes much longer and snapshot removal takes much longer (minutes instead of seconds) on PetaSAN in comparism to our NAS4Free storages.
Last edited on January 22, 2020, 9:17 am by Syscon · #43
admin
2,930 Posts
January 22, 2020, 10:41 amQuote from admin on January 22, 2020, 10:41 amok i understand now. currently vmware snapshots is controlled totally within vmware, no ceph snapshot integration,this kind of "hardware assisted" snapshots integration is in our list of features to add. as i understand freenas/nas4free do have support for this.
ok i understand now. currently vmware snapshots is controlled totally within vmware, no ceph snapshot integration,this kind of "hardware assisted" snapshots integration is in our list of features to add. as i understand freenas/nas4free do have support for this.
Syscon
23 Posts
January 22, 2020, 11:57 amQuote from Syscon on January 22, 2020, 11:57 amBut does that mean that with PetaSAN I will have problems (the ones as described earlier) when taking snapshots of VM's with VMWare?
Also using NAS4Free, we are only using snapshotting via VMWare like i try now on PetaSAN. And only on PetaSAN the VM's have problems during a snapshot...
(We schedule snapshots (3 per day, so always 3 snaps per VM) in case of viruses or other problems with these VM's).
But does that mean that with PetaSAN I will have problems (the ones as described earlier) when taking snapshots of VM's with VMWare?
Also using NAS4Free, we are only using snapshotting via VMWare like i try now on PetaSAN. And only on PetaSAN the VM's have problems during a snapshot...
(We schedule snapshots (3 per day, so always 3 snaps per VM) in case of viruses or other problems with these VM's).
Last edited on January 22, 2020, 12:09 pm by Syscon · #45
admin
2,930 Posts
January 23, 2020, 7:57 amQuote from admin on January 23, 2020, 7:57 amIt will work, it will be handled by vmware, but will not be as efficient till we integrate it with ceph snapshots.
It will work, it will be handled by vmware, but will not be as efficient till we integrate it with ceph snapshots.
Last edited on January 23, 2020, 7:57 am by admin · #46
Syscon
23 Posts
January 23, 2020, 1:52 pmQuote from Syscon on January 23, 2020, 1:52 pmSo do I understand well that this article is a good resume of the situation? -> https://virtunetsystems.com/news/improving-the-performance-of-ceph-storage-vmware/
The solution would then be (until integration of Ceph snapshots) to implement VirtuCache?
So do I understand well that this article is a good resume of the situation? -> https://virtunetsystems.com/news/improving-the-performance-of-ceph-storage-vmware/
The solution would then be (until integration of Ceph snapshots) to implement VirtuCache?
admin
2,930 Posts
January 23, 2020, 3:00 pmQuote from admin on January 23, 2020, 3:00 pm
- i am not sure the link you sent does snapshot specific acceleration or just regular io. For regular io we already offer write cache support in 2.4.0 to decrease latency of hdds, for all flash you do not need caching.
- vmware snapshots work, but as you said are slower that what you already have, if the performance is acceptable then you should just use them as is.
- you can achieve something similar with PetaSAN's replication feature, it uses ceph snapshots and it is possible to replicate to a different disk within the same cluster.
- i am not sure the link you sent does snapshot specific acceleration or just regular io. For regular io we already offer write cache support in 2.4.0 to decrease latency of hdds, for all flash you do not need caching.
- vmware snapshots work, but as you said are slower that what you already have, if the performance is acceptable then you should just use them as is.
- you can achieve something similar with PetaSAN's replication feature, it uses ceph snapshots and it is possible to replicate to a different disk within the same cluster.
kitcho
1 Post
September 2, 2022, 9:57 amQuote from kitcho on September 2, 2022, 9:57 amI think no one is between Admin and Syscon. Only 2 users are replying each other. Is there any one who will reply me..? 
I think no one is between Admin and Syscon. Only 2 users are replying each other. Is there any one who will reply me..?
Last edited on September 2, 2022, 10:01 am by kitcho · #49
admin
2,930 Posts
September 2, 2022, 1:59 pmQuote from admin on September 2, 2022, 1:59 pmwhat/where is your question ?
what/where is your question ?
PetaSAN in our 2 datacenters for production
Syscon
23 Posts
Quote from Syscon on January 21, 2020, 12:36 pmThank you, a reboot of NODE03 helped to get the Ceph status healthy again.
Now we started to test snapshotting.
We have two PetaSAN storages, both with 3 NODES and same hardware (see also start of this tread), per NODE:
Dell Power Edge R620 (R620 chassis + mainboard v2)
Server chassis has space for 10 2.5" SAS/SSD disks
CPU: 2x 1.70GHz/Ten Core/QPi 7.20/Cache 25 MB/TDP 70W/64-bit XEON E5-2650L v2
128 GB memory
Dell PERC H310 Mini Mono
1x KST-SEDC500M/1920G SSD for OS
3x KST-SEDC500M/1920G SSD ODS's (second PetaSAN setup has 8 OSD’s per NODE)
2x 10GB+2x1GB network (iSCSI+Heartbeat are combined via a BOND on both 10GB ports; Management owns 1x1GB adapter)
We are testing snapshotting in both our 2 VMWare (5.5) environments, using iSCSI (one PetaSAN operating in each environment).
In both our PetaSAN/VMWare setups a snapshot causes the VM to freeze for a few seconds or more (depending on the size of the VM and the amount of snapshots) in case we:
- create a snapshot
- delete a snapshot
- delete all snapshots
Furthermore, the time to remove an existing snapshot takes minutes.
One of the setups has only 20% storage (the other 50%) in use.
Tested VM isn’t big (40GB) and in rest (no workload), HOST has 10GB network interface and not a lot of workload. Snapshot is made without the VM’s memory.
Checking Disk and Network Throughput, IOP’s, Latency, CPU, Memory during a snapshot doesn’t show any alarming values.
When running the same VM’s on our NAS4Free storages (in the same VMWare environments) removing a snapshot takes seconds instead of minutes with PetaSAN.
Are there things I need to adjust and/or tune in PetaSAN (or VMWare) to avoid this problem? Thanks in advance!
Thank you, a reboot of NODE03 helped to get the Ceph status healthy again.
Now we started to test snapshotting.
We have two PetaSAN storages, both with 3 NODES and same hardware (see also start of this tread), per NODE:
Dell Power Edge R620 (R620 chassis + mainboard v2)
Server chassis has space for 10 2.5" SAS/SSD disks
CPU: 2x 1.70GHz/Ten Core/QPi 7.20/Cache 25 MB/TDP 70W/64-bit XEON E5-2650L v2
128 GB memory
Dell PERC H310 Mini Mono
1x KST-SEDC500M/1920G SSD for OS
3x KST-SEDC500M/1920G SSD ODS's (second PetaSAN setup has 8 OSD’s per NODE)
2x 10GB+2x1GB network (iSCSI+Heartbeat are combined via a BOND on both 10GB ports; Management owns 1x1GB adapter)
We are testing snapshotting in both our 2 VMWare (5.5) environments, using iSCSI (one PetaSAN operating in each environment).
In both our PetaSAN/VMWare setups a snapshot causes the VM to freeze for a few seconds or more (depending on the size of the VM and the amount of snapshots) in case we:
- create a snapshot
- delete a snapshot
- delete all snapshots
Furthermore, the time to remove an existing snapshot takes minutes.
One of the setups has only 20% storage (the other 50%) in use.
Tested VM isn’t big (40GB) and in rest (no workload), HOST has 10GB network interface and not a lot of workload. Snapshot is made without the VM’s memory.
Checking Disk and Network Throughput, IOP’s, Latency, CPU, Memory during a snapshot doesn’t show any alarming values.
When running the same VM’s on our NAS4Free storages (in the same VMWare environments) removing a snapshot takes seconds instead of minutes with PetaSAN.
Are there things I need to adjust and/or tune in PetaSAN (or VMWare) to avoid this problem? Thanks in advance!
admin
2,930 Posts
Quote from admin on January 21, 2020, 8:27 pmThe vmware freeze of a few seconds during snapshot creation is due to flushing all pending writes.
when you measure deletion, is this using snapshot commands via cli commands or do you are you referring to PetaSAN's disk replication ?
The vmware freeze of a few seconds during snapshot creation is due to flushing all pending writes.
when you measure deletion, is this using snapshot commands via cli commands or do you are you referring to PetaSAN's disk replication ?
Syscon
23 Posts
Quote from Syscon on January 22, 2020, 9:14 amSorry for not being clear, I am referring to creating (or removing) a snapshot using vCenter on a specific VM. This VM then freezes much longer and snapshot removal takes much longer (minutes instead of seconds) on PetaSAN in comparism to our NAS4Free storages.
Sorry for not being clear, I am referring to creating (or removing) a snapshot using vCenter on a specific VM. This VM then freezes much longer and snapshot removal takes much longer (minutes instead of seconds) on PetaSAN in comparism to our NAS4Free storages.
admin
2,930 Posts
Quote from admin on January 22, 2020, 10:41 amok i understand now. currently vmware snapshots is controlled totally within vmware, no ceph snapshot integration,this kind of "hardware assisted" snapshots integration is in our list of features to add. as i understand freenas/nas4free do have support for this.
ok i understand now. currently vmware snapshots is controlled totally within vmware, no ceph snapshot integration,this kind of "hardware assisted" snapshots integration is in our list of features to add. as i understand freenas/nas4free do have support for this.
Syscon
23 Posts
Quote from Syscon on January 22, 2020, 11:57 amBut does that mean that with PetaSAN I will have problems (the ones as described earlier) when taking snapshots of VM's with VMWare?
Also using NAS4Free, we are only using snapshotting via VMWare like i try now on PetaSAN. And only on PetaSAN the VM's have problems during a snapshot...
(We schedule snapshots (3 per day, so always 3 snaps per VM) in case of viruses or other problems with these VM's).
But does that mean that with PetaSAN I will have problems (the ones as described earlier) when taking snapshots of VM's with VMWare?
Also using NAS4Free, we are only using snapshotting via VMWare like i try now on PetaSAN. And only on PetaSAN the VM's have problems during a snapshot...
(We schedule snapshots (3 per day, so always 3 snaps per VM) in case of viruses or other problems with these VM's).
admin
2,930 Posts
Quote from admin on January 23, 2020, 7:57 amIt will work, it will be handled by vmware, but will not be as efficient till we integrate it with ceph snapshots.
It will work, it will be handled by vmware, but will not be as efficient till we integrate it with ceph snapshots.
Syscon
23 Posts
Quote from Syscon on January 23, 2020, 1:52 pmSo do I understand well that this article is a good resume of the situation? -> https://virtunetsystems.com/news/improving-the-performance-of-ceph-storage-vmware/
The solution would then be (until integration of Ceph snapshots) to implement VirtuCache?
So do I understand well that this article is a good resume of the situation? -> https://virtunetsystems.com/news/improving-the-performance-of-ceph-storage-vmware/
The solution would then be (until integration of Ceph snapshots) to implement VirtuCache?
admin
2,930 Posts
Quote from admin on January 23, 2020, 3:00 pm
- i am not sure the link you sent does snapshot specific acceleration or just regular io. For regular io we already offer write cache support in 2.4.0 to decrease latency of hdds, for all flash you do not need caching.
- vmware snapshots work, but as you said are slower that what you already have, if the performance is acceptable then you should just use them as is.
- you can achieve something similar with PetaSAN's replication feature, it uses ceph snapshots and it is possible to replicate to a different disk within the same cluster.
- i am not sure the link you sent does snapshot specific acceleration or just regular io. For regular io we already offer write cache support in 2.4.0 to decrease latency of hdds, for all flash you do not need caching.
- vmware snapshots work, but as you said are slower that what you already have, if the performance is acceptable then you should just use them as is.
- you can achieve something similar with PetaSAN's replication feature, it uses ceph snapshots and it is possible to replicate to a different disk within the same cluster.
kitcho
1 Post
Quote from kitcho on September 2, 2022, 9:57 amI think no one is between Admin and Syscon. Only 2 users are replying each other. Is there any one who will reply me..?
I think no one is between Admin and Syscon. Only 2 users are replying each other. Is there any one who will reply me..?
admin
2,930 Posts
Quote from admin on September 2, 2022, 1:59 pmwhat/where is your question ?
what/where is your question ?