Slow Transfer
Pages: 1 2
Chrisjmuk
17 Posts
December 19, 2018, 10:09 amQuote from Chrisjmuk on December 19, 2018, 10:09 amWe are seeing slow network traffic when we are doing live migrations on Hyper-v it will drop to a few Kbps then back to over a 100Mbps it seems to buffer on our failover cluster server between recieving and then sending to petasan
Has anyone else seen this issue before? as i said its it a like a burst of traffic and then drop and then back up again.
Our PetaSAN Setup
3x Petasan nodes in replication
all SSD
1GB Connections x2 using iscsi and multipath IO on failover cluster
We are seeing slow network traffic when we are doing live migrations on Hyper-v it will drop to a few Kbps then back to over a 100Mbps it seems to buffer on our failover cluster server between recieving and then sending to petasan
Has anyone else seen this issue before? as i said its it a like a burst of traffic and then drop and then back up again.
Our PetaSAN Setup
3x Petasan nodes in replication
all SSD
1GB Connections x2 using iscsi and multipath IO on failover cluster
admin
2,930 Posts
December 19, 2018, 3:30 pmQuote from admin on December 19, 2018, 3:30 pmDo you see this issue only during live migration, are all other io running better ?
Is your cluster health status OK ?
Do you see any high system load on the dashboard charts ?
Could this be an issue on the hyper-v nodes ?
Do you see this issue only during live migration, are all other io running better ?
Is your cluster health status OK ?
Do you see any high system load on the dashboard charts ?
Could this be an issue on the hyper-v nodes ?
Chrisjmuk
17 Posts
December 19, 2018, 4:19 pmQuote from Chrisjmuk on December 19, 2018, 4:19 pmIt seems to be an issue with the Failover Cluster, when i add the SOFS for application or even simple, i get aroun 5MBps transfer rate, but when i transfer direct to petesan i get around 400MBps and when i go direct to the server i get maked out at 100MBps
Any ideas?
It seems to be an issue with the Failover Cluster, when i add the SOFS for application or even simple, i get aroun 5MBps transfer rate, but when i transfer direct to petesan i get around 400MBps and when i go direct to the server i get maked out at 100MBps
Any ideas?
admin
2,930 Posts
December 19, 2018, 9:07 pmQuote from admin on December 19, 2018, 9:07 pmYou should not see a difference in speed. Can you check the speed an iSCSI disk from the SOFS node, then the speed if you write directly to the csv volume from the same node ( ie not using the SOFS share ).
You should not see a difference in speed. Can you check the speed an iSCSI disk from the SOFS node, then the speed if you write directly to the csv volume from the same node ( ie not using the SOFS share ).
Chrisjmuk
17 Posts
December 20, 2018, 10:22 amQuote from Chrisjmuk on December 20, 2018, 10:22 amThere is something strange going on, of i copy files direct to the iscsi the network transfer will keep going even after the file has transfered. its like its buffering
I am using MPIO with 6 connections in total between the 3 petasan nodes i have also tried it with just 2 connections to the same node and does the same thing.
here are the images of the transfer speed
SOFS - https://imgur.com/a/XnVNBLP
Direct - FOC NODE - https://imgur.com/a/qLcc4i2
When going direct to the iscsi i cant tell because it will show the file in the folder and then buffer to the petasan but the transfer speed seems to be good.
There is something strange going on, of i copy files direct to the iscsi the network transfer will keep going even after the file has transfered. its like its buffering
I am using MPIO with 6 connections in total between the 3 petasan nodes i have also tried it with just 2 connections to the same node and does the same thing.
here are the images of the transfer speed
SOFS - https://imgur.com/a/XnVNBLP
Direct - FOC NODE - https://imgur.com/a/qLcc4i2
When going direct to the iscsi i cant tell because it will show the file in the folder and then buffer to the petasan but the transfer speed seems to be good.
Chrisjmuk
17 Posts
December 20, 2018, 10:53 amQuote from Chrisjmuk on December 20, 2018, 10:53 amSo the issue does seem to be petasan. even though it has dual 1GB connections when i transfer it will go to max 400Mbps between the two which is 40Mbps which is about right from what i am seeing, why isnt it going faster? what would be the bottleneck? i dont think its the network, and PetaSAN is all pure SSD in osd. i am not using a jurnal though as it said i didnt need to if it was SSD
So the issue does seem to be petasan. even though it has dual 1GB connections when i transfer it will go to max 400Mbps between the two which is 40Mbps which is about right from what i am seeing, why isnt it going faster? what would be the bottleneck? i dont think its the network, and PetaSAN is all pure SSD in osd. i am not using a jurnal though as it said i didnt need to if it was SSD
Chrisjmuk
17 Posts
December 20, 2018, 11:25 amQuote from Chrisjmuk on December 20, 2018, 11:25 am
Cluster Throughput
Write - 33MB/s
Read - 214 MB/s
What could be the cause of the write being so slow?
They are all pure SSD's 1X SSD for OS and 2x 1TB SSD OSD in each node with 2x 1GBPS backend connection.
Cluster Throughput
Write - 33MB/s
Read - 214 MB/s
What could be the cause of the write being so slow?
They are all pure SSD's 1X SSD for OS and 2x 1TB SSD OSD in each node with 2x 1GBPS backend connection.
Last edited on December 20, 2018, 11:26 am by Chrisjmuk · #7
admin
2,930 Posts
December 20, 2018, 1:04 pmQuote from admin on December 20, 2018, 1:04 pmI presume this is the speed of a single operation such as file copy. If you have many concurrent operations it will scale. Try coping several files at once.
Still the write speed for single operayion is below average, Ceph does use sync writes to guarantee data is flushed to storage. Some SSDS have poor sync write speed. What model are you using ?
Note you can also measure sync write speed from the node console blue screen menu, but needs to be done on a raw unused disk.
I presume this is the speed of a single operation such as file copy. If you have many concurrent operations it will scale. Try coping several files at once.
Still the write speed for single operayion is below average, Ceph does use sync writes to guarantee data is flushed to storage. Some SSDS have poor sync write speed. What model are you using ?
Note you can also measure sync write speed from the node console blue screen menu, but needs to be done on a raw unused disk.
Last edited on December 20, 2018, 1:07 pm by admin · #8
Chrisjmuk
17 Posts
December 20, 2018, 1:16 pmQuote from Chrisjmuk on December 20, 2018, 1:16 pmThat was from the benchmark from the petasan GUI.
OS - INTEL_SSDSC2BB12
OSD - Crucial_CT1024MX
OSD - Micron_1100_MTFD
I know they are not the best but should be much better than 30MB/s
That was from the benchmark from the petasan GUI.
OS - INTEL_SSDSC2BB12
OSD - Crucial_CT1024MX
OSD - Micron_1100_MTFD
I know they are not the best but should be much better than 30MB/s
Chrisjmuk
17 Posts
December 20, 2018, 1:59 pmQuote from Chrisjmuk on December 20, 2018, 1:59 pmBeen doing some reading up, i have seen that increasing the network MTU could help and also changing the RBD cache can also help?
is this worth trying?
Been doing some reading up, i have seen that increasing the network MTU could help and also changing the RBD cache can also help?
is this worth trying?
Pages: 1 2
Slow Transfer
Chrisjmuk
17 Posts
Quote from Chrisjmuk on December 19, 2018, 10:09 amWe are seeing slow network traffic when we are doing live migrations on Hyper-v it will drop to a few Kbps then back to over a 100Mbps it seems to buffer on our failover cluster server between recieving and then sending to petasan
Has anyone else seen this issue before? as i said its it a like a burst of traffic and then drop and then back up again.
Our PetaSAN Setup
3x Petasan nodes in replication
all SSD
1GB Connections x2 using iscsi and multipath IO on failover cluster
We are seeing slow network traffic when we are doing live migrations on Hyper-v it will drop to a few Kbps then back to over a 100Mbps it seems to buffer on our failover cluster server between recieving and then sending to petasan
Has anyone else seen this issue before? as i said its it a like a burst of traffic and then drop and then back up again.
Our PetaSAN Setup
3x Petasan nodes in replication
all SSD
1GB Connections x2 using iscsi and multipath IO on failover cluster
admin
2,930 Posts
Quote from admin on December 19, 2018, 3:30 pmDo you see this issue only during live migration, are all other io running better ?
Is your cluster health status OK ?
Do you see any high system load on the dashboard charts ?
Could this be an issue on the hyper-v nodes ?
Do you see this issue only during live migration, are all other io running better ?
Is your cluster health status OK ?
Do you see any high system load on the dashboard charts ?
Could this be an issue on the hyper-v nodes ?
Chrisjmuk
17 Posts
Quote from Chrisjmuk on December 19, 2018, 4:19 pmIt seems to be an issue with the Failover Cluster, when i add the SOFS for application or even simple, i get aroun 5MBps transfer rate, but when i transfer direct to petesan i get around 400MBps and when i go direct to the server i get maked out at 100MBps
Any ideas?
It seems to be an issue with the Failover Cluster, when i add the SOFS for application or even simple, i get aroun 5MBps transfer rate, but when i transfer direct to petesan i get around 400MBps and when i go direct to the server i get maked out at 100MBps
Any ideas?
admin
2,930 Posts
Quote from admin on December 19, 2018, 9:07 pmYou should not see a difference in speed. Can you check the speed an iSCSI disk from the SOFS node, then the speed if you write directly to the csv volume from the same node ( ie not using the SOFS share ).
You should not see a difference in speed. Can you check the speed an iSCSI disk from the SOFS node, then the speed if you write directly to the csv volume from the same node ( ie not using the SOFS share ).
Chrisjmuk
17 Posts
Quote from Chrisjmuk on December 20, 2018, 10:22 amThere is something strange going on, of i copy files direct to the iscsi the network transfer will keep going even after the file has transfered. its like its buffering
I am using MPIO with 6 connections in total between the 3 petasan nodes i have also tried it with just 2 connections to the same node and does the same thing.
here are the images of the transfer speed
SOFS - https://imgur.com/a/XnVNBLP
Direct - FOC NODE - https://imgur.com/a/qLcc4i2
When going direct to the iscsi i cant tell because it will show the file in the folder and then buffer to the petasan but the transfer speed seems to be good.
There is something strange going on, of i copy files direct to the iscsi the network transfer will keep going even after the file has transfered. its like its buffering
I am using MPIO with 6 connections in total between the 3 petasan nodes i have also tried it with just 2 connections to the same node and does the same thing.
here are the images of the transfer speed
SOFS - https://imgur.com/a/XnVNBLP
Direct - FOC NODE - https://imgur.com/a/qLcc4i2
When going direct to the iscsi i cant tell because it will show the file in the folder and then buffer to the petasan but the transfer speed seems to be good.
Chrisjmuk
17 Posts
Quote from Chrisjmuk on December 20, 2018, 10:53 amSo the issue does seem to be petasan. even though it has dual 1GB connections when i transfer it will go to max 400Mbps between the two which is 40Mbps which is about right from what i am seeing, why isnt it going faster? what would be the bottleneck? i dont think its the network, and PetaSAN is all pure SSD in osd. i am not using a jurnal though as it said i didnt need to if it was SSD
So the issue does seem to be petasan. even though it has dual 1GB connections when i transfer it will go to max 400Mbps between the two which is 40Mbps which is about right from what i am seeing, why isnt it going faster? what would be the bottleneck? i dont think its the network, and PetaSAN is all pure SSD in osd. i am not using a jurnal though as it said i didnt need to if it was SSD
Chrisjmuk
17 Posts
Quote from Chrisjmuk on December 20, 2018, 11:25 amCluster Throughput
Write - 33MB/s
Read - 214 MB/s
What could be the cause of the write being so slow?
They are all pure SSD's 1X SSD for OS and 2x 1TB SSD OSD in each node with 2x 1GBPS backend connection.
Cluster Throughput
Write - 33MB/s
Read - 214 MB/s
What could be the cause of the write being so slow?
They are all pure SSD's 1X SSD for OS and 2x 1TB SSD OSD in each node with 2x 1GBPS backend connection.
admin
2,930 Posts
Quote from admin on December 20, 2018, 1:04 pmI presume this is the speed of a single operation such as file copy. If you have many concurrent operations it will scale. Try coping several files at once.
Still the write speed for single operayion is below average, Ceph does use sync writes to guarantee data is flushed to storage. Some SSDS have poor sync write speed. What model are you using ?
Note you can also measure sync write speed from the node console blue screen menu, but needs to be done on a raw unused disk.
I presume this is the speed of a single operation such as file copy. If you have many concurrent operations it will scale. Try coping several files at once.
Still the write speed for single operayion is below average, Ceph does use sync writes to guarantee data is flushed to storage. Some SSDS have poor sync write speed. What model are you using ?
Note you can also measure sync write speed from the node console blue screen menu, but needs to be done on a raw unused disk.
Chrisjmuk
17 Posts
Quote from Chrisjmuk on December 20, 2018, 1:16 pmThat was from the benchmark from the petasan GUI.
OS - INTEL_SSDSC2BB12
OSD - Crucial_CT1024MX
OSD - Micron_1100_MTFD
I know they are not the best but should be much better than 30MB/s
That was from the benchmark from the petasan GUI.
OS - INTEL_SSDSC2BB12
OSD - Crucial_CT1024MX
OSD - Micron_1100_MTFD
I know they are not the best but should be much better than 30MB/s
Chrisjmuk
17 Posts
Quote from Chrisjmuk on December 20, 2018, 1:59 pmBeen doing some reading up, i have seen that increasing the network MTU could help and also changing the RBD cache can also help?
is this worth trying?
Been doing some reading up, i have seen that increasing the network MTU could help and also changing the RBD cache can also help?
is this worth trying?