Cluster Speed/Throughput help
jcid
14 Posts
February 14, 2020, 2:30 amQuote from jcid on February 14, 2020, 2:30 amHello,
My cluster setup is as follows:
3 node cluster with Raid1 OS and two stand alone 800GB SSD DATA disks. 32 CPU and 128GB Ram. Using 2GB ISCSI MPIO PortGroup with VMWare ESX6. 1GB interlink between SAN nodes for replication. PetaSAN Benchmark throughput shows 78MB/s and 191 MB/s. Running a dd test on the local datastore1 SSD gives me about 450MB/s so I know there capable. Testing from a guest vm only gives me about 75Mb/s.
Using all SSD (6 SSD Cluster 2 per node. 4TB Replicated ISCSI disk with 6 active paths) using 122PG's. No journal or Cache disks.
Any reason there would be such low throughput?
Hello,
My cluster setup is as follows:
3 node cluster with Raid1 OS and two stand alone 800GB SSD DATA disks. 32 CPU and 128GB Ram. Using 2GB ISCSI MPIO PortGroup with VMWare ESX6. 1GB interlink between SAN nodes for replication. PetaSAN Benchmark throughput shows 78MB/s and 191 MB/s. Running a dd test on the local datastore1 SSD gives me about 450MB/s so I know there capable. Testing from a guest vm only gives me about 75Mb/s.
Using all SSD (6 SSD Cluster 2 per node. 4TB Replicated ISCSI disk with 6 active paths) using 122PG's. No journal or Cache disks.
Any reason there would be such low throughput?
Last edited on February 14, 2020, 2:33 am by jcid · #1
admin
2,930 Posts
February 14, 2020, 12:38 pmQuote from admin on February 14, 2020, 12:38 pm1 G bps = 125 MB/s will not be enough to give you 450 MB/s x 2 replication factor. the dd command is use the page cache, test it with direct and sync flags to bypass the cache.
1 G bps = 125 MB/s will not be enough to give you 450 MB/s x 2 replication factor. the dd command is use the page cache, test it with direct and sync flags to bypass the cache.
jcid
14 Posts
February 14, 2020, 3:16 pmQuote from jcid on February 14, 2020, 3:16 pmBoth of these result in about the same..
dd if=/dev/zero of=/tmp/test1.img bs=1G count=1 oflag=dsync
dd if=/dev/zero of=/tmp/test1.img bs=1G count=1 oflag=sync
1073741824 bytes (1.1 GB) copied, 2.55593 s, 420 MB/s (local SSD)
Ok, So the interlink needs to be as fast as the ISCSI interfaces basically then? Ok. Ill have to rebuild this again lol and slap in the 10G interfaces. This is fun! 😉
Great SAN and Support!!
Both of these result in about the same..
dd if=/dev/zero of=/tmp/test1.img bs=1G count=1 oflag=dsync
dd if=/dev/zero of=/tmp/test1.img bs=1G count=1 oflag=sync
1073741824 bytes (1.1 GB) copied, 2.55593 s, 420 MB/s (local SSD)
Ok, So the interlink needs to be as fast as the ISCSI interfaces basically then? Ok. Ill have to rebuild this again lol and slap in the 10G interfaces. This is fun! 😉
Great SAN and Support!!
Cluster Speed/Throughput help
jcid
14 Posts
Quote from jcid on February 14, 2020, 2:30 amHello,
My cluster setup is as follows:
3 node cluster with Raid1 OS and two stand alone 800GB SSD DATA disks. 32 CPU and 128GB Ram. Using 2GB ISCSI MPIO PortGroup with VMWare ESX6. 1GB interlink between SAN nodes for replication. PetaSAN Benchmark throughput shows 78MB/s and 191 MB/s. Running a dd test on the local datastore1 SSD gives me about 450MB/s so I know there capable. Testing from a guest vm only gives me about 75Mb/s.
Using all SSD (6 SSD Cluster 2 per node. 4TB Replicated ISCSI disk with 6 active paths) using 122PG's. No journal or Cache disks.
Any reason there would be such low throughput?
Hello,
My cluster setup is as follows:
3 node cluster with Raid1 OS and two stand alone 800GB SSD DATA disks. 32 CPU and 128GB Ram. Using 2GB ISCSI MPIO PortGroup with VMWare ESX6. 1GB interlink between SAN nodes for replication. PetaSAN Benchmark throughput shows 78MB/s and 191 MB/s. Running a dd test on the local datastore1 SSD gives me about 450MB/s so I know there capable. Testing from a guest vm only gives me about 75Mb/s.
Using all SSD (6 SSD Cluster 2 per node. 4TB Replicated ISCSI disk with 6 active paths) using 122PG's. No journal or Cache disks.
Any reason there would be such low throughput?
admin
2,930 Posts
Quote from admin on February 14, 2020, 12:38 pm1 G bps = 125 MB/s will not be enough to give you 450 MB/s x 2 replication factor. the dd command is use the page cache, test it with direct and sync flags to bypass the cache.
1 G bps = 125 MB/s will not be enough to give you 450 MB/s x 2 replication factor. the dd command is use the page cache, test it with direct and sync flags to bypass the cache.
jcid
14 Posts
Quote from jcid on February 14, 2020, 3:16 pmBoth of these result in about the same..
dd if=/dev/zero of=/tmp/test1.img bs=1G count=1 oflag=dsync
dd if=/dev/zero of=/tmp/test1.img bs=1G count=1 oflag=sync
1073741824 bytes (1.1 GB) copied, 2.55593 s, 420 MB/s (local SSD)
Ok, So the interlink needs to be as fast as the ISCSI interfaces basically then? Ok. Ill have to rebuild this again lol and slap in the 10G interfaces. This is fun! 😉
Great SAN and Support!!
Both of these result in about the same..
dd if=/dev/zero of=/tmp/test1.img bs=1G count=1 oflag=dsync
dd if=/dev/zero of=/tmp/test1.img bs=1G count=1 oflag=sync
1073741824 bytes (1.1 GB) copied, 2.55593 s, 420 MB/s (local SSD)
Ok, So the interlink needs to be as fast as the ISCSI interfaces basically then? Ok. Ill have to rebuild this again lol and slap in the 10G interfaces. This is fun! 😉
Great SAN and Support!!