Forums

Home / Forums

You need to log in to create posts and topics. Login · Register

Esxi Server Freeze.

Pages: 1 2 3 4

admin
2,930 Posts
February 11, 2019, 2:51 pm

i can't be sure if it is clean or not, a re-install would have taken a couple of minutes, but it up to you.

Re performance

1 )  This is not the first time you installed PetaSAN, is the performance worse than before or how does it compare ?

2)  From the benchmark page can you run 1 min test with:

  • 4k iops 1 client 1 thread
  • 4k iops 2 clients 64 threads
  • 4M Thoughput 1 client 64 threads

3)  What MB/s do you get in VMotion ?

#21

msalem
87 Posts
February 11, 2019, 3:39 pm

-- Its clean from what I can see no errors in the logs, since I deleted the pools and images, checked IPMI and everything looks stable. Re-installing will take time, they are 6 nodes in 4/2 config in EC + rbd.

I am running DD command on VM's on the LUNs , mutiple: writes = 350MB and Reads are 1.4GB ..

But as I mentioned - anything VMware related is very slow.. we are talking 10MB -- 40MB.

I will run the benchmarks and reply back.

#22

msalem
87 Posts
February 11, 2019, 4:05 pm
  • 4k iops 1 client 1 thread

Results

Cluster IOPS

Write Read

194      67

Write Resource Load:

Node   Memory Util%          CPU Util%    Network Util%          Disks Util% Actions

Avg     Max    Avg     Max    Avg     Max

srocceph5       20        16        20        5          9          10        13       

srocceph2       22        16        20        4          8          12        17       

srocceph4       20        15        20        4          9          10        15       

srocceph1       20        20        64        10        22        10        15       

srocceph3       25        17        43        13        26        10        14       

Read Resource Load:

Node   Memory Util%          CPU Util%    Network Util%          Disks Util% Actions

Avg     Max    Avg     Max    Avg     Max

srocceph1       20        19        71        10        21        13        18       

srocceph3       25        17        41        13        25        13        18       

srocceph5       20        15        19        4          8          13        15       

srocceph4       21        15        18        4          8          12        19       

srocceph2       21        15        20        4          8          12        16       

  • 4k iops 2 clients 64 threads

 

Results

Cluster IOPS

Write        Read

10633        2357

Write Resource Load:

Node         Memory Util%          CPU Util%    Network Util%       Disks Util%   Actions

Avg           Max    Avg     Max    Avg     Max

srocceph2 22        49        60        5          11        30        35       

srocceph4 21        49        60        5          12        30        40       

srocceph1 21        55        99        9          16        30        40       

srocceph3 25        52        71        11        24        31        42       

Read Resource Load:

Node         Memory Util%          CPU Util%    Network Util%       Disks Util%   Actions

Avg           Max    Avg     Max    Avg     Max

srocceph1 21        35        76        5          11        73        85       

srocceph3 26        35        45        7          14        74        93       

srocceph4 20        33        41        2          4          72        88       

srocceph2 21        34        40        2          5          73        82       

  • 4M Thoughput 1 client 64 threads

 

Results

Cluster Throughput

Write    Read

1418 MB/s         1344 MB/s

Write Resource Load:

Node     Memory Util%   CPU Util%           Network Util%   Disks Util%               Actions

Avg        Max       Avg        Max       Avg        Max

srocceph5          14          18          22          20          34          11          13

srocceph1          21          22          69          25          42          11          17

srocceph3          26          20          43          24          49          11          15

srocceph4          21          18          20          19          33          11          15

srocceph2          22          18          21          19          33          11          14

Read Resource Load:

Node     Memory Util%   CPU Util%           Network Util%   Disks Util%               Actions

Avg        Max       Avg        Max       Avg        Max

srocceph5          14          15          19          14          22          22          27

srocceph3          25          17          40          21          34          21          32

srocceph2          22          15          19          14          21          20          24

srocceph1          21          19          72          19          36          23          32

srocceph4          21          14          20          13          20          20          27

#23

admin
2,930 Posts
February 11, 2019, 4:40 pm

I am running DD command on VM's on the LUNs , mutiple: writes = 350MB and Reads are 1.4GB ..
But as I mentioned - anything VMware related is very slow.. we are talking 10MB -- 40MB.

Just to clarify..are you ok with read/write speed from vm running in WMware, but not with VMotion/backups ie non read/write  io within the vm ?....when you say "anything vmware related" you mean vmotion not vm io ? we did do some recent improvements to the vmotion speed (up to to 400MB/s per lun) that we will soon post.

Generally your cluster is slow, maybe ok for backups etc but for virtualization it is not good. Your latency is 15 ms read 5 ms write due to spinning hdds, random iops are 2.3 k read and 11k write.  Your hdds are over 90% busy in read iops. The journal is helping a bit for writes. An ssd cluster will give you about 0.6 ms read latency.

If you will use it for virtualization, you should at least get a controller with write back cache for the spinners + add more of them.

Last edited on February 11, 2019, 4:53 pm by admin · #24

msalem
87 Posts
February 11, 2019, 4:59 pm

Thanks Admin for the review.

I would expect better speeds really from Vmware atleast, in terms of the VM's the hardware is high performance hardware. so VM IO should be a bit better than this.

I have a few VM's doing some tests ad well, I will shut them down and test again. with the same nodes and tests.
Thanks again

#25

msalem
87 Posts
February 11, 2019, 5:12 pm

New Results:

 

  • 4k iops 1 client 1 thread

Results

Cluster IOPS

Write Read

466      50

Write Resource Load:

Node   Memory Util%          CPU Util%    Network Util%          Disks Util% Actions

Avg     Max    Avg     Max    Avg     Max

srocceph5       14        2          4          0          0          1          1         

srocceph3       26        2          4          0          0          1          1         

srocceph2       21        2          6          0          0          1          1         

srocceph1       21        5          51        0          0          1          1         

srocceph4       21        2          4          0          0          1          1         

Read Resource Load:

Node   Memory Util%          CPU Util%    Network Util%          Disks Util% Actions

Avg     Max    Avg     Max    Avg     Max

srocceph4       21        1          3          0          0          2          4         

srocceph1       21        3          68        0          0          3          4         

srocceph2       21        1          3          0          0          2          4         

srocceph3       26        1          4          0          0          3          4         

srocceph5       14        1          3          0          0          3          4         

 

 

  • 4k iops 2 clients 64 threads

 

Results

Cluster IOPS

Write Read

13278  2807

Write Resource Load:

Node   Memory Util%          CPU Util%    Network Util%             Disks Util%   Actions

Avg     Max    Avg     Max    Avg     Max

srocceph1       21        54        77        2          7          31        37           

srocceph3       25        51        65        2          7          30        39           

srocceph2       22        51        63        2          7          30        35           

srocceph4       21        50        63        2          7          30        37           

Read Resource Load:

Node   Memory Util%          CPU Util%    Network Util%             Disks Util%   Actions

Avg     Max    Avg     Max    Avg     Max

srocceph1       21        32        73        0          0          77        89           

srocceph3       26        31        45        0          0          77        92           

srocceph4       21        29        38        0          0          74        89           

srocceph2       22        31        41        0          0          77        86           

 

  • 4M Thoughput 1 client 64 threads

 

Results

Cluster Throughput

Write    Read

1422 MB/s         1432 MB/s

Write Resource Load:

Node     Memory Util%   CPU Util%           Network Util%   Disks Util%               Actions

Avg        Max       Avg        Max       Avg        Max

srocceph2          22          4            8            15          25          1            1

srocceph1          21          7            70          14          25          1            2

srocceph3          26          4            7            14          25          1            2

srocceph4          21          4            7            15          25          1            2

srocceph5          14          4            8            16          27          1            2

Read Resource Load:

Node     Memory Util%   CPU Util%           Network Util%   Disks Util%               Actions

Avg        Max       Avg        Max       Avg        Max

srocceph1          21          5            65          11          19          9            11

srocceph4          21          2            4            11          19          10          12

srocceph2          22          2            4            12          20          9            12

srocceph3          26          2            5            11          19          9            14

srocceph5          14          2            4            12          21          10          12

#26

admin
2,930 Posts
February 11, 2019, 5:21 pm

same results..so for iops intensive workloads, you should at least get a controller with write back cache for the spinners. Using spinning disk without controller is not recommended.

Last edited on February 11, 2019, 5:23 pm by admin · #27

msalem
87 Posts
February 11, 2019, 5:34 pm

This is the model of the Server:

Supermicro SuperStorage Server 6028R-E1CR12T - 12x SATA/SAS - LSI 3108 12G SAS onboard RAID - Dual 10-Gigabit Ethernet - 920W Redundant

#28

msalem
87 Posts
February 11, 2019, 5:56 pm

One more question,

the avarage load is on a normal state: 3 to 10 is this normal.

 

Thanks

#29

admin
2,930 Posts
February 11, 2019, 6:12 pm

what do you mean ?

#30
Pages: 1 2 3 4