Forums

Home / Forums

You need to log in to create posts and topics. Login · Register

Disks performance


abbas
8 Posts
March 14, 2022, 5:32 pm

First of all, thanks. I have a doubt, how could I know if the performance I am getting in the system with Petasan is good? I have seen the technical characteristics of the disks and together we think that it should have better IOPS and reading and writing speed.

I appreciate recommendations to check the performance I'm getting.

Thank you very much in advance.

#1

admin
2,930 Posts
March 14, 2022, 6:30 pm

The performance varies with hardware. Have you already run benchmarks on the cluster from the benchmark pages ? can you run tests now or are you in production ? what is your current config ?

#2

abbas
8 Posts
March 15, 2022, 6:17 pm

Hi again,

Yes I can run the tests, I attach the results:

Test 4k IOPS Threads per Client 16 duration 1 min.

Results

Cluster IOPS

Write Read
7596 83878

Test 4m throughput Threads per Client 16 duration 1 min.

Results

Cluster Throughput

Write Read
2346 MB/s 2641 MB/s

Config: 3 nodes with 2 local storage service

POOL disk 8 NVME WD Red SN700 1000GB

type replicated - rbd - PG 512 - Size 2 - 4 NVME each node

If you need any additional information please ask me

Thanks again.

Best regards

 

#3

admin
2,930 Posts
March 16, 2022, 11:52 am

Read iops is good but write is low. Throughput is good.

We recommend using enterprise class SSD, among other things they have high FUA/sync iops write speed. Not sure what the WD Red SN700 but it could have limited FUA/sync iops: You can test raw disk speed from PetaSAN blue console screen on a raw disk: it will destroy on data on it so do not use a running OSD or disk with data. The test will report the write sync iops. You could also use fio tool to do the same.

Back to the cluster benchmark test you did from ui: what was the % utlization of the disk and cpu ? It is also useful to run the test for 5 min and look at the utilization charts for cpu and disk utilization in dashboard.

Also if you have 4 disks per node you may get higher results by increasing the threads in the test from 16 to say 64

Last edited on March 16, 2022, 11:52 am by admin · #4

abbas
8 Posts
March 16, 2022, 8:25 pm

Test 4k iops 64 threads 5 minutes:

Results

  • Cluster IOPS
  1. Write Read
    19764 125667
  • Write Resource Load:
  1. Node Memory Util% CPU Util% Network Util% Disks Util% Actions
    Avg Max Avg Max Journals OSDs
    Avg Max Avg Max
    NODO1      53 41 48 0 2 0 0 32 97
  • Read Resource Load:
  1. Node Memory Util% CPU Util% Network Util% Disks Util% Actions
    Avg Max Avg Max Journals OSDs
    Avg Max Avg Max
    NODO1     53 23 32 2 5 0 0 33 100

 

write Test fio in a nvme  :

Starting 8 processes
Jobs: 8 (f=8): [w(8)][100.0%][r=0KiB/s,w=28.4MiB/s][r=0,w=7271 IOPS][eta 00m:00s]
xx: (groupid=0, jobs=8): err= 0: pid=186314: Wed Mar 16 21:24:07 2022
write: IOPS=10.9k, BW=42.8MiB/s (44.8MB/s)(430MiB/10046msec)
clat (usec): min=16, max=62065, avg=729.46, stdev=5984.30
lat (usec): min=16, max=62065, avg=729.53, stdev=5984.30
clat percentiles (usec):
| 1.00th=[ 23], 5.00th=[ 25], 10.00th=[ 26], 20.00th=[ 26],
| 30.00th=[ 26], 40.00th=[ 26], 50.00th=[ 26], 60.00th=[ 26],
| 70.00th=[ 26], 80.00th=[ 26], 90.00th=[ 28], 95.00th=[ 29],
| 99.00th=[52691], 99.50th=[56886], 99.90th=[60031], 99.95th=[61080],
| 99.99th=[62129]
bw ( KiB/s): min= 2480, max=40152, per=12.56%, avg=5500.46, stdev=8173.79, samples=160
iops : min= 620, max=10038, avg=1374.90, stdev=2043.50, samples=160
lat (usec) : 20=0.14%, 50=96.74%, 100=0.85%, 250=0.25%, 500=0.03%
lat (msec) : 2=0.04%, 4=0.06%, 10=0.54%, 20=0.21%, 50=0.06%
lat (msec) : 100=1.07%
cpu : usr=0.28%, sys=0.27%, ctx=109990, majf=0, minf=63
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,109968,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
WRITE: bw=42.8MiB/s (44.8MB/s), 42.8MiB/s-42.8MiB/s (44.8MB/s-44.8MB/s), io=430MiB (450MB), run=10046-10046msec

Disk stats (read/write):
nvme3n1: ios=115/108688, merge=0/0, ticks=220/71520, in_queue=82204, util=100.00%

thanks

Last edited on March 16, 2022, 8:29 pm by abbas · #5

abbas
8 Posts
March 16, 2022, 8:37 pm

can I  test raw disk speed from PetaSAN blue console screen on a raw disk? I can not find it. Thanks for everything

 

Last edited on March 16, 2022, 8:37 pm by abbas · #6

admin
2,930 Posts
March 17, 2022, 9:09 pm

the numbers are better now. do you see any saturation on the cpu % util or disk % util charts on dashboard ?

disk speed test  is on the blue console menu, displayed on the node monitor not accessed from browser

#7