Forums

Home / Forums

You need to log in to create posts and topics. Login · Register

Slow write/read help

Pages: 1 2

bmensah8dgrp
9 Posts
May 28, 2024, 10:07 am

I have a pretty decent setup but am experiencing very poor performance, i think i may have configured ceph wrong. my hardware

management:

  • 3 vms with 4 cpu and 8GB ram
  • 2 x 10GB nics

iscsi target:

  • 2 x hp dl360p gen9
  • 2 x 40GBs nic (for iscsi subnet 1 & 2)
  • 2 x 10GB nic (for mgt and backend)
  • v4  cpu with 32GB DDR4 RAM

 

Storage:

  • 3 x hp dl380 gen 9
  • v4 cpu with 64GB DDR4 RAM
  • 6 x 10TB HGST
  • 4 x 10GB nic (for mgt and backend)
  • 2 x 512 enterprise SSD for Cache

 

Total of ~165TB capacity (replicated max size: 3 min size: 2)

 

Am using this purely for iscsi for vmware transitioning to hyperv, am getting writes of ~15Mbps and reads ~120Mbps.

 

My initial thoughts are am missing journals on the osd nodes and might need to move cache to iscsi service nodes, rebuild the iscsi nodes with 8 x 512GB ssd and change ceph config to create another pool with ssd's just for cache and writeback and flush to osd, rebuild OSD nodes one after the other and use the ssd for journal.

 

am i thinking in the right direction to fix my performance issues? any feedback will be greatly appreciated.

#1

admin
2,930 Posts
May 28, 2024, 10:54 am

Are you using the SSDs for journal or cache ?

How are you testing performance to get the 15-120 MB/s ?

#2

bmensah8dgrp
9 Posts
May 28, 2024, 10:57 am

am using ssd for cache, testing inside a vm using iscsi disk created in petasan. it was the only vm i used AJA System Test.

#3

admin
2,930 Posts
May 28, 2024, 11:14 am

can you give more detail on aja test: is it 4kb or 1MB,  1 thread or many threads ? can you test using crytaldismark on windows or fio on linux.

#4

bmensah8dgrp
9 Posts
May 28, 2024, 11:19 am

running the tests now will upload results shortly, is the hardware setup correct? and what of my next steps?

#5

bmensah8dgrp
9 Posts
May 28, 2024, 11:25 am

------------------------------------------------------------------------------
CrystalDiskMark 8.0.5 x64 (C) 2007-2024 hiyohiyo
Crystal Dew World: https://crystalmark.info/
------------------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

[Read]
SEQ 1MiB (Q= 8, T= 1): 4.404 MB/s [ 4.2 IOPS] <1177741.26 us>
SEQ 1MiB (Q= 1, T= 1): 46.551 MB/s [ 44.4 IOPS] < 22425.59 us>
RND 4KiB (Q= 32, T= 1): 18.621 MB/s [ 4546.1 IOPS] < 6699.99 us>
RND 4KiB (Q= 1, T= 1): 0.140 MB/s [ 34.2 IOPS] < 28591.23 us>

[Write]
SEQ 1MiB (Q= 8, T= 1): 36.067 MB/s [ 34.4 IOPS] <228951.00 us>
SEQ 1MiB (Q= 1, T= 1): 6.080 MB/s [ 5.8 IOPS] <147608.44 us>
RND 4KiB (Q= 32, T= 1): 0.663 MB/s [ 161.9 IOPS] <172718.84 us>
RND 4KiB (Q= 1, T= 1): 0.029 MB/s [ 7.1 IOPS] <137748.20 us>

Profile: Default
Test: 1 GiB (x1) [C: 55% (22/39GiB)]
Mode: [Admin]
Time: Measure 5 sec / Interval 5 sec
Date: 2024/05/28 12:23:10
OS: Windows Server 2022 Standard Evaluation 21H2 [10.0 Build 20348] (x64)

#6

bmensah8dgrp
9 Posts
May 28, 2024, 11:38 am

------------------------------------------------------------------------------
CrystalDiskMark 8.0.5 x64 (C) 2007-2024 hiyohiyo
Crystal Dew World: https://crystalmark.info/
------------------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

[Read]
SEQ 1MiB (Q= 8, T= 1): 1.887 MB/s [ 1.8 IOPS] <2142610.85 us>
SEQ 1MiB (Q= 1, T= 1): 26.426 MB/s [ 25.2 IOPS] < 39488.25 us>
RND 4KiB (Q= 32, T= 1): 2.808 MB/s [ 685.5 IOPS] < 45785.88 us>
RND 4KiB (Q= 1, T= 1): 0.572 MB/s [ 139.6 IOPS] < 7150.30 us>

[Write]
SEQ 1MiB (Q= 8, T= 1): 31.879 MB/s [ 30.4 IOPS] <225507.66 us>
SEQ 1MiB (Q= 1, T= 1): 6.919 MB/s [ 6.6 IOPS] <129213.66 us>
RND 4KiB (Q= 32, T= 1): 0.634 MB/s [ 154.8 IOPS] <189627.63 us>
RND 4KiB (Q= 1, T= 1): 0.096 MB/s [ 23.4 IOPS] < 35284.80 us>

Profile: Default
Test: 4 GiB (x1) [C: 55% (22/39GiB)]
Mode: [Admin]
Time: Measure 5 sec / Interval 5 sec
Date: 2024/05/28 12:37:24
OS: Windows Server 2022 Standard Evaluation 21H2 [10.0 Build 20348] (x64)

#7

admin
2,930 Posts
May 28, 2024, 12:00 pm

1) Can you reconfigure and use SSD as journals instead of cache. Assuming you can delete data, delete pool, stop the OSD on each node via

systemctl stop ceph-osd.target

then from ui delete the OSD and cache disk, then recreate journal and OSDs.

2) Are you using vmware or hyperv in your test ? Can you test from baremetal Windows >

 

#8

bmensah8dgrp
9 Posts
May 28, 2024, 12:03 pm
  1. yes i can, we are currently testing so no production data yet.
  2. vmware, yes i can test baremetal (will post results shortly)
#9

bmensah8dgrp
9 Posts
May 28, 2024, 12:23 pm

------------------------------------------------------------------------------
CrystalDiskMark 8.0.5 x64 (C) 2007-2024 hiyohiyo
Crystal Dew World: https://crystalmark.info/
------------------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

[Read]
SEQ 1MiB (Q= 8, T= 1): 211.790 MB/s [ 202.0 IOPS] < 34539.67 us>
SEQ 1MiB (Q= 1, T= 1): 42.357 MB/s [ 40.4 IOPS] < 24266.87 us>
RND 4KiB (Q= 32, T= 1): 7.642 MB/s [ 1865.7 IOPS] < 13612.50 us>
RND 4KiB (Q= 1, T= 1): 8.146 MB/s [ 1988.8 IOPS] < 501.08 us>

[Write]
SEQ 1MiB (Q= 8, T= 1): 39.640 MB/s [ 37.8 IOPS] <196714.19 us>
SEQ 1MiB (Q= 1, T= 1): 11.322 MB/s [ 10.8 IOPS] < 92405.17 us>
RND 4KiB (Q= 32, T= 1): 0.618 MB/s [ 150.9 IOPS] <192261.38 us>
RND 4KiB (Q= 1, T= 1): 0.112 MB/s [ 27.3 IOPS] < 34243.70 us>

Profile: Default
Test: 1 GiB (x1)
Mode: [Admin]
Time: Measure 5 sec / Interval 5 sec
Date: 2024/05/28 13:13:16
OS: Windows Server 2022 Datacenter 21H2 [10.0 Build 20348] (x64)

 

 

------------------------------------------------------------------------------
CrystalDiskMark 8.0.5 x64 (C) 2007-2024 hiyohiyo
Crystal Dew World: https://crystalmark.info/
------------------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

[Read]
SEQ 1MiB (Q= 8, T= 1): 103.591 MB/s [ 98.8 IOPS] < 74943.82 us>
SEQ 1MiB (Q= 1, T= 1): 81.149 MB/s [ 77.4 IOPS] < 12914.89 us>
RND 4KiB (Q= 32, T= 1): 9.650 MB/s [ 2356.0 IOPS] < 13426.63 us>
RND 4KiB (Q= 1, T= 1): 8.543 MB/s [ 2085.7 IOPS] < 478.27 us>

[Write]
SEQ 1MiB (Q= 8, T= 1): 27.894 MB/s [ 26.6 IOPS] <273407.17 us>
SEQ 1MiB (Q= 1, T= 1): 4.823 MB/s [ 4.6 IOPS] <216563.23 us>
RND 4KiB (Q= 32, T= 1): 0.950 MB/s [ 231.9 IOPS] <113346.44 us>
RND 4KiB (Q= 1, T= 1): 0.144 MB/s [ 35.2 IOPS] < 28348.43 us>

Profile: Default
Test: 4 GiB (x1)
Mode: [Admin]
Time: Measure 5 sec / Interval 5 sec
Date: 2024/05/28 13:24:19
OS: Windows Server 2022 Datacenter 21H2 [10.0 Build 20348] (x64)

#10
Pages: 1 2