Forums

Home / Forums

You need to log in to create posts and topics. Login · Register

Storage calculation


killerodin
33 Posts
September 3, 2020, 1:52 pm

Can somebody please explain me the storage calculation? No matter how I calculate it, I cannot get to the values that are shown to me in the dashboard. Here our configuration

We have 14 Storage Nodes with 2 OSD each and 2.73 TB per OSD. One of the Nodes is deliberately offline. In summary we should have 70.94 TB online, right? That's also shown in the dashboard unter Storage. That makes sense to me.
Now we have split the 14 Nodes into 2 Pools. 7 per each pool. Let's named them "Pool A" and "Pool B".
The offline node is in Pool A. So there I have 12 OSD's should be in summary 32.76 TB. In Pool B I have 14 OSD's should be in summary 38.22 TB.
Up to here, everything is clear.

Now at the dashboard I look at the View Chart "Cluster Storage". Then I get the following values:

"Pool A"
Available Current: 1.78 TB
Used Current: 23.05 TB
"Pool B"
Available Current: 5.48 TB
Used Current: 21.53 TB
All
Available Current: 33.30 TB
Used Current: 44.60 TB
Total Current: 78.00 TB

How exactly do the values come about? No matter how I calculate, I cannot understand the values.

Thank you for the Explanation

#1

admin
2,930 Posts
September 3, 2020, 4:31 pm

What version do you use ?

what is your workload: iSCSI / CIFS/NFS..direct Ceph ?

What pool replication size or EC profile do you use

what is the output of

ceph df detail

Last edited on September 3, 2020, 4:34 pm by admin · #2

killerodin
33 Posts
September 4, 2020, 7:05 am

We use Version 2.6.0

Actually we have a pool replication size of 3 min 2 and only a workload over iSCSI

ceph df detail:

RAW STORAGE:
CLASS SIZE AVAIL USED RAW USED %RAW USED
hdd 71 TiB 30 TiB 41 TiB 41 TiB 57.30
TOTAL 71 TiB 30 TiB 41 TiB 41 TiB 57.30

POOLS:
POOL ID STORED OBJECTS USED %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR
KXCHMUE-700A 5 7.0 TiB 1.94M 21 TiB 81.18 1.6 TiB N/A N/A 1.94M 0 B 0 B
KXCHMUE-700B 6 6.5 TiB 1.81M 20 TiB 56.78 5.0 TiB N/A N/A 1.81M 0 B 0 B

#3

admin
2,930 Posts
September 4, 2020, 10:56 am

The charts match the Ceph output.

Raw usage
you used 41T from 71T, you have 30TB left. raw is a total value that does not take care about the individual pool replication and any inbalance of osds used in such pools.

Pool usage:
Pool A and Pool B USED is 21T+ 20T matching the raw used.
The STORED values are 1/3 of the USED values due to x3 replicas
The OBJECTS are the STORED value / 4M the object size of rbd
the tricky part is MAX AVAIL: Pool B you can safely write 5TB, Pool A only 1.6: These figures are calculated estimates they take into account the max filled OSD which may not be the same across your OSDs due to crush distribution inbalance.
you can view your OSD storage distribution (something we are adding in 2.7) via

ceph osd df

in case of large inbalance you can switch the balancer on

ceph balancer on
ceph balancer status

for further  explanation see

http://www.petasan.org/forums/?view=thread&id=586

http://www.petasan.org/forums/?view=thread&id=697

 

Last edited on September 4, 2020, 11:13 am by admin · #4

killerodin
33 Posts
September 4, 2020, 12:51 pm

Ok, thanks a lot. Now it's clearer for me.

I see a large inbalance on Pool A.

ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS
0 hdd 2.72800 1.00000 2.7 TiB 1.7 TiB 1.7 TiB 108 KiB 2.9 GiB 1.1 TiB 60.69 1.06 70 up
1 hdd 2.72800 1.00000 2.7 TiB 2.0 TiB 2.0 TiB 104 KiB 3.4 GiB 752 GiB 73.10 1.28 87 up
2 hdd 2.72800 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down
3 hdd 2.72800 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down
4 hdd 2.72800 1.00000 2.7 TiB 2.2 TiB 2.2 TiB 180 KiB 3.7 GiB 495 GiB 82.28 1.44 97 up
5 hdd 2.72800 1.00000 2.7 TiB 1.7 TiB 1.7 TiB 92 KiB 3.0 GiB 1002 GiB 64.12 1.12 75 up
6 hdd 2.72800 1.00000 2.7 TiB 1.5 TiB 1.4 TiB 104 KiB 2.6 GiB 1.3 TiB 53.23 0.93 62 up
7 hdd 2.72800 1.00000 2.7 TiB 1.8 TiB 1.8 TiB 32 KiB 3.3 GiB 928 GiB 66.78 1.17 81 up
8 hdd 2.72800 1.00000 2.7 TiB 1.6 TiB 1.6 TiB 152 KiB 2.8 GiB 1.1 TiB 58.27 1.02 69 up
9 hdd 2.72800 1.00000 2.7 TiB 1.4 TiB 1.4 TiB 112 KiB 2.6 GiB 1.3 TiB 52.06 0.91 60 up
10 hdd 2.72800 1.00000 2.7 TiB 2.2 TiB 2.2 TiB 132 KiB 3.7 GiB 581 GiB 79.22 1.38 90 up
11 hdd 2.72800 1.00000 2.7 TiB 1.5 TiB 1.5 TiB 132 KiB 2.7 GiB 1.2 TiB 56.56 0.99 67 up
12 hdd 2.72800 1.00000 2.7 TiB 1.7 TiB 1.7 TiB 68 KiB 3.0 GiB 1.1 TiB 60.63 1.06 68 up
13 hdd 2.72800 1.00000 2.7 TiB 1.7 TiB 1.7 TiB 260 KiB 3.0 GiB 1.0 TiB 63.10 1.10 74 up
14 hdd 2.72800 1.00000 2.7 TiB 1.4 TiB 1.4 TiB 92 KiB 2.7 GiB 1.4 TiB 50.19 0.88 294 up
15 hdd 2.72800 1.00000 2.7 TiB 1.3 TiB 1.3 TiB 120 KiB 2.6 GiB 1.4 TiB 47.52 0.83 286 up
16 hdd 2.72800 1.00000 2.7 TiB 1.4 TiB 1.4 TiB 280 KiB 2.6 GiB 1.4 TiB 50.52 0.88 297 up
17 hdd 2.72800 1.00000 2.7 TiB 1.4 TiB 1.4 TiB 156 KiB 2.8 GiB 1.3 TiB 52.52 0.92 310 up
18 hdd 2.72800 1.00000 2.7 TiB 1.5 TiB 1.5 TiB 164 KiB 2.9 GiB 1.2 TiB 55.95 0.98 324 up
19 hdd 2.72800 1.00000 2.7 TiB 1.3 TiB 1.3 TiB 68 KiB 2.8 GiB 1.4 TiB 48.07 0.84 279 up
20 hdd 2.72800 1.00000 2.7 TiB 1.4 TiB 1.4 TiB 168 KiB 2.7 GiB 1.3 TiB 51.47 0.90 296 up
21 hdd 2.72800 1.00000 2.7 TiB 1.4 TiB 1.4 TiB 128 KiB 3.3 GiB 1.3 TiB 52.33 0.91 310 up
22 hdd 2.72800 1.00000 2.7 TiB 1.4 TiB 1.4 TiB 160 KiB 2.7 GiB 1.3 TiB 51.67 0.90 301 up
23 hdd 2.72800 1.00000 2.7 TiB 1.5 TiB 1.5 TiB 92 KiB 2.8 GiB 1.3 TiB 54.01 0.94 310 up
24 hdd 2.72800 1.00000 2.7 TiB 1.3 TiB 1.3 TiB 200 KiB 2.6 GiB 1.4 TiB 48.66 0.85 281 up
25 hdd 2.72800 1.00000 2.7 TiB 1.4 TiB 1.4 TiB 60 KiB 2.9 GiB 1.4 TiB 50.02 0.87 288 up
26 hdd 2.72800 1.00000 2.7 TiB 1.5 TiB 1.5 TiB 56 KiB 2.9 GiB 1.2 TiB 55.24 0.96 317 up
27 hdd 2.72800 1.00000 2.7 TiB 1.4 TiB 1.4 TiB 136 KiB 2.7 GiB 1.3 TiB 51.64 0.90 307 up
TOTAL 71 TiB 41 TiB 41 TiB 3.3 MiB 75 GiB 30 TiB 57.30
MIN/MAX VAR: 0.83/1.44 STDDEV: 9.11

 

In the longer term we will probably add more OSD's. In the short term the balancer will probably have to suffice.
Thanks a lot

 

 

#5

killerodin
33 Posts
September 7, 2020, 6:49 am

Hey

What's the reason if nothing happens when I start the balancer?

here is the output from ceph balancer status

{
"last_optimize_duration": "0:00:00.000621",
"plans": [],
"mode": "none",
"active": true,
"optimize_result": "Please do \"ceph balancer mode\" to choose a valid mode first",
"last_optimize_started": "Mon Sep 7 08:43:46 2020"
}

But ceph osd df is the same as last week...

ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS
0 hdd 2.72800 1.00000 2.7 TiB 1.7 TiB 1.7 TiB 56 KiB 3.0 GiB 1.1 TiB 60.73 1.01 70 up
1 hdd 2.72800 1.00000 2.7 TiB 2.0 TiB 2.0 TiB 200 KiB 3.3 GiB 751 GiB 73.13 1.22 87 up
2 hdd 2.72800 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down
3 hdd 2.72800 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down
4 hdd 2.72800 1.00000 2.7 TiB 2.2 TiB 2.2 TiB 72 KiB 4.0 GiB 494 GiB 82.32 1.37 97 up
5 hdd 2.72800 1.00000 2.7 TiB 1.8 TiB 1.7 TiB 128 KiB 3.0 GiB 1002 GiB 64.15 1.07 75 up
6 hdd 2.72800 1.00000 2.7 TiB 1.5 TiB 1.5 TiB 68 KiB 2.9 GiB 1.3 TiB 53.26 0.89 62 up
7 hdd 2.72800 1.00000 2.7 TiB 1.8 TiB 1.8 TiB 76 KiB 3.3 GiB 927 GiB 66.81 1.11 81 up
8 hdd 2.72800 1.00000 2.7 TiB 1.6 TiB 1.6 TiB 256 KiB 2.8 GiB 1.1 TiB 58.29 0.97 69 up
9 hdd 2.72800 1.00000 2.7 TiB 1.4 TiB 1.4 TiB 40 KiB 2.6 GiB 1.3 TiB 52.08 0.87 60 up
10 hdd 2.72800 1.00000 2.7 TiB 2.2 TiB 2.2 TiB 80 KiB 3.6 GiB 580 GiB 79.25 1.32 90 up
11 hdd 2.72800 1.00000 2.7 TiB 1.5 TiB 1.5 TiB 72 KiB 2.7 GiB 1.2 TiB 56.58 0.94 67 up
12 hdd 2.72800 1.00000 2.7 TiB 1.7 TiB 1.7 TiB 164 KiB 3.1 GiB 1.1 TiB 60.66 1.01 68 up
13 hdd 2.72800 1.00000 2.7 TiB 1.7 TiB 1.7 TiB 112 KiB 3.0 GiB 1.0 TiB 63.13 1.05 74 up
14 hdd 2.72800 1.00000 2.7 TiB 1.5 TiB 1.5 TiB 160 KiB 2.8 GiB 1.2 TiB 54.94 0.92 294 up
15 hdd 2.72800 1.00000 2.7 TiB 1.4 TiB 1.4 TiB 88 KiB 2.7 GiB 1.3 TiB 52.01 0.87 286 up
16 hdd 2.72800 1.00000 2.7 TiB 1.5 TiB 1.5 TiB 108 KiB 2.8 GiB 1.2 TiB 55.33 0.92 297 up
17 hdd 2.72800 1.00000 2.7 TiB 1.6 TiB 1.6 TiB 136 KiB 2.9 GiB 1.2 TiB 57.50 0.96 310 up
18 hdd 2.72800 1.00000 2.7 TiB 1.7 TiB 1.7 TiB 68 KiB 3.3 GiB 1.1 TiB 61.24 1.02 324 up
19 hdd 2.72800 1.00000 2.7 TiB 1.4 TiB 1.4 TiB 200 KiB 2.7 GiB 1.3 TiB 52.60 0.88 279 up
20 hdd 2.72800 1.00000 2.7 TiB 1.5 TiB 1.5 TiB 80 KiB 2.8 GiB 1.2 TiB 56.31 0.94 296 up
21 hdd 2.72800 1.00000 2.7 TiB 1.6 TiB 1.6 TiB 244 KiB 2.9 GiB 1.2 TiB 57.27 0.96 310 up
22 hdd 2.72800 1.00000 2.7 TiB 1.5 TiB 1.5 TiB 20 KiB 3.5 GiB 1.2 TiB 56.61 0.94 301 up
23 hdd 2.72800 1.00000 2.7 TiB 1.6 TiB 1.6 TiB 64 KiB 3.0 GiB 1.1 TiB 59.13 0.99 310 up
24 hdd 2.72800 1.00000 2.7 TiB 1.5 TiB 1.5 TiB 160 KiB 2.7 GiB 1.3 TiB 53.26 0.89 281 up
25 hdd 2.72800 1.00000 2.7 TiB 1.5 TiB 1.5 TiB 20 KiB 3.0 GiB 1.2 TiB 54.76 0.91 288 up
26 hdd 2.72800 1.00000 2.7 TiB 1.6 TiB 1.6 TiB 104 KiB 3.2 GiB 1.1 TiB 60.46 1.01 317 up
27 hdd 2.72800 1.00000 2.7 TiB 1.5 TiB 1.5 TiB 260 KiB 2.9 GiB 1.2 TiB 56.54 0.94 307 up
TOTAL 71 TiB 43 TiB 42 TiB 3.0 MiB 79 GiB 28 TiB 59.94
MIN/MAX VAR: 0.87/1.37 STDDEV: 7.67

#6

therm
121 Posts
September 7, 2020, 7:26 am

Hi,

'ceph balancer status' tolds you:

"optimize_result": "Please do \"ceph balancer mode\" to choose a valid mode first",

So regarding

https://docs.ceph.com/docs/mimic/mgr/balancer/

you have to set

ceph balancer mode upmap (the new mode)

or 

ceph balancer mode crush-compat (the old one)

If the balancing does not work for you (as for our cluster with multiple and hybrid pools) there is an alternative by using
'reweight-by-utilization' for example this way:

/usr/bin/ceph osd reweight-by-utilization 110 0.05 5 --no-increasing

This has to be run manually several times to get a good distribution.
#7