Forums

Home / Forums

You need to log in to create posts and topics. Login · Register

Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull


killerodin
33 Posts
October 5, 2022, 4:25 pm

Hello

We had some memory issues and added 9 new OSD's to the cluster. But now we got the following messages:

 




 
The 900 PG's are all from the pool. Does the problem solve itself or do we have to do something actively here?
The misplaced objects are going down steadily (we can see this in the monitoring).

Thanks a lot

#1

admin
2,930 Posts
October 6, 2022, 9:32 am

If your OSDs are nearing full, the best thing is to add new storage disks. Else the issue can resolve by itself in some cases. What is the output of

ceph osd df

#2

killerodin
33 Posts
October 6, 2022, 11:03 am

The output ist

ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS
15 hdd 16.42999 1.00000 16 TiB 9.3 TiB 9.2 TiB 19 KiB 15 GiB 7.1 TiB 56.63 1.03 411 up
16 hdd 16.42999 1.00000 16 TiB 8.9 TiB 8.8 TiB 1.8 MiB 14 GiB 7.6 TiB 53.97 0.99 393 up
17 hdd 16.42999 1.00000 16 TiB 9.1 TiB 9.0 TiB 2.6 MiB 15 GiB 7.3 TiB 55.35 1.01 401 up
18 hdd 16.42999 1.00000 16 TiB 9.2 TiB 9.1 TiB 3.7 MiB 14 GiB 7.3 TiB 55.82 1.02 403 up
19 hdd 16.42999 1.00000 16 TiB 9.1 TiB 9.1 TiB 523 KiB 14 GiB 7.3 TiB 55.63 1.02 404 up
20 hdd 16.42999 1.00000 16 TiB 8.6 TiB 8.5 TiB 1.6 MiB 14 GiB 7.8 TiB 52.31 0.96 381 up
21 hdd 16.42999 1.00000 16 TiB 8.9 TiB 8.9 TiB 581 KiB 14 GiB 7.5 TiB 54.31 0.99 395 up
22 hdd 16.42999 1.00000 16 TiB 8.6 TiB 8.5 TiB 538 KiB 14 GiB 7.8 TiB 52.28 0.96 380 up
0 ssd 3.49300 1.00000 3.5 TiB 2.3 TiB 2.2 TiB 89 KiB 7.4 GiB 1.2 TiB 64.57 1.18 62 up
1 ssd 3.49300 1.00000 3.5 TiB 2.2 TiB 2.1 TiB 84 MiB 7.0 GiB 1.3 TiB 61.74 1.13 60 up
2 ssd 3.49300 1.00000 3.5 TiB 2.3 TiB 2.3 TiB 2.9 MiB 7.0 GiB 1.2 TiB 64.76 1.18 62 up
9 ssd 3.49300 1.00000 3.5 TiB 2.5 TiB 2.5 TiB 1.9 MiB 6.9 GiB 982 GiB 72.56 1.33 71 up
12 ssd 3.49300 1.00000 3.5 TiB 3.0 TiB 3.0 TiB 0 B 7.6 GiB 531 GiB 85.17 1.56 76 up
23 ssd 3.49300 1.00000 3.5 TiB 2.5 TiB 2.5 TiB 2.2 MiB 6.6 GiB 976 GiB 72.71 1.33 69 up
26 ssd 3.49300 1.00000 3.5 TiB 2.2 TiB 2.2 TiB 8.9 MiB 6.6 GiB 1.3 TiB 63.47 1.16 63 up
28 ssd 3.49199 1.00000 3.5 TiB 2.3 TiB 2.3 TiB 4.6 MiB 6.1 GiB 1.2 TiB 65.50 1.20 63 up
29 ssd 3.49199 1.00000 3.5 TiB 2.8 TiB 2.8 TiB 2.9 MiB 6.6 GiB 688 GiB 80.76 1.48 78 up
30 ssd 3.49199 1.00000 3.5 TiB 2.4 TiB 2.4 TiB 2.1 MiB 6.4 GiB 1.1 TiB 68.14 1.24 66 up
37 ssd 3.49199 1.00000 3.5 TiB 2.7 TiB 2.7 TiB 1000 KiB 6.8 GiB 779 GiB 78.23 1.43 73 up
41 ssd 6.98599 1.00000 7.0 TiB 1017 GiB 1015 GiB 248 KiB 2.3 GiB 6.0 TiB 14.22 0.26 31 up
3 ssd 3.49300 1.00000 3.5 TiB 2.8 TiB 2.8 TiB 0 B 7.7 GiB 719 GiB 79.89 1.46 73 up
4 ssd 3.49300 1.00000 3.5 TiB 2.3 TiB 2.3 TiB 2.4 MiB 7.2 GiB 1.1 TiB 67.19 1.23 67 up
5 ssd 3.49300 1.00000 3.5 TiB 2.5 TiB 2.5 TiB 970 KiB 7.4 GiB 1.0 TiB 70.68 1.29 67 up
10 ssd 3.49300 1.00000 3.5 TiB 2.5 TiB 2.5 TiB 0 B 7.1 GiB 1.0 TiB 70.81 1.29 68 up
13 ssd 3.49300 1.00000 3.5 TiB 2.3 TiB 2.3 TiB 202 KiB 6.9 GiB 1.2 TiB 66.24 1.21 64 up
24 ssd 3.49300 1.00000 3.5 TiB 2.7 TiB 2.7 TiB 1.7 MiB 6.8 GiB 846 GiB 76.36 1.39 77 up
27 ssd 3.49300 1.00000 3.5 TiB 2.3 TiB 2.3 TiB 3.5 MiB 6.4 GiB 1.2 TiB 66.38 1.21 66 up
31 ssd 3.49199 1.00000 3.5 TiB 2.3 TiB 2.3 TiB 742 KiB 6.3 GiB 1.2 TiB 65.44 1.20 64 up
32 ssd 3.49199 1.00000 3.5 TiB 2.1 TiB 2.1 TiB 578 KiB 6.0 GiB 1.4 TiB 60.99 1.11 59 up
33 ssd 3.49199 1.00000 3.5 TiB 2.3 TiB 2.3 TiB 82 MiB 6.4 GiB 1.2 TiB 65.60 1.20 67 up
38 ssd 3.49199 1.00000 3.5 TiB 2.2 TiB 2.2 TiB 979 KiB 6.5 GiB 1.3 TiB 61.87 1.13 62 up
42 ssd 6.98599 1.00000 7.0 TiB 1.7 TiB 1.6 TiB 524 KiB 3.5 GiB 5.3 TiB 23.66 0.43 45 up
6 ssd 3.49300 1.00000 3.5 TiB 2.4 TiB 2.4 TiB 2.0 MiB 7.0 GiB 1.1 TiB 67.91 1.24 65 up
7 ssd 3.49300 1.00000 3.5 TiB 2.3 TiB 2.3 TiB 69 MiB 7.3 GiB 1.2 TiB 65.53 1.20 63 up
8 ssd 3.49300 1.00000 3.5 TiB 2.3 TiB 2.3 TiB 0 B 7.0 GiB 1.2 TiB 65.34 1.19 64 up
11 ssd 3.49300 1.00000 3.5 TiB 2.2 TiB 2.2 TiB 0 B 6.8 GiB 1.3 TiB 63.15 1.15 61 up
14 ssd 3.49300 1.00000 3.5 TiB 2.3 TiB 2.3 TiB 95 KiB 7.5 GiB 1.2 TiB 66.44 1.21 63 up
25 ssd 3.49300 1.00000 3.5 TiB 2.1 TiB 2.1 TiB 276 KiB 6.4 GiB 1.4 TiB 60.85 1.11 62 up
34 ssd 3.49199 1.00000 3.5 TiB 2.3 TiB 2.3 TiB 3.5 MiB 6.4 GiB 1.2 TiB 65.32 1.19 66 up
35 ssd 3.49199 1.00000 3.5 TiB 2.1 TiB 2.1 TiB 1.3 MiB 6.1 GiB 1.4 TiB 60.99 1.11 64 up
36 ssd 3.49199 1.00000 3.5 TiB 2.6 TiB 2.6 TiB 1002 KiB 6.4 GiB 944 GiB 73.60 1.34 72 up
39 ssd 3.49199 1.00000 3.5 TiB 2.6 TiB 2.5 TiB 2.8 MiB 7.0 GiB 960 GiB 73.15 1.34 67 up
40 ssd 3.49300 1.00000 3.5 TiB 2.4 TiB 2.3 TiB 3.2 MiB 6.4 GiB 1.1 TiB 67.32 1.23 61 up
43 ssd 6.98599 1.00000 7.0 TiB 3.3 TiB 3.2 TiB 88 MiB 6.6 GiB 3.7 TiB 46.56 0.85 83 up
44 ssd 6.98599 1.00000 7.0 TiB 2.6 TiB 2.6 TiB 5.0 MiB 5.0 GiB 4.4 TiB 37.08 0.68 70 up
45 ssd 6.98599 1.00000 7.0 TiB 2.2 TiB 2.2 TiB 1.3 MiB 4.2 GiB 4.7 TiB 32.16 0.59 61 up
46 ssd 6.98599 1.00000 7.0 TiB 2.4 TiB 2.4 TiB 805 KiB 4.8 GiB 4.6 TiB 34.70 0.63 71 up
47 ssd 6.98599 1.00000 7.0 TiB 2.3 TiB 2.3 TiB 1.9 MiB 4.3 GiB 4.7 TiB 33.10 0.60 65 up
48 ssd 6.98599 1.00000 7.0 TiB 2.1 TiB 2.1 TiB 2.9 MiB 3.9 GiB 4.9 TiB 29.59 0.54 55 up
49 ssd 6.98630 1.00000 7.0 TiB 1.4 TiB 1.4 TiB 4.3 MiB 2.8 GiB 5.6 TiB 19.36 0.35 37 up
TOTAL 310 TiB 169 TiB 169 TiB 404 MiB 376 GiB 140 TiB 54.74

Then it should resolve by itself. I was just worried about the"1 pg backfill_toofull" message. We have never had that when we have integrated new OSD's. However, we have also never integrated so many new ones at once.

 

Thanks a lot

#3