Forums

Home / Forums

You need to log in to create posts and topics. Login · Register

Network folder full

Pages: 1 2

Frankie
6 Posts
November 24, 2021, 3:14 pm

Hi,

I have a cluster of 3 nodes each node has 2 osd. For a fault the third node is turned off. the network share is 4tb, I deleted a very heavy folder (almost 1tb to make space) but I do not synchronize the cancellation. The share is full.

p-001 osd 3 83% osd 4 74%
p-002 osd 5 76% osd 6 77%

ceph health

HEALTH_WARN 1 nearfull osd(s); 3 pool(s) nearfull; Degraded data redundancy: 7123573/21370719 objects degraded (33.333%), 320 pgs degraded, 320 pgs undersized; 320 pgs not deep-scrubbed in time;

320 pgs not scrubbed in time; 1/3 mons down, quorum p-001,p-002

 

 

PG_DEGRADED Degraded data redundancy: 7123583/21370749 objects degraded (33.333%), 320 pgs degraded, 320 pgs undersized
pg 1.39 is stuck undersized for 436071.924248, current state active+undersized+degraded, last acting [3,4]
pg 1.3a is stuck undersized for 436235.057015, current state active+undersized+degraded, last acting [4,3]
pg 1.3b is stuck undersized for 436235.064191, current state active+undersized+degraded, last acting [4,2]
pg 1.3c is stuck undersized for 436071.912444, current state active+undersized+degraded, last acting [2,4]
pg 1.3d is stuck undersized for 436065.221494, current state active+undersized+degraded, last acting [3,5]
pg 1.3e is stuck undersized for 436228.395122, current state active+undersized+degraded, last acting [5,2]
pg 1.3f is stuck undersized for 436065.220457, current state active+undersized+degraded, last acting [3,5]
pg 1.40 is stuck undersized for 436228.380327, current state active+undersized+degraded, last acting [5,2]
pg 1.41 is stuck undersized for 436071.928088, current state active+undersized+degraded, last acting [3,4]
pg 1.42 is stuck undersized for 436235.064132, current state active+undersized+degraded, last acting [4,3]
pg 1.43 is stuck undersized for 436071.910839, current state active+undersized+degraded, last acting [2,4]
pg 1.44 is stuck undersized for 436228.387157, current state active+undersized+degraded, last acting [5,3]
pg 1.45 is stuck undersized for 436235.060483, current state active+undersized+degraded, last acting [4,3]
pg 1.46 is stuck undersized for 436065.224701, current state active+undersized+degraded, last acting [2,5]
pg 1.47 is stuck undersized for 436235.068591, current state active+undersized+degraded, last acting [4,2]
pg 1.48 is stuck undersized for 436065.214149, current state active+undersized+degraded, last acting [3,5]
pg 1.49 is stuck undersized for 436071.929893, current state active+undersized+degraded, last acting [3,4]
pg 1.4a is stuck undersized for 436071.913020, current state active+undersized+degraded, last acting [2,4]
pg 1.4b is stuck undersized for 436235.073491, current state active+undersized+degraded, last acting [4,2]
pg 1.4c is stuck undersized for 436065.228279, current state active+undersized+degraded, last acting [2,5]
pg 1.4d is stuck undersized for 436228.389386, current state active+undersized+degraded, last acting [5,3]
pg 2.38 is stuck undersized for 436228.387091, current state active+undersized+degraded, last acting [5,2]
pg 2.39 is stuck undersized for 436228.384080, current state active+undersized+degraded, last acting [5,3]
pg 2.3a is stuck undersized for 436228.386575, current state active+undersized+degraded, last acting [5,3]
pg 2.3b is active+undersized+degraded, acting [2,5]
pg 2.3c is stuck undersized for 436071.910915, current state active+undersized+degraded, last acting [2,4]
pg 2.3d is stuck undersized for 436228.381472, current state active+undersized+degraded, last acting [5,2]
pg 2.3e is stuck undersized for 436235.061295, current state active+undersized+degraded, last acting [4,3]
pg 2.3f is stuck undersized for 436071.926897, current state active+undersized+degraded, last acting [3,4]
pg 2.40 is stuck undersized for 436065.222724, current state active+undersized+degraded, last acting [3,5]
pg 2.41 is stuck undersized for 436235.053420, current state active+undersized+degraded, last acting [4,2]
pg 2.42 is stuck undersized for 436228.382724, current state active+undersized+degraded, last acting [5,3]
pg 2.43 is stuck undersized for 436065.215023, current state active+undersized+degraded, last acting [3,5]
pg 2.44 is stuck undersized for 436228.389498, current state active+undersized+degraded, last acting [5,2]
pg 2.45 is stuck undersized for 436235.048049, current state active+undersized+degraded, last acting [4,2]
pg 2.46 is stuck undersized for 436071.910351, current state active+undersized+degraded, last acting [2,4]
pg 2.47 is stuck undersized for 436228.383950, current state active+undersized+degraded, last acting [5,2]
pg 2.48 is stuck undersized for 436235.071995, current state active+undersized+degraded, last acting [4,3]
pg 2.49 is stuck undersized for 436065.225970, current state active+undersized+degraded, last acting [2,5]
pg 2.4a is stuck undersized for 436235.047426, current state active+undersized+degraded, last acting [4,2]
pg 2.4b is stuck undersized for 436065.227879, current state active+undersized+degraded, last acting [2,5]
pg 2.4e is stuck undersized for 436065.224521, current state active+undersized+degraded, last acting [2,5]
pg 2.4f is stuck undersized for 436065.223068, current state active+undersized+degraded, last acting [3,5]
pg 3.38 is stuck undersized for 436228.389728, current state active+undersized+degraded, last acting [5,2]
pg 3.39 is stuck undersized for 436071.925872, current state active+undersized+degraded, last acting [3,4]
pg 3.3a is stuck undersized for 436071.912294, current state active+undersized+degraded, last acting [2,4]
pg 3.3b is stuck undersized for 436235.044281, current state active+undersized+degraded, last acting [4,2]
pg 3.3c is stuck undersized for 436071.911648, current state active+undersized+degraded, last acting [2,4]
pg 3.3d is stuck undersized for 436235.075065, current state active+undersized+degraded, last acting [4,3]
pg 3.3e is stuck undersized for 436235.068948, current state active+undersized+degraded, last acting [4,2]
pg 3.3f is stuck undersized for 436235.066321, current state active+undersized+degraded, last acting [4,3]
PG_NOT_DEEP_SCRUBBED 320 pgs not deep-scrubbed in time
pg 2.4e not deep-scrubbed since 2021-10-23 04:08:37.459247
pg 1.4d not deep-scrubbed since 2021-10-14 04:03:16.493711
pg 2.4f not deep-scrubbed since 2021-10-19 16:00:28.298636
pg 1.4c not deep-scrubbed since 2021-10-23 05:06:26.635988
pg 2.48 not deep-scrubbed since 2021-10-20 15:21:00.489987
pg 1.4b not deep-scrubbed since 2021-10-21 00:32:31.476136
pg 2.49 not deep-scrubbed since 2021-10-21 15:06:03.567184
pg 1.4a not deep-scrubbed since 2021-10-25 10:12:18.818619
pg 2.4a not deep-scrubbed since 2021-10-21 22:18:01.775421
pg 1.49 not deep-scrubbed since 2021-10-26 00:40:04.835318
pg 2.4b not deep-scrubbed since 2021-10-24 09:16:15.033388
pg 1.48 not deep-scrubbed since 2021-10-24 09:16:16.683504
pg 2.44 not deep-scrubbed since 2021-10-22 06:13:32.823312
pg 1.47 not deep-scrubbed since 2021-10-23 11:40:55.090946
pg 2.45 not deep-scrubbed since 2021-10-20 08:38:47.050011
pg 1.46 not deep-scrubbed since 2021-10-22 09:40:31.009867
pg 2.46 not deep-scrubbed since 2021-10-22 08:29:00.360555
pg 1.45 not deep-scrubbed since 2021-10-20 06:14:21.559948
pg 2.47 not deep-scrubbed since 2021-10-25 21:42:08.620106
pg 1.44 not deep-scrubbed since 2021-10-26 09:43:13.469079
pg 2.40 not deep-scrubbed since 2021-10-13 22:30:10.936415
pg 1.43 not deep-scrubbed since 2021-10-20 05:06:07.456276
pg 2.41 not deep-scrubbed since 2021-10-25 12:58:16.095429
pg 1.42 not deep-scrubbed since 2021-10-26 03:47:35.750935
pg 2.42 not deep-scrubbed since 2021-10-21 07:23:38.759214
pg 1.41 not deep-scrubbed since 2021-10-26 08:11:07.359517
pg 2.43 not deep-scrubbed since 2021-10-18 23:26:00.875389
pg 1.40 not deep-scrubbed since 2021-10-26 06:50:13.906860
pg 3.3d not deep-scrubbed since 2021-10-26 08:17:20.032897
pg 2.3c not deep-scrubbed since 2021-10-22 12:49:23.070474
pg 1.3f not deep-scrubbed since 2021-10-15 11:00:02.839382
pg 3.3c not deep-scrubbed since 2021-10-16 06:52:53.281347
pg 2.3d not deep-scrubbed since 2021-10-22 23:24:25.938902
pg 1.3e not deep-scrubbed since 2021-10-24 00:23:49.326500
pg 3.3f not deep-scrubbed since 2021-10-21 08:39:31.130110
pg 2.3e not deep-scrubbed since 2021-10-19 13:41:47.476872
pg 1.3d not deep-scrubbed since 2021-10-18 13:28:08.018827
pg 3.3e not deep-scrubbed since 2021-10-24 07:01:14.880346
pg 2.3f not deep-scrubbed since 2021-10-23 12:48:45.693678
pg 1.3c not deep-scrubbed since 2021-10-22 05:06:23.468608
pg 3.39 not deep-scrubbed since 2021-10-20 23:25:32.265623
pg 2.38 not deep-scrubbed since 2021-10-26 08:12:52.041170
pg 1.3b not deep-scrubbed since 2021-10-20 17:21:51.110163
pg 3.38 not deep-scrubbed since 2021-10-19 16:05:37.158553
pg 2.39 not deep-scrubbed since 2021-10-19 22:12:13.829894
pg 1.3a not deep-scrubbed since 2021-10-17 07:15:00.607797
pg 3.3b not deep-scrubbed since 2021-10-14 01:57:48.032767
pg 2.3a not deep-scrubbed since 2021-10-15 01:29:56.251685
pg 1.39 not deep-scrubbed since 2021-10-20 23:20:27.770843
pg 3.3a not deep-scrubbed since 2021-10-23 04:13:33.023764
270 more pgs...
PG_NOT_SCRUBBED 320 pgs not scrubbed in time
pg 2.4e not scrubbed since 2021-10-23 04:08:37.459247
pg 1.4d not scrubbed since 2021-10-21 02:55:35.917909
pg 2.4f not scrubbed since 2021-10-19 16:00:28.298636
pg 1.4c not scrubbed since 2021-10-23 05:06:26.635988
pg 2.48 not scrubbed since 2021-10-20 15:21:00.489987
pg 1.4b not scrubbed since 2021-10-21 00:32:31.476136
pg 2.49 not scrubbed since 2021-10-21 15:06:03.567184
pg 1.4a not scrubbed since 2021-10-25 10:12:18.818619
pg 2.4a not scrubbed since 2021-10-21 22:18:01.775421
pg 1.49 not scrubbed since 2021-10-26 00:40:04.835318
pg 2.4b not scrubbed since 2021-10-24 09:16:15.033388
pg 1.48 not scrubbed since 2021-10-24 09:16:16.683504
pg 2.44 not scrubbed since 2021-10-22 06:13:32.823312
pg 1.47 not scrubbed since 2021-10-23 11:40:55.090946
pg 2.45 not scrubbed since 2021-10-20 08:38:47.050011
pg 1.46 not scrubbed since 2021-10-22 09:40:31.009867
pg 2.46 not scrubbed since 2021-10-22 08:29:00.360555
pg 1.45 not scrubbed since 2021-10-20 06:14:21.559948
pg 2.47 not scrubbed since 2021-10-25 21:42:08.620106
pg 1.44 not scrubbed since 2021-10-26 09:43:13.469079
pg 2.40 not scrubbed since 2021-10-20 22:17:24.436075
pg 1.43 not scrubbed since 2021-10-20 05:06:07.456276
pg 2.41 not scrubbed since 2021-10-25 12:58:16.095429
pg 1.42 not scrubbed since 2021-10-26 03:47:35.750935
pg 2.42 not scrubbed since 2021-10-21 07:23:38.759214
pg 1.41 not scrubbed since 2021-10-26 08:11:07.359517
pg 2.43 not scrubbed since 2021-10-18 23:26:00.875389
pg 1.40 not scrubbed since 2021-10-26 06:50:13.906860
pg 3.3d not scrubbed since 2021-10-26 08:17:20.032897
pg 2.3c not scrubbed since 2021-10-22 12:49:23.070474
pg 1.3f not scrubbed since 2021-10-21 22:00:39.969493
pg 3.3c not scrubbed since 2021-10-23 04:18:01.814731
pg 2.3d not scrubbed since 2021-10-22 23:24:25.938902
pg 1.3e not scrubbed since 2021-10-24 00:23:49.326500
pg 3.3f not scrubbed since 2021-10-21 08:39:31.130110
pg 2.3e not scrubbed since 2021-10-19 13:41:47.476872
pg 1.3d not scrubbed since 2021-10-18 13:28:08.018827
pg 3.3e not scrubbed since 2021-10-24 07:01:14.880346
pg 2.3f not scrubbed since 2021-10-23 12:48:45.693678
pg 1.3c not scrubbed since 2021-10-22 05:06:23.468608
pg 3.39 not scrubbed since 2021-10-20 23:25:32.265623
pg 2.38 not scrubbed since 2021-10-26 08:12:52.041170
pg 1.3b not scrubbed since 2021-10-20 17:21:51.110163
pg 3.38 not scrubbed since 2021-10-19 16:05:37.158553
pg 2.39 not scrubbed since 2021-10-19 22:12:13.829894
pg 1.3a not scrubbed since 2021-10-24 02:28:16.955567
pg 3.3b not scrubbed since 2021-10-21 00:37:02.487811
pg 2.3a not scrubbed since 2021-10-21 22:00:38.189966
pg 1.39 not scrubbed since 2021-10-20 23:20:27.770843
pg 3.3a not scrubbed since 2021-10-23 04:13:33.023764
270 more pgs...
MON_DOWN 1/3 mons down, quorum p-001,p-002
mon.p-003 (rank 0) addr [v2:10.10.0.3:3300/0,v1:10.1.0.3:6789/0] is down (out of quorum)

How can I solve this problem?

Thank you

Frankie

 

 

 

#1

admin
2,930 Posts
November 24, 2021, 3:56 pm

The correct way is to add more storage / OSDs. Else you can try to adjust / lower the OSD crush weight for OSD 3 by say 10%, so for example if your OSD disk was 4TB capacity, you will put a crush weight of 3.6 ,you can do so from maintenance window.

Last edited on November 24, 2021, 3:56 pm by admin · #2

Frankie
6 Posts
November 24, 2021, 4:14 pm

Thank you for answer.
But why don't I sync the space of the deleted folder?

#3

admin
2,930 Posts
November 24, 2021, 4:52 pm

not sure i understand, if you talking about deleting a top level share, do so from the ui so the gateways (cifs/nfs) will stop serving it, if it is folder within a share, then you can connect a client to that share and delete the folder. For deleting a folder you can also do this from command line, under /mnt/cephfs/cifs|nfs/share/...

#4

Frankie
6 Posts
November 25, 2021, 10:00 am

I have 3 shares in:

/mnt/cephfs/cifs/

  • fs01
  • user
  • backup

when i deleting one folder, the dimension not changed. The pool cephfs_data is nearfull.

 

#5

Frankie
6 Posts
November 25, 2021, 10:23 am

My cephf status is:

cluster:
id: 08960832-6f92-4b35-89f5-e4ca6e517d7e
health: HEALTH_WARN
1 backfillfull osd(s)
3 pool(s) backfillfull
Degraded data redundancy: 7148415/21445245 objects degraded (33.333%), 320 pgs degraded, 320 pgs undersized
320 pgs not deep-scrubbed in time
320 pgs not scrubbed in time
1/3 mons down, quorum petasan1,petasan2

services:
mon: 3 daemons, quorum p-001,p-002 (age 5d), out of quorum: p-003
mgr: p-001(active, since 5d), standbys: p-002
mds: cephfs:1 {0=p-002=up:active} 1 up:standby
osd: 6 osds: 4 up (since 5d), 4 in (since 4w)

task status:
scrub status:
mds.p-002: idle

data:
pools: 3 pools, 320 pgs
objects: 7.15M objects, 3.5 TiB
usage: 7.3 TiB used, 1.4 TiB / 8.7 TiB avail
pgs: 7148415/21445245 objects degraded (33.333%)
320 active+undersized+degraded

io:
client: 171 KiB/s rd, 58 KiB/s wr, 0 op/s rd, 2 op/s wr

#6

admin
2,930 Posts
November 25, 2021, 9:04 pm

 

i would recommend you adjust OSD weight as earlier.

do you say when you delete files with large size , it does not reflect on OSD available storage ?

can you show output of

df

ceph df

ceph osd df

#7

Frankie
6 Posts
November 25, 2021, 11:22 pm

df
Filesystem 1K-blocks Used Available Use% Mounted on
udev 24697876 0 24697876 0% /dev
tmpfs 4944464 7272 4937192 1% /run
/dev/sda3 15416264 10090656 5309224 66% /
tmpfs 24722316 0 24722316 0% /dev/shm
tmpfs 5120 0 5120 0% /run/lock
tmpfs 24722316 0 24722316 0% /sys/fs/cgroup
/dev/sda2 129039 1 129039 1% /boot/efi
/dev/sda5 46052168 173232 45862552 1% /opt/petasan/config
/dev/sda4 30832548 126100 30690064 1% /var/lib/ceph
tmpfs 24722316 24 24722292 1% /var/lib/ceph/osd/ceph-3
tmpfs 24722316 24 24722292 1% /var/lib/ceph/osd/ceph-2
10.10.0.1,10.10.0.2,10.10.0.3:/ 3794284544 3688087552 106196992 98% /mnt/cephfs
/dev/vda 209664000 150244248 59419752 72% /mnt/cache
10.1.0.1:gfs-vol 46052096 633728 45862528 2% /opt/petasan/config/shared

 

ceph df

RAW STORAGE:
CLASS SIZE AVAIL USED RAW USED %RAW USED
hdd 8.7 TiB 1.3 TiB 7.3 TiB 7.4 TiB 84.71
TOTAL 8.7 TiB 1.3 TiB 7.3 TiB 7.4 TiB 84.71

POOLS:
POOL ID STORED OBJECTS USED %USED MAX AVAIL
rbd 1 115 GiB 32.06k 231 GiB 43.17 152 GiB
cephfs_data 2 3.4 TiB 6.85M 7.1 TiB 96.00 152 GiB
cephfs_metadata 3 13 GiB 276.19k 13 GiB 4.14 152 GiB

ceph osd df

ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS
2 hdd 2.18259 1.00000 2.2 TiB 1.7 TiB 1.7 TiB 9.4 GiB 5.1 GiB 521 GiB 76.69 0.91 160 up
3 hdd 2.18259 1.00000 2.2 TiB 2.0 TiB 2.0 TiB 6.3 GiB 6.3 GiB 162 GiB 92.73 1.09 160 up
4 hdd 2.18259 1.00000 2.2 TiB 1.8 TiB 1.8 TiB 7.9 GiB 5.8 GiB 358 GiB 83.99 0.99 161 up
5 hdd 2.18259 1.00000 2.2 TiB 1.9 TiB 1.9 TiB 8.1 GiB 5.5 GiB 325 GiB 85.44 1.01 159 up
0 hdd 2.18259 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down
1 hdd 2.18259 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down
TOTAL 8.7 TiB 7.4 TiB 7.3 TiB 32 GiB 23 GiB 1.3 TiB 84.71
MIN/MAX VAR: 0.91/1.09 STDDEV: 5.70

do you say when you delete files with large size , it does not reflect on OSD available storage ?

 

Exact

thank you

#8

admin
2,930 Posts
November 26, 2021, 12:24 am

The df command shows you are using 3.5 TB, you believe this is incorrect ?

can you show output of:

du  -ch -d 2  /mnt/cephfs

is it showing different usage ?

do you have any clients with connections to deleted folders ?

can you describe what clients you have : are they all Windows accessing CIFS shares ?

can you give more detail on:

I deleted a very heavy folder (almost 1tb to make space) but I do not synchronize the cancellation.

was it from Windows ?

Last edited on November 26, 2021, 12:25 am by admin · #9

Frankie
6 Posts
November 26, 2021, 8:36 am

du -ch -d 2 /mnt/cephfs

231G /mnt/cephfs/cifs/backup
8.0G /mnt/cephfs/cifs/user
350G /mnt/cephfs/cifs/fs01
589G /mnt/cephfs/cifs
589G /mnt/cephfs
589G total

Yes, the dimension si correct 3.5tb. I deleted directly from cmdl, many folders inside . The shares are all like network shared in windows and for proof I deleted from windows a folder of 10gb, but he did not change anything.

Thank you

#10
Pages: 1 2