pgs inconsistent
lberrezoug
5 Posts
January 14, 2025, 2:08 amQuote from lberrezoug on January 14, 2025, 2:08 amHi,
I'm looking for help with an issue where two PGs are in an inconsistent state, and the commands ceph pg deep-scrub $pgid and ceph pg repair $pgid seem unable to resolve it.
From further analysis using the command rados list-inconsistent-obj $pgid --format=json-pretty, it appears that:
- The first PG, identified as 10.f, is associated with Primary OSD 11.
- The second PG, identified as 10.16, is associated with Primary OSD 18.
Both OSDs are in the UP state, and the SMART reports for the corresponding disks show no anomalies. However, even after running ceph osd repair all, the problem persists.
Using rados list-inconsistent-obj $pgid --format=json-pretty and the rbd info image command, I found that these two PGs are linked to two RADOS images "‘image-00001, image-00002" used by VMware.
I'm at a bit of a loss, and any help or recommendations would be greatly appreciated. I understand that this issue is not directly related to PetaSAN but rather to Ceph. However, I kindly ask for your assistance in finding a solution.
Best regards,
For more details:
- I have 4 identical nodes in terms of hardware.
- Each node has 6 hdd disks for data and 2 ssd disks for journaling and 2nvme for cache
- All pools are in replication mode 3.
- My PetaSAN cluster was deployed less than six months ago.
- At the time of writing this message i noticed that i have another pg 10.1c in an inconsistent state, i'm trying to fix it and i hope it doesn't persist like the other two. it also affects the rados image ‘image-00002’ and other OSDs but still primary OSD 11.
ceph health detail
HEALTH_ERR 1 clients failing to respond to cache pressure; 5 scrub errors; Too many repaired reads on 1 OSDs; Possible data damage: 3 pgs inconsistent
[WRN] MDS_CLIENT_RECALL: 1 clients failing to respond to cache pressure
mds.node-stor1(mds.0): Client NFS-1XX-XXX-XXX-103 failing to respond to cache pressure client_id: 9489331
[ERR] OSD_SCRUB_ERRORS: 5 scrub errors
[WRN] OSD_TOO_MANY_REPAIRS: Too many repaired reads on 1 OSDs
osd.11 had 30 reads repaired
[ERR] PG_DAMAGED: Possible data damage: 3 pgs inconsistent
pg 10.f is active+clean+scrubbing+deep+inconsistent, acting [18,12,11]
pg 10.16 is active+clean+inconsistent, acting [11,1,23]
pg 10.1c is active+clean+inconsistent, acting [11,20,16]
ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 350.67114 root default
-5 87.66779 host node -stor1
6 hdd 14.61130 osd.6 up 1.00000 1.00000
7 hdd 14.61130 osd.7 up 1.00000 1.00000
8 hdd 14.61130 osd.8 up 1.00000 1.00000
9 hdd 14.61130 osd.9 up 1.00000 1.00000
10 hdd 14.61130 osd.10 up 1.00000 1.00000
11 hdd 14.61130 osd.11 up 1.00000 1.00000
-7 87.66779 host node-stor2
12 hdd 14.61130 osd.12 up 1.00000 1.00000
13 hdd 14.61130 osd.13 up 1.00000 1.00000
14 hdd 14.61130 osd.14 up 1.00000 1.00000
15 hdd 14.61130 osd.15 up 1.00000 1.00000
16 hdd 14.61130 osd.16 up 1.00000 1.00000
17 hdd 14.61130 osd.17 up 1.00000 1.00000
-3 87.66779 host node-stor3
0 hdd 14.61130 osd.0 up 1.00000 1.00000
1 hdd 14.61130 osd.1 up 1.00000 1.00000
2 hdd 14.61130 osd.2 up 1.00000 1.00000
3 hdd 14.61130 osd.3 up 1.00000 1.00000
4 hdd 14.61130 osd.4 up 1.00000 1.00000
5 hdd 14.61130 osd.5 up 1.00000 1.00000
-9 87.66779 host node-stor4
18 hdd 14.61130 osd.18 up 1.00000 1.00000
19 hdd 14.61130 osd.19 up 1.00000 1.00000
20 hdd 14.61130 osd.20 up 1.00000 1.00000
21 hdd 14.61130 osd.21 up 1.00000 1.00000
22 hdd 14.61130 osd.22 up 1.00000 1.00000
23 hdd 14.61130 osd.23 up 1.00000 1.00000
ceph osd df
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS
6 hdd 14.61130 1.00000 15 TiB 1.1 TiB 1.0 TiB 0 B 2.0 GiB 14 TiB 7.40 0.66 20 up
7 hdd 14.61130 1.00000 15 TiB 1.9 TiB 1.8 TiB 3 KiB 3.7 GiB 13 TiB 13.02 1.17 27 up
8 hdd 14.61130 1.00000 15 TiB 1.2 TiB 1.2 TiB 3 KiB 2.4 GiB 13 TiB 8.49 0.76 19 up
9 hdd 14.61130 1.00000 15 TiB 2.2 TiB 2.1 TiB 283 KiB 4.3 GiB 12 TiB 14.96 1.34 30 up
10 hdd 14.61130 1.00000 15 TiB 1.4 TiB 1.3 TiB 0 B 2.7 GiB 13 TiB 9.29 0.83 19 up
11 hdd 14.61130 1.00000 15 TiB 2.6 TiB 2.6 TiB 0 B 5.1 GiB 12 TiB 17.88 1.60 29 up
12 hdd 14.61130 1.00000 15 TiB 1.7 TiB 1.7 TiB 0 B 3.1 GiB 13 TiB 11.97 1.07 24 up
13 hdd 14.61130 1.00000 15 TiB 2.0 TiB 1.9 TiB 3 KiB 3.8 GiB 13 TiB 13.59 1.22 27 up
14 hdd 14.61130 1.00000 15 TiB 1.1 TiB 1.0 TiB 0 B 2.0 GiB 14 TiB 7.41 0.66 17 up
15 hdd 14.61130 1.00000 15 TiB 1.5 TiB 1.5 TiB 0 B 2.8 GiB 13 TiB 10.36 0.93 25 up
16 hdd 14.61130 1.00000 15 TiB 583 GiB 523 GiB 0 B 1.3 GiB 14 TiB 3.90 0.35 19 up
17 hdd 14.61130 1.00000 15 TiB 1.6 TiB 1.5 TiB 27 KiB 2.9 GiB 13 TiB 10.92 0.98 23 up
0 hdd 14.61130 1.00000 15 TiB 2.6 TiB 2.6 TiB 248 KiB 5.0 GiB 12 TiB 17.87 1.60 24 up
1 hdd 14.61130 1.00000 15 TiB 1.9 TiB 1.8 TiB 0 B 3.6 GiB 13 TiB 12.78 1.14 32 up
2 hdd 14.61130 1.00000 15 TiB 1.1 TiB 1.0 TiB 0 B 2.2 GiB 14 TiB 7.41 0.66 23 up
3 hdd 14.61130 1.00000 15 TiB 948 GiB 888 GiB 28 KiB 1.9 GiB 14 TiB 6.33 0.57 27 up
4 hdd 14.61130 1.00000 15 TiB 1.5 TiB 1.5 TiB 4 KiB 2.8 GiB 13 TiB 10.35 0.93 20 up
5 hdd 14.61130 1.00000 15 TiB 1.2 TiB 1.2 TiB 0 B 2.3 GiB 13 TiB 8.50 0.76 16 up
18 hdd 14.61130 1.00000 15 TiB 1.5 TiB 1.4 TiB 0 B 2.8 GiB 13 TiB 10.32 0.92 31 up
19 hdd 14.61130 1.00000 15 TiB 1.5 TiB 1.5 TiB 0 B 3 GiB 13 TiB 10.36 0.93 20 up
20 hdd 14.61130 1.00000 15 TiB 1.9 TiB 1.8 TiB 0 B 3.9 GiB 13 TiB 13.04 1.17 27 up
21 hdd 14.61130 1.00000 15 TiB 1.4 TiB 1.3 TiB 0 B 2.7 GiB 13 TiB 9.58 0.86 17 up
22 hdd 14.61130 1.00000 15 TiB 3.4 TiB 3.3 TiB 291 KiB 6.0 GiB 11 TiB 23.01 2.06 41 up
23 hdd 14.61130 1.00000 15 TiB 1.4 TiB 1.3 TiB 27 KiB 2.8 GiB 13 TiB 9.32 0.83 22 up
TOTAL 351 TiB 39 TiB 38 TiB 922 KiB 75 GiB 312 TiB 11.17
MIN/MAX VAR: 0.35/2.06 STDDEV: 4.11
rados list-inconsistent-obj 10.f --format=json-pretty
{
"epoch": 3539,
"inconsistents": [
{
"object": {
"name": "rbd_data.22ceee723c5c.00000000000121eb",
"nspace": "",
"locator": "",
"snap": "head",
"version": 8421742
},
"errors": [],
"union_shard_errors": [
"read_error"
],
"selected_object_info": {
"oid": {
"oid": "rbd_data.22ceee723c5c.00000000000121eb",
"key": "",
"snapid": -2,
"hash": 1737665103,
"max": 0,
"pool": 10,
"namespace": ""
},
"version": "3554'8421742",
"prior_version": "3554'8421426",
"last_reqid": "client.143219.0:563462592",
"user_version": 8421742,
"size": 4194304,
"mtime": "2025-01-09T06:07:44.952161+0100",
"local_mtime": "2025-01-09T06:07:44.953033+0100",
"lost": 0,
"flags": [
"dirty",
"omap_digest"
],
"truncate_seq": 0,
"truncate_size": 0,
"data_digest": "0xffffffff",
"omap_digest": "0xffffffff",
"expected_object_size": 4194304,
"expected_write_size": 4194304,
"alloc_hint_flags": 0,
"manifest": {
"type": 0
},
"watchers": {}
},
"shards": [
{
"osd": 11,
"primary": false,
"errors": [
"read_error"
],
"size": 4194304
},
{
"osd": 12,
"primary": false,
"errors": [],
"size": 4194304,
"omap_digest": "0xffffffff",
"data_digest": "0xf4bf72f2"
},
{
"osd": 18,
"primary": true,
"errors": [],
"size": 4194304,
"omap_digest": "0xffffffff",
"data_digest": "0xf4bf72f2"
}
]
},
{
"object": {
"name": "rbd_data.261b03a6aa264.0000000000009a34",
"nspace": "",
"locator": "",
"snap": "head",
"version": 8351559
},
"errors": [],
"union_shard_errors": [
"read_error"
],
"selected_object_info": {
"oid": {
"oid": "rbd_data.261b03a6aa264.0000000000009a34",
"key": "",
"snapid": -2,
"hash": 2089893551,
"max": 0,
"pool": 10,
"namespace": ""
},
"version": "3524'8351559",
"prior_version": "3524'8351558",
"last_reqid": "client.143219.0:556722557",
"user_version": 8351559,
"size": 4194304,
"mtime": "2025-01-08T20:56:46.225679+0100",
"local_mtime": "2025-01-08T20:56:46.225556+0100",
"lost": 0,
"flags": [
"dirty",
"omap_digest"
],
"truncate_seq": 0,
"truncate_size": 0,
"data_digest": "0xffffffff",
"omap_digest": "0xffffffff",
"expected_object_size": 4194304,
"expected_write_size": 4194304,
"alloc_hint_flags": 0,
"manifest": {
"type": 0
},
"watchers": {}
},
"shards": [
{
"osd": 11,
"primary": false,
"errors": [
"read_error"
],
"size": 4194304
},
{
"osd": 12,
"primary": false,
"errors": [],
"size": 4194304,
"omap_digest": "0xffffffff",
"data_digest": "0x66b5055b"
},
{
"osd": 18,
"primary": true,
"errors": [],
"size": 4194304,
"omap_digest": "0xffffffff",
"data_digest": "0x66b5055b"
}
]
}
]
}
rados list-inconsistent-obj 10.16 --format=json-pretty
{
"epoch": 3533,
"inconsistents": [
{
"object": {
"name": "rbd_data.261b03a6aa264.0000000000008921",
"nspace": "",
"locator": "",
"snap": "head",
"version": 9756930
},
"errors": [],
"union_shard_errors": [
"read_error"
],
"selected_object_info": {
"oid": {
"oid": "rbd_data.261b03a6aa264.0000000000008921",
"key": "",
"snapid": -2,
"hash": 2078265174,
"max": 0,
"pool": 10,
"namespace": ""
},
"version": "3524'9756930",
"prior_version": "3524'9756929",
"last_reqid": "client.143219.0:556715005",
"user_version": 9756930,
"size": 4194304,
"mtime": "2025-01-08T20:56:42.556007+0100",
"local_mtime": "2025-01-08T20:56:42.556296+0100",
"lost": 0,
"flags": [
"dirty",
"omap_digest"
],
"truncate_seq": 0,
"truncate_size": 0,
"data_digest": "0xffffffff",
"omap_digest": "0xffffffff",
"expected_object_size": 4194304,
"expected_write_size": 4194304,
"alloc_hint_flags": 0,
"manifest": {
"type": 0
},
"watchers": {}
},
"shards": [
{
"osd": 1,
"primary": false,
"errors": [],
"size": 4194304,
"omap_digest": "0xffffffff",
"data_digest": "0x98f78947"
},
{
"osd": 11,
"primary": true,
"errors": [
"read_error"
],
"size": 4194304
},
{
"osd": 23,
"primary": false,
"errors": [],
"size": 4194304,
"omap_digest": "0xffffffff",
"data_digest": "0x98f78947"
}
]
}
]
}
rbd image 'image-00001':
size 3 TiB in 786432 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: 22ceee723c5c
block_name_prefix: rbd_data.22ceee723c5c
format: 2
features: layering
op_features:
flags:
create_timestamp: Wed Dec 11 07:47:26 2024
access_timestamp: Wed Dec 11 07:47:26 2024
modify_timestamp: Wed Dec 11 07:47:26 2024
root@orn-45d-stor1:~# rbd info image-00002
rbd image 'image-00002':
size 3 TiB in 786432 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: 261b03a6aa264
block_name_prefix: rbd_data.261b03a6aa264
format: 2
features: layering
op_features:
flags:
create_timestamp: Wed Dec 11 08:07:26 2024
access_timestamp: Wed Dec 11 08:07:26 2024
modify_timestamp: Wed Dec 11 08:07:26 2024
ceph device ls
DEVICE HOST:DEV DAEMONS WEAR LIFE EXPECTANCY
HDSTOR_-_HSAV25S_HS22093014A0F7E55 node-stor1:sda mon.node-stor1 53%
HDSTOR_-_HSAV25S_HS22101214A101BFE node-stor2:sda mon.node-stor2 51%
HDSTOR_-_HSAV25S_HS22101214A101C70 node-stor3:sdb mon.node-stor3 49%
Micron_5400_MTFD_500a07514428bbf1 node-stor4:sdi osd.18 osd.20 osd.22 0%
Micron_5400_MTFD_500a07514428bddf node-stor1:sdi osd.10 osd.6 osd.8 0%
Micron_5400_MTFD_500a0751442ad458 node-stor1:sdj osd.11 osd.7 osd.9 0%
Micron_5400_MTFD_500a0751442ad55e node-stor2:sdj osd.13 osd.15 osd.17 0%
Micron_5400_MTFD_500a0751442ad56a node-stor3:sdi osd.0 osd.2 osd.4 0%
Micron_5400_MTFD_500a0751442ad56e node-stor4:sdj osd.19 osd.21 osd.23 0%
Micron_5400_MTFD_500a0751442ad579 node-stor3:sdj osd.1 osd.3 osd.5 0%
Micron_5400_MTFD_500a0751442adf6c node-stor2:sdi osd.12 osd.14 osd.16 0%
Micron_7300_MTFDHBG1T9TDF_2022327BF1F2 node-stor2:nvme1n1 osd.13 osd.15 osd.17 0%
Micron_7300_MTFDHBG1T9TDF_214132747BDF node-stor2:nvme0n1 osd.12 osd.14 osd.16 35%
Micron_7450_MTFDKBG1T9TFR_221536FA4B7B node-stor1:nvme0n1 osd.11 osd.7 osd.9 0%
Micron_7450_MTFDKBG1T9TFR_221536FA4D68 node-stor4:nvme1n1 osd.18 osd.20 osd.22 0%
Micron_7450_MTFDKBG1T9TFR_221536FA4D78 node-stor4:nvme0n1 osd.19 osd.21 osd.23 51%
Micron_7450_MTFDKBG1T9TFR_221536FA4E67 node-stor3:nvme1n1 osd.0 osd.2 osd.4 0%
Micron_7450_MTFDKBG1T9TFR_221536FA4E7B node-stor3:nvme0n1 osd.1 osd.3 osd.5 58%
Micron_7450_MTFDKBG1T9TFR_22313BEC8410 node-stor1:nvme1n1 osd.10 osd.6 osd.8 48%
ST16000NM005J_5000c500ec286bdb node-stor4:sdf osd.20
ST16000NM005J_5000c500ec43256f node-stor1:sdf osd.11
ST16000NM005J_5000c500ec44302b node-stor2:sdf osd.17
ST16000NM005J_5000c500ec4437ef node-stor3:sdd osd.1
ST16000NM005J_5000c500ec44840f node-stor3:sdg osd.3
ST16000NM005J_5000c500ec44932b node-stor3:sdc osd.0
ST16000NM005J_5000c500ec53ba27 node-stor3:sde osd.2
ST16000NM005J_5000c500ec53ba97 node-stor1:sdd osd.7
ST16000NM005J_5000c500ec53bbf3 node-stor1:sde osd.10
ST16000NM005J_5000c500ec5411f3 node-stor1:sdc osd.6
ST16000NM005J_5000c500ec541507 node-stor1:sdg osd.8
ST16000NM005J_5000c500ec5415af node-stor3:sdh osd.4
ST16000NM005J_5000c500ec544fe3 node-stor1:sdh osd.9
ST16000NM005J_5000c500ec5451fb node-stor2:sde osd.14
ST16000NM005J_5000c500ec5770eb node-stor2:sdg osd.15
ST16000NM005J_5000c500ec5775c3 node-stor3:sdf osd.5
ST16000NM005J_5000c500ec57ec3b node-stor4:sdc osd.18
ST16000NM005J_5000c500ec57ec3f node-stor2:sdh osd.16
ST16000NM005J_5000c500ec57ec53 node-stor4:sde osd.21
ST16000NM005J_5000c500ec57ec67 node-stor4:sdg osd.22
ST16000NM005J_5000c500f29e867f node-stor4:sdh osd.23
ST16000NM005J_5000c500f2abea4f node-stor2:sdc osd.12
ST16000NM005J_5000c500f2ac0b9b node-stor4:sdd osd.19
ST16000NM005J_5000c500f2ac1373 node-stor2:sdd osd.13
smartctl --all /dev/sdf
smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.14.21-08-petasan] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Vendor: SEAGATE
Product: ST16000NM005J
Revision: ET04
Compliance: SPC-5
User Capacity: 16,000,900,661,248 bytes [16.0 TB]
Logical block size: 512 bytes
Physical block size: 4096 bytes
LU is fully provisioned
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Logical Unit id: 0x5000c500ec43256f
Serial number: ZR70Z2FC0000W34716QQ
Device type: disk
Transport protocol: SAS (SPL-3)
Local Time is: Tue Jan 14 03:30:40 2025 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK
Grown defects during certification <not available>
Total blocks reassigned during format <not available>
Total new blocks reassigned <not available>
Power on minutes since format <not available>
Current Drive Temperature: 33 C
Drive Trip Temperature: 60 C
Manufactured in week 42 of year 2023
Specified cycle count over device lifetime: 50000
Accumulated start-stop cycles: 13
Specified load-unload count over device lifetime: 600000
Accumulated load-unload cycles: 4209
Elements in grown defect list: 0
Vendor (Seagate Cache) information
Blocks sent to initiator = 717091008
Blocks received from initiator = 1661229904
Blocks read from cache and sent to initiator = 1742662296
Number of read and write commands whose size <= segment size = 15420970
Number of read and write commands whose size > segment size = 34014
Vendor (Seagate/Hitachi) factory information
number of hours powered up = 6537.55
number of minutes until next internal SMART test = 56
Error counter log:
Errors Corrected by Total Correction Gigabytes Total
ECC rereads/ errors algorithm processed uncorrected
fast | delayed rewrites corrected invocations [10^9 bytes] errors
read: 0 0 0 0 0 15760.315 0
write: 0 0 0 0 0 850.911 0
Non-medium error count: 0
[GLTSD (Global Logging Target Save Disable) set. Enable Save with '-S on']
No Self-tests have been logged
smartctl --all /dev/sdc
smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.14.21-08-petasan] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Vendor: SEAGATE
Product: ST16000NM005J
Revision: ET04
Compliance: SPC-5
User Capacity: 16,000,900,661,248 bytes [16.0 TB]
Logical block size: 512 bytes
Physical block size: 4096 bytes
LU is fully provisioned
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Logical Unit id: 0x5000c500ec5411f3
Serial number: ZR71045F0000W3474RQT
Device type: disk
Transport protocol: SAS (SPL-3)
Local Time is: Tue Jan 14 03:31:47 2025 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK
Grown defects during certification <not available>
Total blocks reassigned during format <not available>
Total new blocks reassigned <not available>
Power on minutes since format <not available>
Current Drive Temperature: 29 C
Drive Trip Temperature: 60 C
Manufactured in week 42 of year 2023
Specified cycle count over device lifetime: 50000
Accumulated start-stop cycles: 11
Specified load-unload count over device lifetime: 600000
Accumulated load-unload cycles: 6097
Elements in grown defect list: 0
Vendor (Seagate Cache) information
Blocks sent to initiator = 3487322904
Blocks received from initiator = 3137788280
Blocks read from cache and sent to initiator = 2103778790
Number of read and write commands whose size <= segment size = 11493458
Number of read and write commands whose size > segment size = 26103
Vendor (Seagate/Hitachi) factory information
number of hours powered up = 6537.57
number of minutes until next internal SMART test = 14
Error counter log:
Errors Corrected by Total Correction Gigabytes Total
ECC rereads/ errors algorithm processed uncorrected
fast | delayed rewrites corrected invocations [10^9 bytes] errors
read: 0 0 0 0 0 14979.649 0
write: 0 0 0 0 0 3808.633 0
Non-medium error count: 0
[GLTSD (Global Logging Target Save Disable) set. Enable Save with '-S on']
No Self-tests have been logged
journalctl -f -u ceph-osd@11
-- Logs begin at Sun 2025-01-05 12:21:06 CET. --
Jan 14 01:48:48 node-stor1 ceph-osd[5967]: 2025-01-14T01:48:48.271+0100 7f2196b4b700 -1 log_channel(cluster) log [ERR] : 10.1c deep-scrub 0 missing, 2 inconsistent objects
Jan 14 01:48:48 node-stor1 ceph-osd[5967]: 2025-01-14T01:48:48.271+0100 7f2196b4b700 -1 log_channel(cluster) log [ERR] : 10.1c deep-scrub 2 errors
Jan 14 02:19:36 node-stor1 ceph-osd[5967]: 2025-01-14T02:19:36.929+0100 7f219834e700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x9000, got 0x108fe2eb, expected 0x95cc874a, device location [0x333a4d09000~1000], logical extent 0x169000~1000, object #10:f25549e6:::rbd_data.22ceee723c5c.00000000000121eb:head#
Jan 14 02:19:36 node-stor1 ceph-osd[5967]: 2025-01-14T02:19:36.929+0100 7f219834e700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x9000, got 0x108fe2eb, expected 0x95cc874a, device location [0x333a4d09000~1000], logical extent 0x169000~1000, object #10:f25549e6:::rbd_data.22ceee723c5c.00000000000121eb:head#
Jan 14 02:19:36 node-stor1 ceph-osd[5967]: 2025-01-14T02:19:36.933+0100 7f219834e700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x9000, got 0x108fe2eb, expected 0x95cc874a, device location [0x333a4d09000~1000], logical extent 0x169000~1000, object #10:f25549e6:::rbd_data.22ceee723c5c.00000000000121eb:head#
Jan 14 02:19:36 node-stor1 ceph-osd[5967]: 2025-01-14T02:19:36.933+0100 7f219834e700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x9000, got 0x108fe2eb, expected 0x95cc874a, device location [0x333a4d09000~1000], logical extent 0x169000~1000, object #10:f25549e6:::rbd_data.22ceee723c5c.00000000000121eb:head#
Jan 14 02:57:08 node-stor1 ceph-osd[5967]: 2025-01-14T02:57:08.891+0100 7f219834e700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x0, got 0xb21969fe, expected 0xea08e29, device location [0x484b14c0000~1000], logical extent 0x380000~1000, object #10:f57c893e:::rbd_data.261b03a6aa264.0000000000009a34:head#
Jan 14 02:57:08 node-stor1 ceph-osd[5967]: 2025-01-14T02:57:08.903+0100 7f219834e700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x0, got 0xb21969fe, expected 0xea08e29, device location [0x484b14c0000~1000], logical extent 0x380000~1000, object #10:f57c893e:::rbd_data.261b03a6aa264.0000000000009a34:head#
Jan 14 02:57:08 node-stor1 ceph-osd[5967]: 2025-01-14T02:57:08.903+0100 7f219834e700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x0, got 0xb21969fe, expected 0xea08e29, device location [0x484b14c0000~1000], logical extent 0x380000~1000, object #10:f57c893e:::rbd_data.261b03a6aa264.0000000000009a34:head#
Jan 14 02:57:08 node-stor1 ceph-osd[5967]: 2025-01-14T02:57:08.903+0100 7f219834e700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x0, got 0xb21969fe, expected 0xea08e29, device location [0x484b14c0000~1000], logical extent 0x380000~1000, object #10:f57c893e:::rbd_data.261b03a6aa264.0000000000009a34:head#
Jan 14 03:54:48 node-stor1 ceph-osd[5967]: 2025-01-14T03:54:48.707+0100 7f2196b4b700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x1000, got 0xc8021cc4, expected 0x6706be76, device location [0x45f6c021000~1000], logical extent 0x281000~1000, object #10:3a57c27b:::rbd_data.261b03a6aa264.0000000000007add:head#
Jan 14 03:54:48 node-stor1 ceph-osd[5967]: 2025-01-14T03:54:48.707+0100 7f2196b4b700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x1000, got 0xc8021cc4, expected 0x6706be76, device location [0x45f6c021000~1000], logical extent 0x281000~1000, object #10:3a57c27b:::rbd_data.261b03a6aa264.0000000000007add:head#
Jan 14 03:54:48 node-stor1 ceph-osd[5967]: 2025-01-14T03:54:48.711+0100 7f2196b4b700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x1000, got 0xc8021cc4, expected 0x6706be76, device location [0x45f6c021000~1000], logical extent 0x281000~1000, object #10:3a57c27b:::rbd_data.261b03a6aa264.0000000000007add:head#
Jan 14 03:54:48 node-stor1 ceph-osd[5967]: 2025-01-14T03:54:48.711+0100 7f2196b4b700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x1000, got 0xc8021cc4, expected 0x6706be76, device location [0x45f6c021000~1000], logical extent 0x281000~1000, object #10:3a57c27b:::rbd_data.261b03a6aa264.0000000000007add:head#
Jan 14 03:54:50 node-stor1 ceph-osd[5967]: 2025-01-14T03:54:50.075+0100 7f2196b4b700 -1 log_channel(cluster) log [ERR] : 10.1c shard 11 soid 10:3a57c27b:::rbd_data.261b03a6aa264.0000000000007add:head : candidate had a read error
Hi,
I'm looking for help with an issue where two PGs are in an inconsistent state, and the commands ceph pg deep-scrub $pgid and ceph pg repair $pgid seem unable to resolve it.
From further analysis using the command rados list-inconsistent-obj $pgid --format=json-pretty, it appears that:
- The first PG, identified as 10.f, is associated with Primary OSD 11.
- The second PG, identified as 10.16, is associated with Primary OSD 18.
Both OSDs are in the UP state, and the SMART reports for the corresponding disks show no anomalies. However, even after running ceph osd repair all, the problem persists.
Using rados list-inconsistent-obj $pgid --format=json-pretty and the rbd info image command, I found that these two PGs are linked to two RADOS images "‘image-00001, image-00002" used by VMware.
I'm at a bit of a loss, and any help or recommendations would be greatly appreciated. I understand that this issue is not directly related to PetaSAN but rather to Ceph. However, I kindly ask for your assistance in finding a solution.
Best regards,
For more details:
- I have 4 identical nodes in terms of hardware.
- Each node has 6 hdd disks for data and 2 ssd disks for journaling and 2nvme for cache
- All pools are in replication mode 3.
- My PetaSAN cluster was deployed less than six months ago.
- At the time of writing this message i noticed that i have another pg 10.1c in an inconsistent state, i'm trying to fix it and i hope it doesn't persist like the other two. it also affects the rados image ‘image-00002’ and other OSDs but still primary OSD 11.
ceph health detail
HEALTH_ERR 1 clients failing to respond to cache pressure; 5 scrub errors; Too many repaired reads on 1 OSDs; Possible data damage: 3 pgs inconsistent
[WRN] MDS_CLIENT_RECALL: 1 clients failing to respond to cache pressure
mds.node-stor1(mds.0): Client NFS-1XX-XXX-XXX-103 failing to respond to cache pressure client_id: 9489331
[ERR] OSD_SCRUB_ERRORS: 5 scrub errors
[WRN] OSD_TOO_MANY_REPAIRS: Too many repaired reads on 1 OSDs
osd.11 had 30 reads repaired
[ERR] PG_DAMAGED: Possible data damage: 3 pgs inconsistent
pg 10.f is active+clean+scrubbing+deep+inconsistent, acting [18,12,11]
pg 10.16 is active+clean+inconsistent, acting [11,1,23]
pg 10.1c is active+clean+inconsistent, acting [11,20,16]
ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 350.67114 root default
-5 87.66779 host node -stor1
6 hdd 14.61130 osd.6 up 1.00000 1.00000
7 hdd 14.61130 osd.7 up 1.00000 1.00000
8 hdd 14.61130 osd.8 up 1.00000 1.00000
9 hdd 14.61130 osd.9 up 1.00000 1.00000
10 hdd 14.61130 osd.10 up 1.00000 1.00000
11 hdd 14.61130 osd.11 up 1.00000 1.00000
-7 87.66779 host node-stor2
12 hdd 14.61130 osd.12 up 1.00000 1.00000
13 hdd 14.61130 osd.13 up 1.00000 1.00000
14 hdd 14.61130 osd.14 up 1.00000 1.00000
15 hdd 14.61130 osd.15 up 1.00000 1.00000
16 hdd 14.61130 osd.16 up 1.00000 1.00000
17 hdd 14.61130 osd.17 up 1.00000 1.00000
-3 87.66779 host node-stor3
0 hdd 14.61130 osd.0 up 1.00000 1.00000
1 hdd 14.61130 osd.1 up 1.00000 1.00000
2 hdd 14.61130 osd.2 up 1.00000 1.00000
3 hdd 14.61130 osd.3 up 1.00000 1.00000
4 hdd 14.61130 osd.4 up 1.00000 1.00000
5 hdd 14.61130 osd.5 up 1.00000 1.00000
-9 87.66779 host node-stor4
18 hdd 14.61130 osd.18 up 1.00000 1.00000
19 hdd 14.61130 osd.19 up 1.00000 1.00000
20 hdd 14.61130 osd.20 up 1.00000 1.00000
21 hdd 14.61130 osd.21 up 1.00000 1.00000
22 hdd 14.61130 osd.22 up 1.00000 1.00000
23 hdd 14.61130 osd.23 up 1.00000 1.00000
ceph osd df
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS
6 hdd 14.61130 1.00000 15 TiB 1.1 TiB 1.0 TiB 0 B 2.0 GiB 14 TiB 7.40 0.66 20 up
7 hdd 14.61130 1.00000 15 TiB 1.9 TiB 1.8 TiB 3 KiB 3.7 GiB 13 TiB 13.02 1.17 27 up
8 hdd 14.61130 1.00000 15 TiB 1.2 TiB 1.2 TiB 3 KiB 2.4 GiB 13 TiB 8.49 0.76 19 up
9 hdd 14.61130 1.00000 15 TiB 2.2 TiB 2.1 TiB 283 KiB 4.3 GiB 12 TiB 14.96 1.34 30 up
10 hdd 14.61130 1.00000 15 TiB 1.4 TiB 1.3 TiB 0 B 2.7 GiB 13 TiB 9.29 0.83 19 up
11 hdd 14.61130 1.00000 15 TiB 2.6 TiB 2.6 TiB 0 B 5.1 GiB 12 TiB 17.88 1.60 29 up
12 hdd 14.61130 1.00000 15 TiB 1.7 TiB 1.7 TiB 0 B 3.1 GiB 13 TiB 11.97 1.07 24 up
13 hdd 14.61130 1.00000 15 TiB 2.0 TiB 1.9 TiB 3 KiB 3.8 GiB 13 TiB 13.59 1.22 27 up
14 hdd 14.61130 1.00000 15 TiB 1.1 TiB 1.0 TiB 0 B 2.0 GiB 14 TiB 7.41 0.66 17 up
15 hdd 14.61130 1.00000 15 TiB 1.5 TiB 1.5 TiB 0 B 2.8 GiB 13 TiB 10.36 0.93 25 up
16 hdd 14.61130 1.00000 15 TiB 583 GiB 523 GiB 0 B 1.3 GiB 14 TiB 3.90 0.35 19 up
17 hdd 14.61130 1.00000 15 TiB 1.6 TiB 1.5 TiB 27 KiB 2.9 GiB 13 TiB 10.92 0.98 23 up
0 hdd 14.61130 1.00000 15 TiB 2.6 TiB 2.6 TiB 248 KiB 5.0 GiB 12 TiB 17.87 1.60 24 up
1 hdd 14.61130 1.00000 15 TiB 1.9 TiB 1.8 TiB 0 B 3.6 GiB 13 TiB 12.78 1.14 32 up
2 hdd 14.61130 1.00000 15 TiB 1.1 TiB 1.0 TiB 0 B 2.2 GiB 14 TiB 7.41 0.66 23 up
3 hdd 14.61130 1.00000 15 TiB 948 GiB 888 GiB 28 KiB 1.9 GiB 14 TiB 6.33 0.57 27 up
4 hdd 14.61130 1.00000 15 TiB 1.5 TiB 1.5 TiB 4 KiB 2.8 GiB 13 TiB 10.35 0.93 20 up
5 hdd 14.61130 1.00000 15 TiB 1.2 TiB 1.2 TiB 0 B 2.3 GiB 13 TiB 8.50 0.76 16 up
18 hdd 14.61130 1.00000 15 TiB 1.5 TiB 1.4 TiB 0 B 2.8 GiB 13 TiB 10.32 0.92 31 up
19 hdd 14.61130 1.00000 15 TiB 1.5 TiB 1.5 TiB 0 B 3 GiB 13 TiB 10.36 0.93 20 up
20 hdd 14.61130 1.00000 15 TiB 1.9 TiB 1.8 TiB 0 B 3.9 GiB 13 TiB 13.04 1.17 27 up
21 hdd 14.61130 1.00000 15 TiB 1.4 TiB 1.3 TiB 0 B 2.7 GiB 13 TiB 9.58 0.86 17 up
22 hdd 14.61130 1.00000 15 TiB 3.4 TiB 3.3 TiB 291 KiB 6.0 GiB 11 TiB 23.01 2.06 41 up
23 hdd 14.61130 1.00000 15 TiB 1.4 TiB 1.3 TiB 27 KiB 2.8 GiB 13 TiB 9.32 0.83 22 up
TOTAL 351 TiB 39 TiB 38 TiB 922 KiB 75 GiB 312 TiB 11.17
MIN/MAX VAR: 0.35/2.06 STDDEV: 4.11
rados list-inconsistent-obj 10.f --format=json-pretty
{
"epoch": 3539,
"inconsistents": [
{
"object": {
"name": "rbd_data.22ceee723c5c.00000000000121eb",
"nspace": "",
"locator": "",
"snap": "head",
"version": 8421742
},
"errors": [],
"union_shard_errors": [
"read_error"
],
"selected_object_info": {
"oid": {
"oid": "rbd_data.22ceee723c5c.00000000000121eb",
"key": "",
"snapid": -2,
"hash": 1737665103,
"max": 0,
"pool": 10,
"namespace": ""
},
"version": "3554'8421742",
"prior_version": "3554'8421426",
"last_reqid": "client.143219.0:563462592",
"user_version": 8421742,
"size": 4194304,
"mtime": "2025-01-09T06:07:44.952161+0100",
"local_mtime": "2025-01-09T06:07:44.953033+0100",
"lost": 0,
"flags": [
"dirty",
"omap_digest"
],
"truncate_seq": 0,
"truncate_size": 0,
"data_digest": "0xffffffff",
"omap_digest": "0xffffffff",
"expected_object_size": 4194304,
"expected_write_size": 4194304,
"alloc_hint_flags": 0,
"manifest": {
"type": 0
},
"watchers": {}
},
"shards": [
{
"osd": 11,
"primary": false,
"errors": [
"read_error"
],
"size": 4194304
},
{
"osd": 12,
"primary": false,
"errors": [],
"size": 4194304,
"omap_digest": "0xffffffff",
"data_digest": "0xf4bf72f2"
},
{
"osd": 18,
"primary": true,
"errors": [],
"size": 4194304,
"omap_digest": "0xffffffff",
"data_digest": "0xf4bf72f2"
}
]
},
{
"object": {
"name": "rbd_data.261b03a6aa264.0000000000009a34",
"nspace": "",
"locator": "",
"snap": "head",
"version": 8351559
},
"errors": [],
"union_shard_errors": [
"read_error"
],
"selected_object_info": {
"oid": {
"oid": "rbd_data.261b03a6aa264.0000000000009a34",
"key": "",
"snapid": -2,
"hash": 2089893551,
"max": 0,
"pool": 10,
"namespace": ""
},
"version": "3524'8351559",
"prior_version": "3524'8351558",
"last_reqid": "client.143219.0:556722557",
"user_version": 8351559,
"size": 4194304,
"mtime": "2025-01-08T20:56:46.225679+0100",
"local_mtime": "2025-01-08T20:56:46.225556+0100",
"lost": 0,
"flags": [
"dirty",
"omap_digest"
],
"truncate_seq": 0,
"truncate_size": 0,
"data_digest": "0xffffffff",
"omap_digest": "0xffffffff",
"expected_object_size": 4194304,
"expected_write_size": 4194304,
"alloc_hint_flags": 0,
"manifest": {
"type": 0
},
"watchers": {}
},
"shards": [
{
"osd": 11,
"primary": false,
"errors": [
"read_error"
],
"size": 4194304
},
{
"osd": 12,
"primary": false,
"errors": [],
"size": 4194304,
"omap_digest": "0xffffffff",
"data_digest": "0x66b5055b"
},
{
"osd": 18,
"primary": true,
"errors": [],
"size": 4194304,
"omap_digest": "0xffffffff",
"data_digest": "0x66b5055b"
}
]
}
]
}
rados list-inconsistent-obj 10.16 --format=json-pretty
{
"epoch": 3533,
"inconsistents": [
{
"object": {
"name": "rbd_data.261b03a6aa264.0000000000008921",
"nspace": "",
"locator": "",
"snap": "head",
"version": 9756930
},
"errors": [],
"union_shard_errors": [
"read_error"
],
"selected_object_info": {
"oid": {
"oid": "rbd_data.261b03a6aa264.0000000000008921",
"key": "",
"snapid": -2,
"hash": 2078265174,
"max": 0,
"pool": 10,
"namespace": ""
},
"version": "3524'9756930",
"prior_version": "3524'9756929",
"last_reqid": "client.143219.0:556715005",
"user_version": 9756930,
"size": 4194304,
"mtime": "2025-01-08T20:56:42.556007+0100",
"local_mtime": "2025-01-08T20:56:42.556296+0100",
"lost": 0,
"flags": [
"dirty",
"omap_digest"
],
"truncate_seq": 0,
"truncate_size": 0,
"data_digest": "0xffffffff",
"omap_digest": "0xffffffff",
"expected_object_size": 4194304,
"expected_write_size": 4194304,
"alloc_hint_flags": 0,
"manifest": {
"type": 0
},
"watchers": {}
},
"shards": [
{
"osd": 1,
"primary": false,
"errors": [],
"size": 4194304,
"omap_digest": "0xffffffff",
"data_digest": "0x98f78947"
},
{
"osd": 11,
"primary": true,
"errors": [
"read_error"
],
"size": 4194304
},
{
"osd": 23,
"primary": false,
"errors": [],
"size": 4194304,
"omap_digest": "0xffffffff",
"data_digest": "0x98f78947"
}
]
}
]
}
rbd image 'image-00001':
size 3 TiB in 786432 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: 22ceee723c5c
block_name_prefix: rbd_data.22ceee723c5c
format: 2
features: layering
op_features:
flags:
create_timestamp: Wed Dec 11 07:47:26 2024
access_timestamp: Wed Dec 11 07:47:26 2024
modify_timestamp: Wed Dec 11 07:47:26 2024
root@orn-45d-stor1:~# rbd info image-00002
rbd image 'image-00002':
size 3 TiB in 786432 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: 261b03a6aa264
block_name_prefix: rbd_data.261b03a6aa264
format: 2
features: layering
op_features:
flags:
create_timestamp: Wed Dec 11 08:07:26 2024
access_timestamp: Wed Dec 11 08:07:26 2024
modify_timestamp: Wed Dec 11 08:07:26 2024
ceph device ls
DEVICE HOST:DEV DAEMONS WEAR LIFE EXPECTANCY
HDSTOR_-_HSAV25S_HS22093014A0F7E55 node-stor1:sda mon.node-stor1 53%
HDSTOR_-_HSAV25S_HS22101214A101BFE node-stor2:sda mon.node-stor2 51%
HDSTOR_-_HSAV25S_HS22101214A101C70 node-stor3:sdb mon.node-stor3 49%
Micron_5400_MTFD_500a07514428bbf1 node-stor4:sdi osd.18 osd.20 osd.22 0%
Micron_5400_MTFD_500a07514428bddf node-stor1:sdi osd.10 osd.6 osd.8 0%
Micron_5400_MTFD_500a0751442ad458 node-stor1:sdj osd.11 osd.7 osd.9 0%
Micron_5400_MTFD_500a0751442ad55e node-stor2:sdj osd.13 osd.15 osd.17 0%
Micron_5400_MTFD_500a0751442ad56a node-stor3:sdi osd.0 osd.2 osd.4 0%
Micron_5400_MTFD_500a0751442ad56e node-stor4:sdj osd.19 osd.21 osd.23 0%
Micron_5400_MTFD_500a0751442ad579 node-stor3:sdj osd.1 osd.3 osd.5 0%
Micron_5400_MTFD_500a0751442adf6c node-stor2:sdi osd.12 osd.14 osd.16 0%
Micron_7300_MTFDHBG1T9TDF_2022327BF1F2 node-stor2:nvme1n1 osd.13 osd.15 osd.17 0%
Micron_7300_MTFDHBG1T9TDF_214132747BDF node-stor2:nvme0n1 osd.12 osd.14 osd.16 35%
Micron_7450_MTFDKBG1T9TFR_221536FA4B7B node-stor1:nvme0n1 osd.11 osd.7 osd.9 0%
Micron_7450_MTFDKBG1T9TFR_221536FA4D68 node-stor4:nvme1n1 osd.18 osd.20 osd.22 0%
Micron_7450_MTFDKBG1T9TFR_221536FA4D78 node-stor4:nvme0n1 osd.19 osd.21 osd.23 51%
Micron_7450_MTFDKBG1T9TFR_221536FA4E67 node-stor3:nvme1n1 osd.0 osd.2 osd.4 0%
Micron_7450_MTFDKBG1T9TFR_221536FA4E7B node-stor3:nvme0n1 osd.1 osd.3 osd.5 58%
Micron_7450_MTFDKBG1T9TFR_22313BEC8410 node-stor1:nvme1n1 osd.10 osd.6 osd.8 48%
ST16000NM005J_5000c500ec286bdb node-stor4:sdf osd.20
ST16000NM005J_5000c500ec43256f node-stor1:sdf osd.11
ST16000NM005J_5000c500ec44302b node-stor2:sdf osd.17
ST16000NM005J_5000c500ec4437ef node-stor3:sdd osd.1
ST16000NM005J_5000c500ec44840f node-stor3:sdg osd.3
ST16000NM005J_5000c500ec44932b node-stor3:sdc osd.0
ST16000NM005J_5000c500ec53ba27 node-stor3:sde osd.2
ST16000NM005J_5000c500ec53ba97 node-stor1:sdd osd.7
ST16000NM005J_5000c500ec53bbf3 node-stor1:sde osd.10
ST16000NM005J_5000c500ec5411f3 node-stor1:sdc osd.6
ST16000NM005J_5000c500ec541507 node-stor1:sdg osd.8
ST16000NM005J_5000c500ec5415af node-stor3:sdh osd.4
ST16000NM005J_5000c500ec544fe3 node-stor1:sdh osd.9
ST16000NM005J_5000c500ec5451fb node-stor2:sde osd.14
ST16000NM005J_5000c500ec5770eb node-stor2:sdg osd.15
ST16000NM005J_5000c500ec5775c3 node-stor3:sdf osd.5
ST16000NM005J_5000c500ec57ec3b node-stor4:sdc osd.18
ST16000NM005J_5000c500ec57ec3f node-stor2:sdh osd.16
ST16000NM005J_5000c500ec57ec53 node-stor4:sde osd.21
ST16000NM005J_5000c500ec57ec67 node-stor4:sdg osd.22
ST16000NM005J_5000c500f29e867f node-stor4:sdh osd.23
ST16000NM005J_5000c500f2abea4f node-stor2:sdc osd.12
ST16000NM005J_5000c500f2ac0b9b node-stor4:sdd osd.19
ST16000NM005J_5000c500f2ac1373 node-stor2:sdd osd.13
smartctl --all /dev/sdf
smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.14.21-08-petasan] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Vendor: SEAGATE
Product: ST16000NM005J
Revision: ET04
Compliance: SPC-5
User Capacity: 16,000,900,661,248 bytes [16.0 TB]
Logical block size: 512 bytes
Physical block size: 4096 bytes
LU is fully provisioned
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Logical Unit id: 0x5000c500ec43256f
Serial number: ZR70Z2FC0000W34716QQ
Device type: disk
Transport protocol: SAS (SPL-3)
Local Time is: Tue Jan 14 03:30:40 2025 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK
Grown defects during certification <not available>
Total blocks reassigned during format <not available>
Total new blocks reassigned <not available>
Power on minutes since format <not available>
Current Drive Temperature: 33 C
Drive Trip Temperature: 60 C
Manufactured in week 42 of year 2023
Specified cycle count over device lifetime: 50000
Accumulated start-stop cycles: 13
Specified load-unload count over device lifetime: 600000
Accumulated load-unload cycles: 4209
Elements in grown defect list: 0
Vendor (Seagate Cache) information
Blocks sent to initiator = 717091008
Blocks received from initiator = 1661229904
Blocks read from cache and sent to initiator = 1742662296
Number of read and write commands whose size <= segment size = 15420970
Number of read and write commands whose size > segment size = 34014
Vendor (Seagate/Hitachi) factory information
number of hours powered up = 6537.55
number of minutes until next internal SMART test = 56
Error counter log:
Errors Corrected by Total Correction Gigabytes Total
ECC rereads/ errors algorithm processed uncorrected
fast | delayed rewrites corrected invocations [10^9 bytes] errors
read: 0 0 0 0 0 15760.315 0
write: 0 0 0 0 0 850.911 0
Non-medium error count: 0
[GLTSD (Global Logging Target Save Disable) set. Enable Save with '-S on']
No Self-tests have been logged
smartctl --all /dev/sdc
smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.14.21-08-petasan] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Vendor: SEAGATE
Product: ST16000NM005J
Revision: ET04
Compliance: SPC-5
User Capacity: 16,000,900,661,248 bytes [16.0 TB]
Logical block size: 512 bytes
Physical block size: 4096 bytes
LU is fully provisioned
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Logical Unit id: 0x5000c500ec5411f3
Serial number: ZR71045F0000W3474RQT
Device type: disk
Transport protocol: SAS (SPL-3)
Local Time is: Tue Jan 14 03:31:47 2025 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK
Grown defects during certification <not available>
Total blocks reassigned during format <not available>
Total new blocks reassigned <not available>
Power on minutes since format <not available>
Current Drive Temperature: 29 C
Drive Trip Temperature: 60 C
Manufactured in week 42 of year 2023
Specified cycle count over device lifetime: 50000
Accumulated start-stop cycles: 11
Specified load-unload count over device lifetime: 600000
Accumulated load-unload cycles: 6097
Elements in grown defect list: 0
Vendor (Seagate Cache) information
Blocks sent to initiator = 3487322904
Blocks received from initiator = 3137788280
Blocks read from cache and sent to initiator = 2103778790
Number of read and write commands whose size <= segment size = 11493458
Number of read and write commands whose size > segment size = 26103
Vendor (Seagate/Hitachi) factory information
number of hours powered up = 6537.57
number of minutes until next internal SMART test = 14
Error counter log:
Errors Corrected by Total Correction Gigabytes Total
ECC rereads/ errors algorithm processed uncorrected
fast | delayed rewrites corrected invocations [10^9 bytes] errors
read: 0 0 0 0 0 14979.649 0
write: 0 0 0 0 0 3808.633 0
Non-medium error count: 0
[GLTSD (Global Logging Target Save Disable) set. Enable Save with '-S on']
No Self-tests have been logged
journalctl -f -u ceph-osd@11
-- Logs begin at Sun 2025-01-05 12:21:06 CET. --
Jan 14 01:48:48 node-stor1 ceph-osd[5967]: 2025-01-14T01:48:48.271+0100 7f2196b4b700 -1 log_channel(cluster) log [ERR] : 10.1c deep-scrub 0 missing, 2 inconsistent objects
Jan 14 01:48:48 node-stor1 ceph-osd[5967]: 2025-01-14T01:48:48.271+0100 7f2196b4b700 -1 log_channel(cluster) log [ERR] : 10.1c deep-scrub 2 errors
Jan 14 02:19:36 node-stor1 ceph-osd[5967]: 2025-01-14T02:19:36.929+0100 7f219834e700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x9000, got 0x108fe2eb, expected 0x95cc874a, device location [0x333a4d09000~1000], logical extent 0x169000~1000, object #10:f25549e6:::rbd_data.22ceee723c5c.00000000000121eb:head#
Jan 14 02:19:36 node-stor1 ceph-osd[5967]: 2025-01-14T02:19:36.929+0100 7f219834e700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x9000, got 0x108fe2eb, expected 0x95cc874a, device location [0x333a4d09000~1000], logical extent 0x169000~1000, object #10:f25549e6:::rbd_data.22ceee723c5c.00000000000121eb:head#
Jan 14 02:19:36 node-stor1 ceph-osd[5967]: 2025-01-14T02:19:36.933+0100 7f219834e700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x9000, got 0x108fe2eb, expected 0x95cc874a, device location [0x333a4d09000~1000], logical extent 0x169000~1000, object #10:f25549e6:::rbd_data.22ceee723c5c.00000000000121eb:head#
Jan 14 02:19:36 node-stor1 ceph-osd[5967]: 2025-01-14T02:19:36.933+0100 7f219834e700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x9000, got 0x108fe2eb, expected 0x95cc874a, device location [0x333a4d09000~1000], logical extent 0x169000~1000, object #10:f25549e6:::rbd_data.22ceee723c5c.00000000000121eb:head#
Jan 14 02:57:08 node-stor1 ceph-osd[5967]: 2025-01-14T02:57:08.891+0100 7f219834e700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x0, got 0xb21969fe, expected 0xea08e29, device location [0x484b14c0000~1000], logical extent 0x380000~1000, object #10:f57c893e:::rbd_data.261b03a6aa264.0000000000009a34:head#
Jan 14 02:57:08 node-stor1 ceph-osd[5967]: 2025-01-14T02:57:08.903+0100 7f219834e700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x0, got 0xb21969fe, expected 0xea08e29, device location [0x484b14c0000~1000], logical extent 0x380000~1000, object #10:f57c893e:::rbd_data.261b03a6aa264.0000000000009a34:head#
Jan 14 02:57:08 node-stor1 ceph-osd[5967]: 2025-01-14T02:57:08.903+0100 7f219834e700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x0, got 0xb21969fe, expected 0xea08e29, device location [0x484b14c0000~1000], logical extent 0x380000~1000, object #10:f57c893e:::rbd_data.261b03a6aa264.0000000000009a34:head#
Jan 14 02:57:08 node-stor1 ceph-osd[5967]: 2025-01-14T02:57:08.903+0100 7f219834e700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x0, got 0xb21969fe, expected 0xea08e29, device location [0x484b14c0000~1000], logical extent 0x380000~1000, object #10:f57c893e:::rbd_data.261b03a6aa264.0000000000009a34:head#
Jan 14 03:54:48 node-stor1 ceph-osd[5967]: 2025-01-14T03:54:48.707+0100 7f2196b4b700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x1000, got 0xc8021cc4, expected 0x6706be76, device location [0x45f6c021000~1000], logical extent 0x281000~1000, object #10:3a57c27b:::rbd_data.261b03a6aa264.0000000000007add:head#
Jan 14 03:54:48 node-stor1 ceph-osd[5967]: 2025-01-14T03:54:48.707+0100 7f2196b4b700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x1000, got 0xc8021cc4, expected 0x6706be76, device location [0x45f6c021000~1000], logical extent 0x281000~1000, object #10:3a57c27b:::rbd_data.261b03a6aa264.0000000000007add:head#
Jan 14 03:54:48 node-stor1 ceph-osd[5967]: 2025-01-14T03:54:48.711+0100 7f2196b4b700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x1000, got 0xc8021cc4, expected 0x6706be76, device location [0x45f6c021000~1000], logical extent 0x281000~1000, object #10:3a57c27b:::rbd_data.261b03a6aa264.0000000000007add:head#
Jan 14 03:54:48 node-stor1 ceph-osd[5967]: 2025-01-14T03:54:48.711+0100 7f2196b4b700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x1000, got 0xc8021cc4, expected 0x6706be76, device location [0x45f6c021000~1000], logical extent 0x281000~1000, object #10:3a57c27b:::rbd_data.261b03a6aa264.0000000000007add:head#
Jan 14 03:54:50 node-stor1 ceph-osd[5967]: 2025-01-14T03:54:50.075+0100 7f2196b4b700 -1 log_channel(cluster) log [ERR] : 10.1c shard 11 soid 10:3a57c27b:::rbd_data.261b03a6aa264.0000000000007add:head : candidate had a read error
Last edited on January 14, 2025, 3:20 am by lberrezoug · #1
lberrezoug
5 Posts
January 14, 2025, 2:22 pmQuote from lberrezoug on January 14, 2025, 2:22 pmUPDATE:
I solved my issue by restarting the osd.11 and running the PG repair again. Since 10 hours the state of the cluster is OK and I hope it will stay that way 🙂
UPDATE:
I solved my issue by restarting the osd.11 and running the PG repair again. Since 10 hours the state of the cluster is OK and I hope it will stay that way 🙂
Last edited on January 14, 2025, 2:22 pm by lberrezoug · #2
pgs inconsistent
lberrezoug
5 Posts
January 14, 2025, 2:08 am
Quote from lberrezoug on January 14, 2025, 2:08 amHi,
I'm looking for help with an issue where two PGs are in an inconsistent state, and the commands
ceph pg deep-scrub $pgidandceph pg repair $pgidseem unable to resolve it.From further analysis using the command
rados list-inconsistent-obj $pgid --format=json-pretty, it appears that:
- The first PG, identified as 10.f, is associated with Primary OSD 11.
- The second PG, identified as 10.16, is associated with Primary OSD 18.
Both OSDs are in the UP state, and the SMART reports for the corresponding disks show no anomalies. However, even after running
ceph osd repair all, the problem persists.Using
rados list-inconsistent-obj $pgid --format=json-prettyand therbd info imagecommand, I found that these two PGs are linked to two RADOS images "‘image-00001, image-00002" used by VMware.I'm at a bit of a loss, and any help or recommendations would be greatly appreciated. I understand that this issue is not directly related to PetaSAN but rather to Ceph. However, I kindly ask for your assistance in finding a solution.
Best regards,
For more details:
- I have 4 identical nodes in terms of hardware.
- Each node has 6 hdd disks for data and 2 ssd disks for journaling and 2nvme for cache
- All pools are in replication mode 3.
- My PetaSAN cluster was deployed less than six months ago.
- At the time of writing this message i noticed that i have another pg 10.1c in an inconsistent state, i'm trying to fix it and i hope it doesn't persist like the other two. it also affects the rados image ‘image-00002’ and other OSDs but still primary OSD 11.
ceph health detail HEALTH_ERR 1 clients failing to respond to cache pressure; 5 scrub errors; Too many repaired reads on 1 OSDs; Possible data damage: 3 pgs inconsistent [WRN] MDS_CLIENT_RECALL: 1 clients failing to respond to cache pressure mds.node-stor1(mds.0): Client NFS-1XX-XXX-XXX-103 failing to respond to cache pressure client_id: 9489331 [ERR] OSD_SCRUB_ERRORS: 5 scrub errors [WRN] OSD_TOO_MANY_REPAIRS: Too many repaired reads on 1 OSDs osd.11 had 30 reads repaired [ERR] PG_DAMAGED: Possible data damage: 3 pgs inconsistent pg 10.f is active+clean+scrubbing+deep+inconsistent, acting [18,12,11] pg 10.16 is active+clean+inconsistent, acting [11,1,23] pg 10.1c is active+clean+inconsistent, acting [11,20,16]ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 350.67114 root default -5 87.66779 host node -stor1 6 hdd 14.61130 osd.6 up 1.00000 1.00000 7 hdd 14.61130 osd.7 up 1.00000 1.00000 8 hdd 14.61130 osd.8 up 1.00000 1.00000 9 hdd 14.61130 osd.9 up 1.00000 1.00000 10 hdd 14.61130 osd.10 up 1.00000 1.00000 11 hdd 14.61130 osd.11 up 1.00000 1.00000 -7 87.66779 host node-stor2 12 hdd 14.61130 osd.12 up 1.00000 1.00000 13 hdd 14.61130 osd.13 up 1.00000 1.00000 14 hdd 14.61130 osd.14 up 1.00000 1.00000 15 hdd 14.61130 osd.15 up 1.00000 1.00000 16 hdd 14.61130 osd.16 up 1.00000 1.00000 17 hdd 14.61130 osd.17 up 1.00000 1.00000 -3 87.66779 host node-stor3 0 hdd 14.61130 osd.0 up 1.00000 1.00000 1 hdd 14.61130 osd.1 up 1.00000 1.00000 2 hdd 14.61130 osd.2 up 1.00000 1.00000 3 hdd 14.61130 osd.3 up 1.00000 1.00000 4 hdd 14.61130 osd.4 up 1.00000 1.00000 5 hdd 14.61130 osd.5 up 1.00000 1.00000 -9 87.66779 host node-stor4 18 hdd 14.61130 osd.18 up 1.00000 1.00000 19 hdd 14.61130 osd.19 up 1.00000 1.00000 20 hdd 14.61130 osd.20 up 1.00000 1.00000 21 hdd 14.61130 osd.21 up 1.00000 1.00000 22 hdd 14.61130 osd.22 up 1.00000 1.00000 23 hdd 14.61130 osd.23 up 1.00000 1.00000 ceph osd df ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 6 hdd 14.61130 1.00000 15 TiB 1.1 TiB 1.0 TiB 0 B 2.0 GiB 14 TiB 7.40 0.66 20 up 7 hdd 14.61130 1.00000 15 TiB 1.9 TiB 1.8 TiB 3 KiB 3.7 GiB 13 TiB 13.02 1.17 27 up 8 hdd 14.61130 1.00000 15 TiB 1.2 TiB 1.2 TiB 3 KiB 2.4 GiB 13 TiB 8.49 0.76 19 up 9 hdd 14.61130 1.00000 15 TiB 2.2 TiB 2.1 TiB 283 KiB 4.3 GiB 12 TiB 14.96 1.34 30 up 10 hdd 14.61130 1.00000 15 TiB 1.4 TiB 1.3 TiB 0 B 2.7 GiB 13 TiB 9.29 0.83 19 up 11 hdd 14.61130 1.00000 15 TiB 2.6 TiB 2.6 TiB 0 B 5.1 GiB 12 TiB 17.88 1.60 29 up 12 hdd 14.61130 1.00000 15 TiB 1.7 TiB 1.7 TiB 0 B 3.1 GiB 13 TiB 11.97 1.07 24 up 13 hdd 14.61130 1.00000 15 TiB 2.0 TiB 1.9 TiB 3 KiB 3.8 GiB 13 TiB 13.59 1.22 27 up 14 hdd 14.61130 1.00000 15 TiB 1.1 TiB 1.0 TiB 0 B 2.0 GiB 14 TiB 7.41 0.66 17 up 15 hdd 14.61130 1.00000 15 TiB 1.5 TiB 1.5 TiB 0 B 2.8 GiB 13 TiB 10.36 0.93 25 up 16 hdd 14.61130 1.00000 15 TiB 583 GiB 523 GiB 0 B 1.3 GiB 14 TiB 3.90 0.35 19 up 17 hdd 14.61130 1.00000 15 TiB 1.6 TiB 1.5 TiB 27 KiB 2.9 GiB 13 TiB 10.92 0.98 23 up 0 hdd 14.61130 1.00000 15 TiB 2.6 TiB 2.6 TiB 248 KiB 5.0 GiB 12 TiB 17.87 1.60 24 up 1 hdd 14.61130 1.00000 15 TiB 1.9 TiB 1.8 TiB 0 B 3.6 GiB 13 TiB 12.78 1.14 32 up 2 hdd 14.61130 1.00000 15 TiB 1.1 TiB 1.0 TiB 0 B 2.2 GiB 14 TiB 7.41 0.66 23 up 3 hdd 14.61130 1.00000 15 TiB 948 GiB 888 GiB 28 KiB 1.9 GiB 14 TiB 6.33 0.57 27 up 4 hdd 14.61130 1.00000 15 TiB 1.5 TiB 1.5 TiB 4 KiB 2.8 GiB 13 TiB 10.35 0.93 20 up 5 hdd 14.61130 1.00000 15 TiB 1.2 TiB 1.2 TiB 0 B 2.3 GiB 13 TiB 8.50 0.76 16 up 18 hdd 14.61130 1.00000 15 TiB 1.5 TiB 1.4 TiB 0 B 2.8 GiB 13 TiB 10.32 0.92 31 up 19 hdd 14.61130 1.00000 15 TiB 1.5 TiB 1.5 TiB 0 B 3 GiB 13 TiB 10.36 0.93 20 up 20 hdd 14.61130 1.00000 15 TiB 1.9 TiB 1.8 TiB 0 B 3.9 GiB 13 TiB 13.04 1.17 27 up 21 hdd 14.61130 1.00000 15 TiB 1.4 TiB 1.3 TiB 0 B 2.7 GiB 13 TiB 9.58 0.86 17 up 22 hdd 14.61130 1.00000 15 TiB 3.4 TiB 3.3 TiB 291 KiB 6.0 GiB 11 TiB 23.01 2.06 41 up 23 hdd 14.61130 1.00000 15 TiB 1.4 TiB 1.3 TiB 27 KiB 2.8 GiB 13 TiB 9.32 0.83 22 up TOTAL 351 TiB 39 TiB 38 TiB 922 KiB 75 GiB 312 TiB 11.17 MIN/MAX VAR: 0.35/2.06 STDDEV: 4.11rados list-inconsistent-obj 10.f --format=json-pretty { "epoch": 3539, "inconsistents": [ { "object": { "name": "rbd_data.22ceee723c5c.00000000000121eb", "nspace": "", "locator": "", "snap": "head", "version": 8421742 }, "errors": [], "union_shard_errors": [ "read_error" ], "selected_object_info": { "oid": { "oid": "rbd_data.22ceee723c5c.00000000000121eb", "key": "", "snapid": -2, "hash": 1737665103, "max": 0, "pool": 10, "namespace": "" }, "version": "3554'8421742", "prior_version": "3554'8421426", "last_reqid": "client.143219.0:563462592", "user_version": 8421742, "size": 4194304, "mtime": "2025-01-09T06:07:44.952161+0100", "local_mtime": "2025-01-09T06:07:44.953033+0100", "lost": 0, "flags": [ "dirty", "omap_digest" ], "truncate_seq": 0, "truncate_size": 0, "data_digest": "0xffffffff", "omap_digest": "0xffffffff", "expected_object_size": 4194304, "expected_write_size": 4194304, "alloc_hint_flags": 0, "manifest": { "type": 0 }, "watchers": {} }, "shards": [ { "osd": 11, "primary": false, "errors": [ "read_error" ], "size": 4194304 }, { "osd": 12, "primary": false, "errors": [], "size": 4194304, "omap_digest": "0xffffffff", "data_digest": "0xf4bf72f2" }, { "osd": 18, "primary": true, "errors": [], "size": 4194304, "omap_digest": "0xffffffff", "data_digest": "0xf4bf72f2" } ] }, { "object": { "name": "rbd_data.261b03a6aa264.0000000000009a34", "nspace": "", "locator": "", "snap": "head", "version": 8351559 }, "errors": [], "union_shard_errors": [ "read_error" ], "selected_object_info": { "oid": { "oid": "rbd_data.261b03a6aa264.0000000000009a34", "key": "", "snapid": -2, "hash": 2089893551, "max": 0, "pool": 10, "namespace": "" }, "version": "3524'8351559", "prior_version": "3524'8351558", "last_reqid": "client.143219.0:556722557", "user_version": 8351559, "size": 4194304, "mtime": "2025-01-08T20:56:46.225679+0100", "local_mtime": "2025-01-08T20:56:46.225556+0100", "lost": 0, "flags": [ "dirty", "omap_digest" ], "truncate_seq": 0, "truncate_size": 0, "data_digest": "0xffffffff", "omap_digest": "0xffffffff", "expected_object_size": 4194304, "expected_write_size": 4194304, "alloc_hint_flags": 0, "manifest": { "type": 0 }, "watchers": {} }, "shards": [ { "osd": 11, "primary": false, "errors": [ "read_error" ], "size": 4194304 }, { "osd": 12, "primary": false, "errors": [], "size": 4194304, "omap_digest": "0xffffffff", "data_digest": "0x66b5055b" }, { "osd": 18, "primary": true, "errors": [], "size": 4194304, "omap_digest": "0xffffffff", "data_digest": "0x66b5055b" } ] } ] }rados list-inconsistent-obj 10.16 --format=json-pretty { "epoch": 3533, "inconsistents": [ { "object": { "name": "rbd_data.261b03a6aa264.0000000000008921", "nspace": "", "locator": "", "snap": "head", "version": 9756930 }, "errors": [], "union_shard_errors": [ "read_error" ], "selected_object_info": { "oid": { "oid": "rbd_data.261b03a6aa264.0000000000008921", "key": "", "snapid": -2, "hash": 2078265174, "max": 0, "pool": 10, "namespace": "" }, "version": "3524'9756930", "prior_version": "3524'9756929", "last_reqid": "client.143219.0:556715005", "user_version": 9756930, "size": 4194304, "mtime": "2025-01-08T20:56:42.556007+0100", "local_mtime": "2025-01-08T20:56:42.556296+0100", "lost": 0, "flags": [ "dirty", "omap_digest" ], "truncate_seq": 0, "truncate_size": 0, "data_digest": "0xffffffff", "omap_digest": "0xffffffff", "expected_object_size": 4194304, "expected_write_size": 4194304, "alloc_hint_flags": 0, "manifest": { "type": 0 }, "watchers": {} }, "shards": [ { "osd": 1, "primary": false, "errors": [], "size": 4194304, "omap_digest": "0xffffffff", "data_digest": "0x98f78947" }, { "osd": 11, "primary": true, "errors": [ "read_error" ], "size": 4194304 }, { "osd": 23, "primary": false, "errors": [], "size": 4194304, "omap_digest": "0xffffffff", "data_digest": "0x98f78947" } ] } ] } rbd image 'image-00001': size 3 TiB in 786432 objects order 22 (4 MiB objects) snapshot_count: 0 id: 22ceee723c5c block_name_prefix: rbd_data.22ceee723c5c format: 2 features: layering op_features: flags: create_timestamp: Wed Dec 11 07:47:26 2024 access_timestamp: Wed Dec 11 07:47:26 2024 modify_timestamp: Wed Dec 11 07:47:26 2024 root@orn-45d-stor1:~# rbd info image-00002 rbd image 'image-00002': size 3 TiB in 786432 objects order 22 (4 MiB objects) snapshot_count: 0 id: 261b03a6aa264 block_name_prefix: rbd_data.261b03a6aa264 format: 2 features: layering op_features: flags: create_timestamp: Wed Dec 11 08:07:26 2024 access_timestamp: Wed Dec 11 08:07:26 2024 modify_timestamp: Wed Dec 11 08:07:26 2024ceph device ls DEVICE HOST:DEV DAEMONS WEAR LIFE EXPECTANCY HDSTOR_-_HSAV25S_HS22093014A0F7E55 node-stor1:sda mon.node-stor1 53% HDSTOR_-_HSAV25S_HS22101214A101BFE node-stor2:sda mon.node-stor2 51% HDSTOR_-_HSAV25S_HS22101214A101C70 node-stor3:sdb mon.node-stor3 49% Micron_5400_MTFD_500a07514428bbf1 node-stor4:sdi osd.18 osd.20 osd.22 0% Micron_5400_MTFD_500a07514428bddf node-stor1:sdi osd.10 osd.6 osd.8 0% Micron_5400_MTFD_500a0751442ad458 node-stor1:sdj osd.11 osd.7 osd.9 0% Micron_5400_MTFD_500a0751442ad55e node-stor2:sdj osd.13 osd.15 osd.17 0% Micron_5400_MTFD_500a0751442ad56a node-stor3:sdi osd.0 osd.2 osd.4 0% Micron_5400_MTFD_500a0751442ad56e node-stor4:sdj osd.19 osd.21 osd.23 0% Micron_5400_MTFD_500a0751442ad579 node-stor3:sdj osd.1 osd.3 osd.5 0% Micron_5400_MTFD_500a0751442adf6c node-stor2:sdi osd.12 osd.14 osd.16 0% Micron_7300_MTFDHBG1T9TDF_2022327BF1F2 node-stor2:nvme1n1 osd.13 osd.15 osd.17 0% Micron_7300_MTFDHBG1T9TDF_214132747BDF node-stor2:nvme0n1 osd.12 osd.14 osd.16 35% Micron_7450_MTFDKBG1T9TFR_221536FA4B7B node-stor1:nvme0n1 osd.11 osd.7 osd.9 0% Micron_7450_MTFDKBG1T9TFR_221536FA4D68 node-stor4:nvme1n1 osd.18 osd.20 osd.22 0% Micron_7450_MTFDKBG1T9TFR_221536FA4D78 node-stor4:nvme0n1 osd.19 osd.21 osd.23 51% Micron_7450_MTFDKBG1T9TFR_221536FA4E67 node-stor3:nvme1n1 osd.0 osd.2 osd.4 0% Micron_7450_MTFDKBG1T9TFR_221536FA4E7B node-stor3:nvme0n1 osd.1 osd.3 osd.5 58% Micron_7450_MTFDKBG1T9TFR_22313BEC8410 node-stor1:nvme1n1 osd.10 osd.6 osd.8 48% ST16000NM005J_5000c500ec286bdb node-stor4:sdf osd.20 ST16000NM005J_5000c500ec43256f node-stor1:sdf osd.11 ST16000NM005J_5000c500ec44302b node-stor2:sdf osd.17 ST16000NM005J_5000c500ec4437ef node-stor3:sdd osd.1 ST16000NM005J_5000c500ec44840f node-stor3:sdg osd.3 ST16000NM005J_5000c500ec44932b node-stor3:sdc osd.0 ST16000NM005J_5000c500ec53ba27 node-stor3:sde osd.2 ST16000NM005J_5000c500ec53ba97 node-stor1:sdd osd.7 ST16000NM005J_5000c500ec53bbf3 node-stor1:sde osd.10 ST16000NM005J_5000c500ec5411f3 node-stor1:sdc osd.6 ST16000NM005J_5000c500ec541507 node-stor1:sdg osd.8 ST16000NM005J_5000c500ec5415af node-stor3:sdh osd.4 ST16000NM005J_5000c500ec544fe3 node-stor1:sdh osd.9 ST16000NM005J_5000c500ec5451fb node-stor2:sde osd.14 ST16000NM005J_5000c500ec5770eb node-stor2:sdg osd.15 ST16000NM005J_5000c500ec5775c3 node-stor3:sdf osd.5 ST16000NM005J_5000c500ec57ec3b node-stor4:sdc osd.18 ST16000NM005J_5000c500ec57ec3f node-stor2:sdh osd.16 ST16000NM005J_5000c500ec57ec53 node-stor4:sde osd.21 ST16000NM005J_5000c500ec57ec67 node-stor4:sdg osd.22 ST16000NM005J_5000c500f29e867f node-stor4:sdh osd.23 ST16000NM005J_5000c500f2abea4f node-stor2:sdc osd.12 ST16000NM005J_5000c500f2ac0b9b node-stor4:sdd osd.19 ST16000NM005J_5000c500f2ac1373 node-stor2:sdd osd.13smartctl --all /dev/sdf smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.14.21-08-petasan] (local build) Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Vendor: SEAGATE Product: ST16000NM005J Revision: ET04 Compliance: SPC-5 User Capacity: 16,000,900,661,248 bytes [16.0 TB] Logical block size: 512 bytes Physical block size: 4096 bytes LU is fully provisioned Rotation Rate: 7200 rpm Form Factor: 3.5 inches Logical Unit id: 0x5000c500ec43256f Serial number: ZR70Z2FC0000W34716QQ Device type: disk Transport protocol: SAS (SPL-3) Local Time is: Tue Jan 14 03:30:40 2025 CET SMART support is: Available - device has SMART capability. SMART support is: Enabled Temperature Warning: Enabled === START OF READ SMART DATA SECTION === SMART Health Status: OK Grown defects during certification <not available> Total blocks reassigned during format <not available> Total new blocks reassigned <not available> Power on minutes since format <not available> Current Drive Temperature: 33 C Drive Trip Temperature: 60 C Manufactured in week 42 of year 2023 Specified cycle count over device lifetime: 50000 Accumulated start-stop cycles: 13 Specified load-unload count over device lifetime: 600000 Accumulated load-unload cycles: 4209 Elements in grown defect list: 0 Vendor (Seagate Cache) information Blocks sent to initiator = 717091008 Blocks received from initiator = 1661229904 Blocks read from cache and sent to initiator = 1742662296 Number of read and write commands whose size <= segment size = 15420970 Number of read and write commands whose size > segment size = 34014 Vendor (Seagate/Hitachi) factory information number of hours powered up = 6537.55 number of minutes until next internal SMART test = 56 Error counter log: Errors Corrected by Total Correction Gigabytes Total ECC rereads/ errors algorithm processed uncorrected fast | delayed rewrites corrected invocations [10^9 bytes] errors read: 0 0 0 0 0 15760.315 0 write: 0 0 0 0 0 850.911 0 Non-medium error count: 0 [GLTSD (Global Logging Target Save Disable) set. Enable Save with '-S on'] No Self-tests have been loggedsmartctl --all /dev/sdc smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.14.21-08-petasan] (local build) Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Vendor: SEAGATE Product: ST16000NM005J Revision: ET04 Compliance: SPC-5 User Capacity: 16,000,900,661,248 bytes [16.0 TB] Logical block size: 512 bytes Physical block size: 4096 bytes LU is fully provisioned Rotation Rate: 7200 rpm Form Factor: 3.5 inches Logical Unit id: 0x5000c500ec5411f3 Serial number: ZR71045F0000W3474RQT Device type: disk Transport protocol: SAS (SPL-3) Local Time is: Tue Jan 14 03:31:47 2025 CET SMART support is: Available - device has SMART capability. SMART support is: Enabled Temperature Warning: Enabled === START OF READ SMART DATA SECTION === SMART Health Status: OK Grown defects during certification <not available> Total blocks reassigned during format <not available> Total new blocks reassigned <not available> Power on minutes since format <not available> Current Drive Temperature: 29 C Drive Trip Temperature: 60 C Manufactured in week 42 of year 2023 Specified cycle count over device lifetime: 50000 Accumulated start-stop cycles: 11 Specified load-unload count over device lifetime: 600000 Accumulated load-unload cycles: 6097 Elements in grown defect list: 0 Vendor (Seagate Cache) information Blocks sent to initiator = 3487322904 Blocks received from initiator = 3137788280 Blocks read from cache and sent to initiator = 2103778790 Number of read and write commands whose size <= segment size = 11493458 Number of read and write commands whose size > segment size = 26103 Vendor (Seagate/Hitachi) factory information number of hours powered up = 6537.57 number of minutes until next internal SMART test = 14 Error counter log: Errors Corrected by Total Correction Gigabytes Total ECC rereads/ errors algorithm processed uncorrected fast | delayed rewrites corrected invocations [10^9 bytes] errors read: 0 0 0 0 0 14979.649 0 write: 0 0 0 0 0 3808.633 0 Non-medium error count: 0 [GLTSD (Global Logging Target Save Disable) set. Enable Save with '-S on'] No Self-tests have been loggedjournalctl -f -u ceph-osd@11 -- Logs begin at Sun 2025-01-05 12:21:06 CET. -- Jan 14 01:48:48 node-stor1 ceph-osd[5967]: 2025-01-14T01:48:48.271+0100 7f2196b4b700 -1 log_channel(cluster) log [ERR] : 10.1c deep-scrub 0 missing, 2 inconsistent objects Jan 14 01:48:48 node-stor1 ceph-osd[5967]: 2025-01-14T01:48:48.271+0100 7f2196b4b700 -1 log_channel(cluster) log [ERR] : 10.1c deep-scrub 2 errors Jan 14 02:19:36 node-stor1 ceph-osd[5967]: 2025-01-14T02:19:36.929+0100 7f219834e700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x9000, got 0x108fe2eb, expected 0x95cc874a, device location [0x333a4d09000~1000], logical extent 0x169000~1000, object #10:f25549e6:::rbd_data.22ceee723c5c.00000000000121eb:head# Jan 14 02:19:36 node-stor1 ceph-osd[5967]: 2025-01-14T02:19:36.929+0100 7f219834e700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x9000, got 0x108fe2eb, expected 0x95cc874a, device location [0x333a4d09000~1000], logical extent 0x169000~1000, object #10:f25549e6:::rbd_data.22ceee723c5c.00000000000121eb:head# Jan 14 02:19:36 node-stor1 ceph-osd[5967]: 2025-01-14T02:19:36.933+0100 7f219834e700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x9000, got 0x108fe2eb, expected 0x95cc874a, device location [0x333a4d09000~1000], logical extent 0x169000~1000, object #10:f25549e6:::rbd_data.22ceee723c5c.00000000000121eb:head# Jan 14 02:19:36 node-stor1 ceph-osd[5967]: 2025-01-14T02:19:36.933+0100 7f219834e700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x9000, got 0x108fe2eb, expected 0x95cc874a, device location [0x333a4d09000~1000], logical extent 0x169000~1000, object #10:f25549e6:::rbd_data.22ceee723c5c.00000000000121eb:head# Jan 14 02:57:08 node-stor1 ceph-osd[5967]: 2025-01-14T02:57:08.891+0100 7f219834e700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x0, got 0xb21969fe, expected 0xea08e29, device location [0x484b14c0000~1000], logical extent 0x380000~1000, object #10:f57c893e:::rbd_data.261b03a6aa264.0000000000009a34:head# Jan 14 02:57:08 node-stor1 ceph-osd[5967]: 2025-01-14T02:57:08.903+0100 7f219834e700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x0, got 0xb21969fe, expected 0xea08e29, device location [0x484b14c0000~1000], logical extent 0x380000~1000, object #10:f57c893e:::rbd_data.261b03a6aa264.0000000000009a34:head# Jan 14 02:57:08 node-stor1 ceph-osd[5967]: 2025-01-14T02:57:08.903+0100 7f219834e700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x0, got 0xb21969fe, expected 0xea08e29, device location [0x484b14c0000~1000], logical extent 0x380000~1000, object #10:f57c893e:::rbd_data.261b03a6aa264.0000000000009a34:head# Jan 14 02:57:08 node-stor1 ceph-osd[5967]: 2025-01-14T02:57:08.903+0100 7f219834e700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x0, got 0xb21969fe, expected 0xea08e29, device location [0x484b14c0000~1000], logical extent 0x380000~1000, object #10:f57c893e:::rbd_data.261b03a6aa264.0000000000009a34:head# Jan 14 03:54:48 node-stor1 ceph-osd[5967]: 2025-01-14T03:54:48.707+0100 7f2196b4b700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x1000, got 0xc8021cc4, expected 0x6706be76, device location [0x45f6c021000~1000], logical extent 0x281000~1000, object #10:3a57c27b:::rbd_data.261b03a6aa264.0000000000007add:head# Jan 14 03:54:48 node-stor1 ceph-osd[5967]: 2025-01-14T03:54:48.707+0100 7f2196b4b700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x1000, got 0xc8021cc4, expected 0x6706be76, device location [0x45f6c021000~1000], logical extent 0x281000~1000, object #10:3a57c27b:::rbd_data.261b03a6aa264.0000000000007add:head# Jan 14 03:54:48 node-stor1 ceph-osd[5967]: 2025-01-14T03:54:48.711+0100 7f2196b4b700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x1000, got 0xc8021cc4, expected 0x6706be76, device location [0x45f6c021000~1000], logical extent 0x281000~1000, object #10:3a57c27b:::rbd_data.261b03a6aa264.0000000000007add:head# Jan 14 03:54:48 node-stor1 ceph-osd[5967]: 2025-01-14T03:54:48.711+0100 7f2196b4b700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x1000, got 0xc8021cc4, expected 0x6706be76, device location [0x45f6c021000~1000], logical extent 0x281000~1000, object #10:3a57c27b:::rbd_data.261b03a6aa264.0000000000007add:head# Jan 14 03:54:50 node-stor1 ceph-osd[5967]: 2025-01-14T03:54:50.075+0100 7f2196b4b700 -1 log_channel(cluster) log [ERR] : 10.1c shard 11 soid 10:3a57c27b:::rbd_data.261b03a6aa264.0000000000007add:head : candidate had a read error
Hi,
I'm looking for help with an issue where two PGs are in an inconsistent state, and the commands ceph pg deep-scrub $pgid and ceph pg repair $pgid seem unable to resolve it.
From further analysis using the command rados list-inconsistent-obj $pgid --format=json-pretty, it appears that:
- The first PG, identified as 10.f, is associated with Primary OSD 11.
- The second PG, identified as 10.16, is associated with Primary OSD 18.
Both OSDs are in the UP state, and the SMART reports for the corresponding disks show no anomalies. However, even after running ceph osd repair all, the problem persists.
Using rados list-inconsistent-obj $pgid --format=json-pretty and the rbd info image command, I found that these two PGs are linked to two RADOS images "‘image-00001, image-00002" used by VMware.
I'm at a bit of a loss, and any help or recommendations would be greatly appreciated. I understand that this issue is not directly related to PetaSAN but rather to Ceph. However, I kindly ask for your assistance in finding a solution.
Best regards,
For more details:
- I have 4 identical nodes in terms of hardware.
- Each node has 6 hdd disks for data and 2 ssd disks for journaling and 2nvme for cache
- All pools are in replication mode 3.
- My PetaSAN cluster was deployed less than six months ago.
- At the time of writing this message i noticed that i have another pg 10.1c in an inconsistent state, i'm trying to fix it and i hope it doesn't persist like the other two. it also affects the rados image ‘image-00002’ and other OSDs but still primary OSD 11.
ceph health detail HEALTH_ERR 1 clients failing to respond to cache pressure; 5 scrub errors; Too many repaired reads on 1 OSDs; Possible data damage: 3 pgs inconsistent [WRN] MDS_CLIENT_RECALL: 1 clients failing to respond to cache pressure mds.node-stor1(mds.0): Client NFS-1XX-XXX-XXX-103 failing to respond to cache pressure client_id: 9489331 [ERR] OSD_SCRUB_ERRORS: 5 scrub errors [WRN] OSD_TOO_MANY_REPAIRS: Too many repaired reads on 1 OSDs osd.11 had 30 reads repaired [ERR] PG_DAMAGED: Possible data damage: 3 pgs inconsistent pg 10.f is active+clean+scrubbing+deep+inconsistent, acting [18,12,11] pg 10.16 is active+clean+inconsistent, acting [11,1,23] pg 10.1c is active+clean+inconsistent, acting [11,20,16]
ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 350.67114 root default -5 87.66779 host node -stor1 6 hdd 14.61130 osd.6 up 1.00000 1.00000 7 hdd 14.61130 osd.7 up 1.00000 1.00000 8 hdd 14.61130 osd.8 up 1.00000 1.00000 9 hdd 14.61130 osd.9 up 1.00000 1.00000 10 hdd 14.61130 osd.10 up 1.00000 1.00000 11 hdd 14.61130 osd.11 up 1.00000 1.00000 -7 87.66779 host node-stor2 12 hdd 14.61130 osd.12 up 1.00000 1.00000 13 hdd 14.61130 osd.13 up 1.00000 1.00000 14 hdd 14.61130 osd.14 up 1.00000 1.00000 15 hdd 14.61130 osd.15 up 1.00000 1.00000 16 hdd 14.61130 osd.16 up 1.00000 1.00000 17 hdd 14.61130 osd.17 up 1.00000 1.00000 -3 87.66779 host node-stor3 0 hdd 14.61130 osd.0 up 1.00000 1.00000 1 hdd 14.61130 osd.1 up 1.00000 1.00000 2 hdd 14.61130 osd.2 up 1.00000 1.00000 3 hdd 14.61130 osd.3 up 1.00000 1.00000 4 hdd 14.61130 osd.4 up 1.00000 1.00000 5 hdd 14.61130 osd.5 up 1.00000 1.00000 -9 87.66779 host node-stor4 18 hdd 14.61130 osd.18 up 1.00000 1.00000 19 hdd 14.61130 osd.19 up 1.00000 1.00000 20 hdd 14.61130 osd.20 up 1.00000 1.00000 21 hdd 14.61130 osd.21 up 1.00000 1.00000 22 hdd 14.61130 osd.22 up 1.00000 1.00000 23 hdd 14.61130 osd.23 up 1.00000 1.00000 ceph osd df ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 6 hdd 14.61130 1.00000 15 TiB 1.1 TiB 1.0 TiB 0 B 2.0 GiB 14 TiB 7.40 0.66 20 up 7 hdd 14.61130 1.00000 15 TiB 1.9 TiB 1.8 TiB 3 KiB 3.7 GiB 13 TiB 13.02 1.17 27 up 8 hdd 14.61130 1.00000 15 TiB 1.2 TiB 1.2 TiB 3 KiB 2.4 GiB 13 TiB 8.49 0.76 19 up 9 hdd 14.61130 1.00000 15 TiB 2.2 TiB 2.1 TiB 283 KiB 4.3 GiB 12 TiB 14.96 1.34 30 up 10 hdd 14.61130 1.00000 15 TiB 1.4 TiB 1.3 TiB 0 B 2.7 GiB 13 TiB 9.29 0.83 19 up 11 hdd 14.61130 1.00000 15 TiB 2.6 TiB 2.6 TiB 0 B 5.1 GiB 12 TiB 17.88 1.60 29 up 12 hdd 14.61130 1.00000 15 TiB 1.7 TiB 1.7 TiB 0 B 3.1 GiB 13 TiB 11.97 1.07 24 up 13 hdd 14.61130 1.00000 15 TiB 2.0 TiB 1.9 TiB 3 KiB 3.8 GiB 13 TiB 13.59 1.22 27 up 14 hdd 14.61130 1.00000 15 TiB 1.1 TiB 1.0 TiB 0 B 2.0 GiB 14 TiB 7.41 0.66 17 up 15 hdd 14.61130 1.00000 15 TiB 1.5 TiB 1.5 TiB 0 B 2.8 GiB 13 TiB 10.36 0.93 25 up 16 hdd 14.61130 1.00000 15 TiB 583 GiB 523 GiB 0 B 1.3 GiB 14 TiB 3.90 0.35 19 up 17 hdd 14.61130 1.00000 15 TiB 1.6 TiB 1.5 TiB 27 KiB 2.9 GiB 13 TiB 10.92 0.98 23 up 0 hdd 14.61130 1.00000 15 TiB 2.6 TiB 2.6 TiB 248 KiB 5.0 GiB 12 TiB 17.87 1.60 24 up 1 hdd 14.61130 1.00000 15 TiB 1.9 TiB 1.8 TiB 0 B 3.6 GiB 13 TiB 12.78 1.14 32 up 2 hdd 14.61130 1.00000 15 TiB 1.1 TiB 1.0 TiB 0 B 2.2 GiB 14 TiB 7.41 0.66 23 up 3 hdd 14.61130 1.00000 15 TiB 948 GiB 888 GiB 28 KiB 1.9 GiB 14 TiB 6.33 0.57 27 up 4 hdd 14.61130 1.00000 15 TiB 1.5 TiB 1.5 TiB 4 KiB 2.8 GiB 13 TiB 10.35 0.93 20 up 5 hdd 14.61130 1.00000 15 TiB 1.2 TiB 1.2 TiB 0 B 2.3 GiB 13 TiB 8.50 0.76 16 up 18 hdd 14.61130 1.00000 15 TiB 1.5 TiB 1.4 TiB 0 B 2.8 GiB 13 TiB 10.32 0.92 31 up 19 hdd 14.61130 1.00000 15 TiB 1.5 TiB 1.5 TiB 0 B 3 GiB 13 TiB 10.36 0.93 20 up 20 hdd 14.61130 1.00000 15 TiB 1.9 TiB 1.8 TiB 0 B 3.9 GiB 13 TiB 13.04 1.17 27 up 21 hdd 14.61130 1.00000 15 TiB 1.4 TiB 1.3 TiB 0 B 2.7 GiB 13 TiB 9.58 0.86 17 up 22 hdd 14.61130 1.00000 15 TiB 3.4 TiB 3.3 TiB 291 KiB 6.0 GiB 11 TiB 23.01 2.06 41 up 23 hdd 14.61130 1.00000 15 TiB 1.4 TiB 1.3 TiB 27 KiB 2.8 GiB 13 TiB 9.32 0.83 22 up TOTAL 351 TiB 39 TiB 38 TiB 922 KiB 75 GiB 312 TiB 11.17 MIN/MAX VAR: 0.35/2.06 STDDEV: 4.11
rados list-inconsistent-obj 10.f --format=json-pretty
{
"epoch": 3539,
"inconsistents": [
{
"object": {
"name": "rbd_data.22ceee723c5c.00000000000121eb",
"nspace": "",
"locator": "",
"snap": "head",
"version": 8421742
},
"errors": [],
"union_shard_errors": [
"read_error"
],
"selected_object_info": {
"oid": {
"oid": "rbd_data.22ceee723c5c.00000000000121eb",
"key": "",
"snapid": -2,
"hash": 1737665103,
"max": 0,
"pool": 10,
"namespace": ""
},
"version": "3554'8421742",
"prior_version": "3554'8421426",
"last_reqid": "client.143219.0:563462592",
"user_version": 8421742,
"size": 4194304,
"mtime": "2025-01-09T06:07:44.952161+0100",
"local_mtime": "2025-01-09T06:07:44.953033+0100",
"lost": 0,
"flags": [
"dirty",
"omap_digest"
],
"truncate_seq": 0,
"truncate_size": 0,
"data_digest": "0xffffffff",
"omap_digest": "0xffffffff",
"expected_object_size": 4194304,
"expected_write_size": 4194304,
"alloc_hint_flags": 0,
"manifest": {
"type": 0
},
"watchers": {}
},
"shards": [
{
"osd": 11,
"primary": false,
"errors": [
"read_error"
],
"size": 4194304
},
{
"osd": 12,
"primary": false,
"errors": [],
"size": 4194304,
"omap_digest": "0xffffffff",
"data_digest": "0xf4bf72f2"
},
{
"osd": 18,
"primary": true,
"errors": [],
"size": 4194304,
"omap_digest": "0xffffffff",
"data_digest": "0xf4bf72f2"
}
]
},
{
"object": {
"name": "rbd_data.261b03a6aa264.0000000000009a34",
"nspace": "",
"locator": "",
"snap": "head",
"version": 8351559
},
"errors": [],
"union_shard_errors": [
"read_error"
],
"selected_object_info": {
"oid": {
"oid": "rbd_data.261b03a6aa264.0000000000009a34",
"key": "",
"snapid": -2,
"hash": 2089893551,
"max": 0,
"pool": 10,
"namespace": ""
},
"version": "3524'8351559",
"prior_version": "3524'8351558",
"last_reqid": "client.143219.0:556722557",
"user_version": 8351559,
"size": 4194304,
"mtime": "2025-01-08T20:56:46.225679+0100",
"local_mtime": "2025-01-08T20:56:46.225556+0100",
"lost": 0,
"flags": [
"dirty",
"omap_digest"
],
"truncate_seq": 0,
"truncate_size": 0,
"data_digest": "0xffffffff",
"omap_digest": "0xffffffff",
"expected_object_size": 4194304,
"expected_write_size": 4194304,
"alloc_hint_flags": 0,
"manifest": {
"type": 0
},
"watchers": {}
},
"shards": [
{
"osd": 11,
"primary": false,
"errors": [
"read_error"
],
"size": 4194304
},
{
"osd": 12,
"primary": false,
"errors": [],
"size": 4194304,
"omap_digest": "0xffffffff",
"data_digest": "0x66b5055b"
},
{
"osd": 18,
"primary": true,
"errors": [],
"size": 4194304,
"omap_digest": "0xffffffff",
"data_digest": "0x66b5055b"
}
]
}
]
}
rados list-inconsistent-obj 10.16 --format=json-pretty
{
"epoch": 3533,
"inconsistents": [
{
"object": {
"name": "rbd_data.261b03a6aa264.0000000000008921",
"nspace": "",
"locator": "",
"snap": "head",
"version": 9756930
},
"errors": [],
"union_shard_errors": [
"read_error"
],
"selected_object_info": {
"oid": {
"oid": "rbd_data.261b03a6aa264.0000000000008921",
"key": "",
"snapid": -2,
"hash": 2078265174,
"max": 0,
"pool": 10,
"namespace": ""
},
"version": "3524'9756930",
"prior_version": "3524'9756929",
"last_reqid": "client.143219.0:556715005",
"user_version": 9756930,
"size": 4194304,
"mtime": "2025-01-08T20:56:42.556007+0100",
"local_mtime": "2025-01-08T20:56:42.556296+0100",
"lost": 0,
"flags": [
"dirty",
"omap_digest"
],
"truncate_seq": 0,
"truncate_size": 0,
"data_digest": "0xffffffff",
"omap_digest": "0xffffffff",
"expected_object_size": 4194304,
"expected_write_size": 4194304,
"alloc_hint_flags": 0,
"manifest": {
"type": 0
},
"watchers": {}
},
"shards": [
{
"osd": 1,
"primary": false,
"errors": [],
"size": 4194304,
"omap_digest": "0xffffffff",
"data_digest": "0x98f78947"
},
{
"osd": 11,
"primary": true,
"errors": [
"read_error"
],
"size": 4194304
},
{
"osd": 23,
"primary": false,
"errors": [],
"size": 4194304,
"omap_digest": "0xffffffff",
"data_digest": "0x98f78947"
}
]
}
]
}
rbd image 'image-00001':
size 3 TiB in 786432 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: 22ceee723c5c
block_name_prefix: rbd_data.22ceee723c5c
format: 2
features: layering
op_features:
flags:
create_timestamp: Wed Dec 11 07:47:26 2024
access_timestamp: Wed Dec 11 07:47:26 2024
modify_timestamp: Wed Dec 11 07:47:26 2024
root@orn-45d-stor1:~# rbd info image-00002
rbd image 'image-00002':
size 3 TiB in 786432 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: 261b03a6aa264
block_name_prefix: rbd_data.261b03a6aa264
format: 2
features: layering
op_features:
flags:
create_timestamp: Wed Dec 11 08:07:26 2024
access_timestamp: Wed Dec 11 08:07:26 2024
modify_timestamp: Wed Dec 11 08:07:26 2024
ceph device ls DEVICE HOST:DEV DAEMONS WEAR LIFE EXPECTANCY HDSTOR_-_HSAV25S_HS22093014A0F7E55 node-stor1:sda mon.node-stor1 53% HDSTOR_-_HSAV25S_HS22101214A101BFE node-stor2:sda mon.node-stor2 51% HDSTOR_-_HSAV25S_HS22101214A101C70 node-stor3:sdb mon.node-stor3 49% Micron_5400_MTFD_500a07514428bbf1 node-stor4:sdi osd.18 osd.20 osd.22 0% Micron_5400_MTFD_500a07514428bddf node-stor1:sdi osd.10 osd.6 osd.8 0% Micron_5400_MTFD_500a0751442ad458 node-stor1:sdj osd.11 osd.7 osd.9 0% Micron_5400_MTFD_500a0751442ad55e node-stor2:sdj osd.13 osd.15 osd.17 0% Micron_5400_MTFD_500a0751442ad56a node-stor3:sdi osd.0 osd.2 osd.4 0% Micron_5400_MTFD_500a0751442ad56e node-stor4:sdj osd.19 osd.21 osd.23 0% Micron_5400_MTFD_500a0751442ad579 node-stor3:sdj osd.1 osd.3 osd.5 0% Micron_5400_MTFD_500a0751442adf6c node-stor2:sdi osd.12 osd.14 osd.16 0% Micron_7300_MTFDHBG1T9TDF_2022327BF1F2 node-stor2:nvme1n1 osd.13 osd.15 osd.17 0% Micron_7300_MTFDHBG1T9TDF_214132747BDF node-stor2:nvme0n1 osd.12 osd.14 osd.16 35% Micron_7450_MTFDKBG1T9TFR_221536FA4B7B node-stor1:nvme0n1 osd.11 osd.7 osd.9 0% Micron_7450_MTFDKBG1T9TFR_221536FA4D68 node-stor4:nvme1n1 osd.18 osd.20 osd.22 0% Micron_7450_MTFDKBG1T9TFR_221536FA4D78 node-stor4:nvme0n1 osd.19 osd.21 osd.23 51% Micron_7450_MTFDKBG1T9TFR_221536FA4E67 node-stor3:nvme1n1 osd.0 osd.2 osd.4 0% Micron_7450_MTFDKBG1T9TFR_221536FA4E7B node-stor3:nvme0n1 osd.1 osd.3 osd.5 58% Micron_7450_MTFDKBG1T9TFR_22313BEC8410 node-stor1:nvme1n1 osd.10 osd.6 osd.8 48% ST16000NM005J_5000c500ec286bdb node-stor4:sdf osd.20 ST16000NM005J_5000c500ec43256f node-stor1:sdf osd.11 ST16000NM005J_5000c500ec44302b node-stor2:sdf osd.17 ST16000NM005J_5000c500ec4437ef node-stor3:sdd osd.1 ST16000NM005J_5000c500ec44840f node-stor3:sdg osd.3 ST16000NM005J_5000c500ec44932b node-stor3:sdc osd.0 ST16000NM005J_5000c500ec53ba27 node-stor3:sde osd.2 ST16000NM005J_5000c500ec53ba97 node-stor1:sdd osd.7 ST16000NM005J_5000c500ec53bbf3 node-stor1:sde osd.10 ST16000NM005J_5000c500ec5411f3 node-stor1:sdc osd.6 ST16000NM005J_5000c500ec541507 node-stor1:sdg osd.8 ST16000NM005J_5000c500ec5415af node-stor3:sdh osd.4 ST16000NM005J_5000c500ec544fe3 node-stor1:sdh osd.9 ST16000NM005J_5000c500ec5451fb node-stor2:sde osd.14 ST16000NM005J_5000c500ec5770eb node-stor2:sdg osd.15 ST16000NM005J_5000c500ec5775c3 node-stor3:sdf osd.5 ST16000NM005J_5000c500ec57ec3b node-stor4:sdc osd.18 ST16000NM005J_5000c500ec57ec3f node-stor2:sdh osd.16 ST16000NM005J_5000c500ec57ec53 node-stor4:sde osd.21 ST16000NM005J_5000c500ec57ec67 node-stor4:sdg osd.22 ST16000NM005J_5000c500f29e867f node-stor4:sdh osd.23 ST16000NM005J_5000c500f2abea4f node-stor2:sdc osd.12 ST16000NM005J_5000c500f2ac0b9b node-stor4:sdd osd.19 ST16000NM005J_5000c500f2ac1373 node-stor2:sdd osd.13
smartctl --all /dev/sdf smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.14.21-08-petasan] (local build) Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Vendor: SEAGATE Product: ST16000NM005J Revision: ET04 Compliance: SPC-5 User Capacity: 16,000,900,661,248 bytes [16.0 TB] Logical block size: 512 bytes Physical block size: 4096 bytes LU is fully provisioned Rotation Rate: 7200 rpm Form Factor: 3.5 inches Logical Unit id: 0x5000c500ec43256f Serial number: ZR70Z2FC0000W34716QQ Device type: disk Transport protocol: SAS (SPL-3) Local Time is: Tue Jan 14 03:30:40 2025 CET SMART support is: Available - device has SMART capability. SMART support is: Enabled Temperature Warning: Enabled === START OF READ SMART DATA SECTION === SMART Health Status: OK Grown defects during certification <not available> Total blocks reassigned during format <not available> Total new blocks reassigned <not available> Power on minutes since format <not available> Current Drive Temperature: 33 C Drive Trip Temperature: 60 C Manufactured in week 42 of year 2023 Specified cycle count over device lifetime: 50000 Accumulated start-stop cycles: 13 Specified load-unload count over device lifetime: 600000 Accumulated load-unload cycles: 4209 Elements in grown defect list: 0 Vendor (Seagate Cache) information Blocks sent to initiator = 717091008 Blocks received from initiator = 1661229904 Blocks read from cache and sent to initiator = 1742662296 Number of read and write commands whose size <= segment size = 15420970 Number of read and write commands whose size > segment size = 34014 Vendor (Seagate/Hitachi) factory information number of hours powered up = 6537.55 number of minutes until next internal SMART test = 56 Error counter log: Errors Corrected by Total Correction Gigabytes Total ECC rereads/ errors algorithm processed uncorrected fast | delayed rewrites corrected invocations [10^9 bytes] errors read: 0 0 0 0 0 15760.315 0 write: 0 0 0 0 0 850.911 0 Non-medium error count: 0 [GLTSD (Global Logging Target Save Disable) set. Enable Save with '-S on'] No Self-tests have been logged
smartctl --all /dev/sdc smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.14.21-08-petasan] (local build) Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Vendor: SEAGATE Product: ST16000NM005J Revision: ET04 Compliance: SPC-5 User Capacity: 16,000,900,661,248 bytes [16.0 TB] Logical block size: 512 bytes Physical block size: 4096 bytes LU is fully provisioned Rotation Rate: 7200 rpm Form Factor: 3.5 inches Logical Unit id: 0x5000c500ec5411f3 Serial number: ZR71045F0000W3474RQT Device type: disk Transport protocol: SAS (SPL-3) Local Time is: Tue Jan 14 03:31:47 2025 CET SMART support is: Available - device has SMART capability. SMART support is: Enabled Temperature Warning: Enabled === START OF READ SMART DATA SECTION === SMART Health Status: OK Grown defects during certification <not available> Total blocks reassigned during format <not available> Total new blocks reassigned <not available> Power on minutes since format <not available> Current Drive Temperature: 29 C Drive Trip Temperature: 60 C Manufactured in week 42 of year 2023 Specified cycle count over device lifetime: 50000 Accumulated start-stop cycles: 11 Specified load-unload count over device lifetime: 600000 Accumulated load-unload cycles: 6097 Elements in grown defect list: 0 Vendor (Seagate Cache) information Blocks sent to initiator = 3487322904 Blocks received from initiator = 3137788280 Blocks read from cache and sent to initiator = 2103778790 Number of read and write commands whose size <= segment size = 11493458 Number of read and write commands whose size > segment size = 26103 Vendor (Seagate/Hitachi) factory information number of hours powered up = 6537.57 number of minutes until next internal SMART test = 14 Error counter log: Errors Corrected by Total Correction Gigabytes Total ECC rereads/ errors algorithm processed uncorrected fast | delayed rewrites corrected invocations [10^9 bytes] errors read: 0 0 0 0 0 14979.649 0 write: 0 0 0 0 0 3808.633 0 Non-medium error count: 0 [GLTSD (Global Logging Target Save Disable) set. Enable Save with '-S on'] No Self-tests have been logged
journalctl -f -u ceph-osd@11 -- Logs begin at Sun 2025-01-05 12:21:06 CET. -- Jan 14 01:48:48 node-stor1 ceph-osd[5967]: 2025-01-14T01:48:48.271+0100 7f2196b4b700 -1 log_channel(cluster) log [ERR] : 10.1c deep-scrub 0 missing, 2 inconsistent objects Jan 14 01:48:48 node-stor1 ceph-osd[5967]: 2025-01-14T01:48:48.271+0100 7f2196b4b700 -1 log_channel(cluster) log [ERR] : 10.1c deep-scrub 2 errors Jan 14 02:19:36 node-stor1 ceph-osd[5967]: 2025-01-14T02:19:36.929+0100 7f219834e700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x9000, got 0x108fe2eb, expected 0x95cc874a, device location [0x333a4d09000~1000], logical extent 0x169000~1000, object #10:f25549e6:::rbd_data.22ceee723c5c.00000000000121eb:head# Jan 14 02:19:36 node-stor1 ceph-osd[5967]: 2025-01-14T02:19:36.929+0100 7f219834e700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x9000, got 0x108fe2eb, expected 0x95cc874a, device location [0x333a4d09000~1000], logical extent 0x169000~1000, object #10:f25549e6:::rbd_data.22ceee723c5c.00000000000121eb:head# Jan 14 02:19:36 node-stor1 ceph-osd[5967]: 2025-01-14T02:19:36.933+0100 7f219834e700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x9000, got 0x108fe2eb, expected 0x95cc874a, device location [0x333a4d09000~1000], logical extent 0x169000~1000, object #10:f25549e6:::rbd_data.22ceee723c5c.00000000000121eb:head# Jan 14 02:19:36 node-stor1 ceph-osd[5967]: 2025-01-14T02:19:36.933+0100 7f219834e700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x9000, got 0x108fe2eb, expected 0x95cc874a, device location [0x333a4d09000~1000], logical extent 0x169000~1000, object #10:f25549e6:::rbd_data.22ceee723c5c.00000000000121eb:head# Jan 14 02:57:08 node-stor1 ceph-osd[5967]: 2025-01-14T02:57:08.891+0100 7f219834e700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x0, got 0xb21969fe, expected 0xea08e29, device location [0x484b14c0000~1000], logical extent 0x380000~1000, object #10:f57c893e:::rbd_data.261b03a6aa264.0000000000009a34:head# Jan 14 02:57:08 node-stor1 ceph-osd[5967]: 2025-01-14T02:57:08.903+0100 7f219834e700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x0, got 0xb21969fe, expected 0xea08e29, device location [0x484b14c0000~1000], logical extent 0x380000~1000, object #10:f57c893e:::rbd_data.261b03a6aa264.0000000000009a34:head# Jan 14 02:57:08 node-stor1 ceph-osd[5967]: 2025-01-14T02:57:08.903+0100 7f219834e700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x0, got 0xb21969fe, expected 0xea08e29, device location [0x484b14c0000~1000], logical extent 0x380000~1000, object #10:f57c893e:::rbd_data.261b03a6aa264.0000000000009a34:head# Jan 14 02:57:08 node-stor1 ceph-osd[5967]: 2025-01-14T02:57:08.903+0100 7f219834e700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x0, got 0xb21969fe, expected 0xea08e29, device location [0x484b14c0000~1000], logical extent 0x380000~1000, object #10:f57c893e:::rbd_data.261b03a6aa264.0000000000009a34:head# Jan 14 03:54:48 node-stor1 ceph-osd[5967]: 2025-01-14T03:54:48.707+0100 7f2196b4b700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x1000, got 0xc8021cc4, expected 0x6706be76, device location [0x45f6c021000~1000], logical extent 0x281000~1000, object #10:3a57c27b:::rbd_data.261b03a6aa264.0000000000007add:head# Jan 14 03:54:48 node-stor1 ceph-osd[5967]: 2025-01-14T03:54:48.707+0100 7f2196b4b700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x1000, got 0xc8021cc4, expected 0x6706be76, device location [0x45f6c021000~1000], logical extent 0x281000~1000, object #10:3a57c27b:::rbd_data.261b03a6aa264.0000000000007add:head# Jan 14 03:54:48 node-stor1 ceph-osd[5967]: 2025-01-14T03:54:48.711+0100 7f2196b4b700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x1000, got 0xc8021cc4, expected 0x6706be76, device location [0x45f6c021000~1000], logical extent 0x281000~1000, object #10:3a57c27b:::rbd_data.261b03a6aa264.0000000000007add:head# Jan 14 03:54:48 node-stor1 ceph-osd[5967]: 2025-01-14T03:54:48.711+0100 7f2196b4b700 -1 bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x1000, got 0xc8021cc4, expected 0x6706be76, device location [0x45f6c021000~1000], logical extent 0x281000~1000, object #10:3a57c27b:::rbd_data.261b03a6aa264.0000000000007add:head# Jan 14 03:54:50 node-stor1 ceph-osd[5967]: 2025-01-14T03:54:50.075+0100 7f2196b4b700 -1 log_channel(cluster) log [ERR] : 10.1c shard 11 soid 10:3a57c27b:::rbd_data.261b03a6aa264.0000000000007add:head : candidate had a read error
Last edited on January 14, 2025, 3:20 am by lberrezoug · #1
lberrezoug
5 Posts
January 14, 2025, 2:22 pm
Quote from lberrezoug on January 14, 2025, 2:22 pmUPDATE:
I solved my issue by restarting the osd.11 and running the PG repair again. Since 10 hours the state of the cluster is OK and I hope it will stay that way 🙂
UPDATE:
I solved my issue by restarting the osd.11 and running the PG repair again. Since 10 hours the state of the cluster is OK and I hope it will stay that way 🙂
Last edited on January 14, 2025, 2:22 pm by lberrezoug · #2