large objects found in pool 'default.rgw.buckets.index'
rhamon
30 Posts
December 20, 2022, 10:54 pmQuote from rhamon on December 20, 2022, 10:54 pmI've just started heavily using S3 with PetaSAN 3.1.0 and getting warnings about "large objects found in pool 'default.rgw.buckets.index'" in `ceph health detail`.
I was expecting dynamic resharding (https://docs.ceph.com/en/latest/radosgw/dynamicresharding/) to prevent or automatically fix my issue above...
What am I missing or need to tune for this to work as expected?
ceph --admin-daemon /var/run/ceph/ceph-client.rgw.*.asok config show | grep resharding
"rgw_dynamic_resharding": "true",
Thanks
I've just started heavily using S3 with PetaSAN 3.1.0 and getting warnings about "large objects found in pool 'default.rgw.buckets.index'" in `ceph health detail`.
I was expecting dynamic resharding (https://docs.ceph.com/en/latest/radosgw/dynamicresharding/) to prevent or automatically fix my issue above...
What am I missing or need to tune for this to work as expected?
ceph --admin-daemon /var/run/ceph/ceph-client.rgw.*.asok config show | grep resharding
"rgw_dynamic_resharding": "true",
Thanks
admin
2,930 Posts
December 21, 2022, 5:23 pmQuote from admin on December 21, 2022, 5:23 pmroughly how many objects do you have in your large buckets ?
are you in a multi-site setup ?
roughly how many objects do you have in your large buckets ?
are you in a multi-site setup ?
rhamon
30 Posts
December 22, 2022, 9:38 pmQuote from rhamon on December 22, 2022, 9:38 pmNot using multi-site, here's radosgw-admin stats for the bucket:
{
"bucket": "storage",
"num_shards": 197,
"placement_rule": "default-placement",
"index_type": "Normal",
"usage": {
"rgw.none": {
"size": 0,
"size_actual": 0,
"size_utilized": 0,
"size_kb": 0,
"size_kb_actual": 0,
"size_kb_utilized": 0,
"num_objects": 18446744073709551615
},
"rgw.main": {
"size": 170558800117025,
"size_actual": 170575466446848,
"size_utilized": 170558800117025,
"size_kb": 166561328240,
"size_kb_actual": 166577603952,
"size_kb_utilized": 166561328240,
"num_objects": 15782176
},
"rgw.multimeta": {
"size": 0,
"size_actual": 0,
"size_utilized": 61290,
"size_kb": 0,
"size_kb_actual": 0,
"size_kb_utilized": 60,
"num_objects": 2270
}
},
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
}
}
And exemple of log entry found for large object:
log_channel(cluster) log [WRN] : Large omap object found. Object: 6:5d7bee93:::.dir.65b393bb-9d7d-4c08-a55e-7934f0064560.23002980.1.23:head PG: 6.c977deba (6.a) Key count: 207663 Size (bytes): 63900535
Thanks
Not using multi-site, here's radosgw-admin stats for the bucket:
{
"bucket": "storage",
"num_shards": 197,
"placement_rule": "default-placement",
"index_type": "Normal",
"usage": {
"rgw.none": {
"size": 0,
"size_actual": 0,
"size_utilized": 0,
"size_kb": 0,
"size_kb_actual": 0,
"size_kb_utilized": 0,
"num_objects": 18446744073709551615
},
"rgw.main": {
"size": 170558800117025,
"size_actual": 170575466446848,
"size_utilized": 170558800117025,
"size_kb": 166561328240,
"size_kb_actual": 166577603952,
"size_kb_utilized": 166561328240,
"num_objects": 15782176
},
"rgw.multimeta": {
"size": 0,
"size_actual": 0,
"size_utilized": 61290,
"size_kb": 0,
"size_kb_actual": 0,
"size_kb_utilized": 60,
"num_objects": 2270
}
},
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
}
}
And exemple of log entry found for large object:
log_channel(cluster) log [WRN] : Large omap object found. Object: 6:5d7bee93:::.dir.65b393bb-9d7d-4c08-a55e-7934f0064560.23002980.1.23:head PG: 6.c977deba (6.a) Key count: 207663 Size (bytes): 63900535
Thanks
Last edited on December 22, 2022, 9:39 pm by rhamon · #3
admin
2,930 Posts
January 4, 2023, 2:10 pmQuote from admin on January 4, 2023, 2:10 pmdo you still see this error or have you resolved it ?
do you still see this error or have you resolved it ?
rhamon
30 Posts
January 9, 2023, 9:11 pmQuote from rhamon on January 9, 2023, 9:11 pmStill getting warning about large objects, now at 31 up from the initial 8 large objects found.
# ceph health detail
HEALTH_WARN 31 large omap objects; 1 pgs not deep-scrubbed in time
[WRN] LARGE_OMAP_OBJECTS: 31 large omap objects
31 large objects found in pool 'default.rgw.buckets.index'
Search the cluster log for 'Large omap object found' for more details.
[WRN] PG_NOT_DEEP_SCRUBBED: 1 pgs not deep-scrubbed in time
pg 17.76 not deep-scrubbed since 2022-12-28T05:09:22.700534-0500
# ceph -s
cluster:
health: HEALTH_WARN
31 large omap objects
1 pgs not deep-scrubbed in time
services:
mon: 3 daemons, quorum [...] (age 8d)
mgr: [...](active, since 5w), standbys: [...]
mds: 3 up:standby
osd: 50 osds: 50 up (since 2w), 50 in (since 5w)
rgw: 3 daemons active ([...])
task status:
data:
pools: 11 pools, 1009 pgs
objects: 75.89M objects, 225 TiB
usage: 373 TiB used, 249 TiB / 622 TiB avail
pgs: 1007 active+clean
1 active+clean+scrubbing
1 active+clean+scrubbing+deep
# grep ' Large omap object found.' /var/log/ceph/ceph.log
[...]
2023-01-09T06:21:07.121852-0500 osd.33 (osd.33) 11432 : cluster [WRN] Large omap object found. Object: 6:09fda159:::.dir.65b393bb-9d7d-4c08-a55e-7934f0064560.21438367.3.3:head PG: 6.9a85bf90 (6.0) Key count: 214960 Size (bytes): 66443716
2023-01-09T14:34:01.315247-0500 osd.33 (osd.33) 11453 : cluster [WRN] Large omap object found. Object: 6:90fa0c6c:::.dir.65b393bb-9d7d-4c08-a55e-7934f0064560.21438367.3.88:head PG: 6.36305f09 (6.9) Key count: 222468 Size (bytes): 68741290
2023-01-09T14:34:12.276768-0500 osd.33 (osd.33) 11454 : cluster [WRN] Large omap object found. Object: 6:9642e5e8:::.dir.65b393bb-9d7d-4c08-a55e-7934f0064560.21438367.3.186:head PG: 6.17a74269 (6.9) Key count: 239416 Size (bytes): 73845551
2023-01-09T14:34:29.206175-0500 osd.33 (osd.33) 11455 : cluster [WRN] Large omap object found. Object: 6:9f9be708:::.dir.65b393bb-9d7d-4c08-a55e-7934f0064560.21438367.3.75:head PG: 6.10e7d9f9 (6.9) Key count: 337766 Size (bytes): 104379073
[...]
# radosgw-admin bucket stats
[
{
"bucket": "s3-bucket",
"num_shards": 197,
"tenant": "",
"zonegroup": "[...]",
"placement_rule": "default-placement",
"explicit_placement": {
"data_pool": "",
"data_extra_pool": "",
"index_pool": ""
},
"index_type": "Normal",
"owner": "owner",
"ver": "0#74980,1#18081,2#188267,3#221442,4#123725,5#34059,6#106611,7#39907,8#179701,9#51548,10#20305,11#200272,12#30211,13#37878,14#22721,15#35845,16#167822,17#23097,18#35721,19#32512,20#144586,21#20189,22#87688,23#22479,24#34607,25#6692,26#35015,27#38888,28#31121,29#55014,30#42483,31#36938,32#202931,33#44493,34#29178,35#11588,36#47592,37#42085,38#27340,39#36008,40#11077,41#115142,42#22737,43#17950,44#266533,45#30256,46#100858,47#144545,48#37577,49#90390,50#58125,51#237849,52#53828,53#164441,54#33349,55#81632,56#73355,57#29466,58#81869,59#34558,60#54234,61#27455,62#235497,63#36500,64#76102,65#34690,66#10612,67#18718,68#33150,69#21702,70#65150,71#126144,72#22914,73#30838,74#250221,75#225236,76#34287,77#120700,78#59226,79#163612,80#30444,81#38140,82#11584,83#68199,84#36880,85#254883,86#28060,87#26871,88#29290,89#95864,90#44233,91#33201,92#18208,93#11541,94#27171,95#36662,96#32895,97#17693,98#119338,99#15044,100#118795,101#44310,102#30669,103#17068,104#32161,105#32920,106#16041,107#94520,108#27135,109#35399,110#38444,111#53650,112#20285,113#17607,114#11519,115#93718,116#36972,117#25299,118#44019,119#5346,120#63197,121#122723,122#92030,123#111157,124#46502,125#113354,126#15074,127#81594,128#18503,129#13313,130#21200,131#127973,132#41494,133#18115,134#17392,135#27703,136#22478,137#9787,138#64426,139#20955,140#64377,141#6789,142#123420,143#25533,144#16474,145#4011,146#337290,147#209891,148#39412,149#144821,150#46543,151#18178,152#22976,153#42171,154#14750,155#46559,156#12148,157#18006,158#14712,159#77078,160#24254,161#57777,162#126363,163#25048,164#21486,165#28449,166#50098,167#82983,168#19755,169#91077,170#20769,171#20005,172#13542,173#3762,174#19423,175#76019,176#37772,177#43040,178#8584,179#20538,180#28992,181#15110,182#16846,183#127675,184#56701,185#33299,186#129077,187#38940,188#108275,189#28650,190#21171,191#77984,192#5239,193#34959,194#14433,195#55604,196#120194",
"master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0,30#0,31#0,32#0,33#0,34#0,35#0,36#0,37#0,38#0,39#0,40#0,41#0,42#0,43#0,44#0,45#0,46#0,47#0,48#0,49#0,50#0,51#0,52#0,53#0,54#0,55#0,56#0,57#0,58#0,59#0,60#0,61#0,62#0,63#0,64#0,65#0,66#0,67#0,68#0,69#0,70#0,71#0,72#0,73#0,74#0,75#0,76#0,77#0,78#0,79#0,80#0,81#0,82#0,83#0,84#0,85#0,86#0,87#0,88#0,89#0,90#0,91#0,92#0,93#0,94#0,95#0,96#0,97#0,98#0,99#0,100#0,101#0,102#0,103#0,104#0,105#0,106#0,107#0,108#0,109#0,110#0,111#0,112#0,113#0,114#0,115#0,116#0,117#0,118#0,119#0,120#0,121#0,122#0,123#0,124#0,125#0,126#0,127#0,128#0,129#0,130#0,131#0,132#0,133#0,134#0,135#0,136#0,137#0,138#0,139#0,140#0,141#0,142#0,143#0,144#0,145#0,146#0,147#0,148#0,149#0,150#0,151#0,152#0,153#0,154#0,155#0,156#0,157#0,158#0,159#0,160#0,161#0,162#0,163#0,164#0,165#0,166#0,167#0,168#0,169#0,170#0,171#0,172#0,173#0,174#0,175#0,176#0,177#0,178#0,179#0,180#0,181#0,182#0,183#0,184#0,185#0,186#0,187#0,188#0,189#0,190#0,191#0,192#0,193#0,194#0,195#0,196#0",
"mtime": "2022-12-19T23:20:42.208563Z",
"creation_time": "2022-07-04T18:43:39.231681Z",
"max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#,11#,12#,13#,14#,15#,16#,17#,18#,19#,20#,21#,22#,23#,24#,25#,26#,27#,28#,29#,30#,31#,32#,33#,34#,35#,36#,37#,38#,39#,40#,41#,42#,43#,44#,45#,46#,47#,48#,49#,50#,51#,52#,53#,54#,55#,56#,57#,58#,59#,60#,61#,62#,63#,64#,65#,66#,67#,68#,69#,70#,71#,72#,73#,74#,75#,76#,77#,78#,79#,80#,81#,82#,83#,84#,85#,86#,87#,88#,89#,90#,91#,92#,93#,94#,95#,96#,97#,98#,99#,100#,101#,102#,103#,104#,105#,106#,107#,108#,109#,110#,111#,112#,113#,114#,115#,116#,117#,118#,119#,120#,121#,122#,123#,124#,125#,126#,127#,128#,129#,130#,131#,132#,133#,134#,135#,136#,137#,138#,139#,140#,141#,142#,143#,144#,145#,146#,147#,148#,149#,150#,151#,152#,153#,154#,155#,156#,157#,158#,159#,160#,161#,162#,163#,164#,165#,166#,167#,168#,169#,170#,171#,172#,173#,174#,175#,176#,177#,178#,179#,180#,181#,182#,183#,184#,185#,186#,187#,188#,189#,190#,191#,192#,193#,194#,195#,196#",
"usage": {
"rgw.main": {
"size": 206156753500892,
"size_actual": 206174454808576,
"size_utilized": 206156753500892,
"size_kb": 201324954591,
"size_kb_actual": 201342241024,
"size_kb_utilized": 201324954591,
"num_objects": 16944078
}}
Still getting warning about large objects, now at 31 up from the initial 8 large objects found.
# ceph health detail
HEALTH_WARN 31 large omap objects; 1 pgs not deep-scrubbed in time
[WRN] LARGE_OMAP_OBJECTS: 31 large omap objects
31 large objects found in pool 'default.rgw.buckets.index'
Search the cluster log for 'Large omap object found' for more details.
[WRN] PG_NOT_DEEP_SCRUBBED: 1 pgs not deep-scrubbed in time
pg 17.76 not deep-scrubbed since 2022-12-28T05:09:22.700534-0500
# ceph -s
cluster:
health: HEALTH_WARN
31 large omap objects
1 pgs not deep-scrubbed in time
services:
mon: 3 daemons, quorum [...] (age 8d)
mgr: [...](active, since 5w), standbys: [...]
mds: 3 up:standby
osd: 50 osds: 50 up (since 2w), 50 in (since 5w)
rgw: 3 daemons active ([...])
task status:
data:
pools: 11 pools, 1009 pgs
objects: 75.89M objects, 225 TiB
usage: 373 TiB used, 249 TiB / 622 TiB avail
pgs: 1007 active+clean
1 active+clean+scrubbing
1 active+clean+scrubbing+deep
# grep ' Large omap object found.' /var/log/ceph/ceph.log
[...]
2023-01-09T06:21:07.121852-0500 osd.33 (osd.33) 11432 : cluster [WRN] Large omap object found. Object: 6:09fda159:::.dir.65b393bb-9d7d-4c08-a55e-7934f0064560.21438367.3.3:head PG: 6.9a85bf90 (6.0) Key count: 214960 Size (bytes): 66443716
2023-01-09T14:34:01.315247-0500 osd.33 (osd.33) 11453 : cluster [WRN] Large omap object found. Object: 6:90fa0c6c:::.dir.65b393bb-9d7d-4c08-a55e-7934f0064560.21438367.3.88:head PG: 6.36305f09 (6.9) Key count: 222468 Size (bytes): 68741290
2023-01-09T14:34:12.276768-0500 osd.33 (osd.33) 11454 : cluster [WRN] Large omap object found. Object: 6:9642e5e8:::.dir.65b393bb-9d7d-4c08-a55e-7934f0064560.21438367.3.186:head PG: 6.17a74269 (6.9) Key count: 239416 Size (bytes): 73845551
2023-01-09T14:34:29.206175-0500 osd.33 (osd.33) 11455 : cluster [WRN] Large omap object found. Object: 6:9f9be708:::.dir.65b393bb-9d7d-4c08-a55e-7934f0064560.21438367.3.75:head PG: 6.10e7d9f9 (6.9) Key count: 337766 Size (bytes): 104379073
[...]
# radosgw-admin bucket stats
[
{
"bucket": "s3-bucket",
"num_shards": 197,
"tenant": "",
"zonegroup": "[...]",
"placement_rule": "default-placement",
"explicit_placement": {
"data_pool": "",
"data_extra_pool": "",
"index_pool": ""
},
"index_type": "Normal",
"owner": "owner",
"ver": "0#74980,1#18081,2#188267,3#221442,4#123725,5#34059,6#106611,7#39907,8#179701,9#51548,10#20305,11#200272,12#30211,13#37878,14#22721,15#35845,16#167822,17#23097,18#35721,19#32512,20#144586,21#20189,22#87688,23#22479,24#34607,25#6692,26#35015,27#38888,28#31121,29#55014,30#42483,31#36938,32#202931,33#44493,34#29178,35#11588,36#47592,37#42085,38#27340,39#36008,40#11077,41#115142,42#22737,43#17950,44#266533,45#30256,46#100858,47#144545,48#37577,49#90390,50#58125,51#237849,52#53828,53#164441,54#33349,55#81632,56#73355,57#29466,58#81869,59#34558,60#54234,61#27455,62#235497,63#36500,64#76102,65#34690,66#10612,67#18718,68#33150,69#21702,70#65150,71#126144,72#22914,73#30838,74#250221,75#225236,76#34287,77#120700,78#59226,79#163612,80#30444,81#38140,82#11584,83#68199,84#36880,85#254883,86#28060,87#26871,88#29290,89#95864,90#44233,91#33201,92#18208,93#11541,94#27171,95#36662,96#32895,97#17693,98#119338,99#15044,100#118795,101#44310,102#30669,103#17068,104#32161,105#32920,106#16041,107#94520,108#27135,109#35399,110#38444,111#53650,112#20285,113#17607,114#11519,115#93718,116#36972,117#25299,118#44019,119#5346,120#63197,121#122723,122#92030,123#111157,124#46502,125#113354,126#15074,127#81594,128#18503,129#13313,130#21200,131#127973,132#41494,133#18115,134#17392,135#27703,136#22478,137#9787,138#64426,139#20955,140#64377,141#6789,142#123420,143#25533,144#16474,145#4011,146#337290,147#209891,148#39412,149#144821,150#46543,151#18178,152#22976,153#42171,154#14750,155#46559,156#12148,157#18006,158#14712,159#77078,160#24254,161#57777,162#126363,163#25048,164#21486,165#28449,166#50098,167#82983,168#19755,169#91077,170#20769,171#20005,172#13542,173#3762,174#19423,175#76019,176#37772,177#43040,178#8584,179#20538,180#28992,181#15110,182#16846,183#127675,184#56701,185#33299,186#129077,187#38940,188#108275,189#28650,190#21171,191#77984,192#5239,193#34959,194#14433,195#55604,196#120194",
"master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0,30#0,31#0,32#0,33#0,34#0,35#0,36#0,37#0,38#0,39#0,40#0,41#0,42#0,43#0,44#0,45#0,46#0,47#0,48#0,49#0,50#0,51#0,52#0,53#0,54#0,55#0,56#0,57#0,58#0,59#0,60#0,61#0,62#0,63#0,64#0,65#0,66#0,67#0,68#0,69#0,70#0,71#0,72#0,73#0,74#0,75#0,76#0,77#0,78#0,79#0,80#0,81#0,82#0,83#0,84#0,85#0,86#0,87#0,88#0,89#0,90#0,91#0,92#0,93#0,94#0,95#0,96#0,97#0,98#0,99#0,100#0,101#0,102#0,103#0,104#0,105#0,106#0,107#0,108#0,109#0,110#0,111#0,112#0,113#0,114#0,115#0,116#0,117#0,118#0,119#0,120#0,121#0,122#0,123#0,124#0,125#0,126#0,127#0,128#0,129#0,130#0,131#0,132#0,133#0,134#0,135#0,136#0,137#0,138#0,139#0,140#0,141#0,142#0,143#0,144#0,145#0,146#0,147#0,148#0,149#0,150#0,151#0,152#0,153#0,154#0,155#0,156#0,157#0,158#0,159#0,160#0,161#0,162#0,163#0,164#0,165#0,166#0,167#0,168#0,169#0,170#0,171#0,172#0,173#0,174#0,175#0,176#0,177#0,178#0,179#0,180#0,181#0,182#0,183#0,184#0,185#0,186#0,187#0,188#0,189#0,190#0,191#0,192#0,193#0,194#0,195#0,196#0",
"mtime": "2022-12-19T23:20:42.208563Z",
"creation_time": "2022-07-04T18:43:39.231681Z",
"max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#,11#,12#,13#,14#,15#,16#,17#,18#,19#,20#,21#,22#,23#,24#,25#,26#,27#,28#,29#,30#,31#,32#,33#,34#,35#,36#,37#,38#,39#,40#,41#,42#,43#,44#,45#,46#,47#,48#,49#,50#,51#,52#,53#,54#,55#,56#,57#,58#,59#,60#,61#,62#,63#,64#,65#,66#,67#,68#,69#,70#,71#,72#,73#,74#,75#,76#,77#,78#,79#,80#,81#,82#,83#,84#,85#,86#,87#,88#,89#,90#,91#,92#,93#,94#,95#,96#,97#,98#,99#,100#,101#,102#,103#,104#,105#,106#,107#,108#,109#,110#,111#,112#,113#,114#,115#,116#,117#,118#,119#,120#,121#,122#,123#,124#,125#,126#,127#,128#,129#,130#,131#,132#,133#,134#,135#,136#,137#,138#,139#,140#,141#,142#,143#,144#,145#,146#,147#,148#,149#,150#,151#,152#,153#,154#,155#,156#,157#,158#,159#,160#,161#,162#,163#,164#,165#,166#,167#,168#,169#,170#,171#,172#,173#,174#,175#,176#,177#,178#,179#,180#,181#,182#,183#,184#,185#,186#,187#,188#,189#,190#,191#,192#,193#,194#,195#,196#",
"usage": {
"rgw.main": {
"size": 206156753500892,
"size_actual": 206174454808576,
"size_utilized": 206156753500892,
"size_kb": 201324954591,
"size_kb_actual": 201342241024,
"size_kb_utilized": 201324954591,
"num_objects": 16944078
}}
f.cuseo
66 Posts
January 29, 2024, 11:41 amQuote from f.cuseo on January 29, 2024, 11:41 amHave you fixed this issue ?
I am experiencing the same problem...
Regards, Fabrizio
Have you fixed this issue ?
I am experiencing the same problem...
Regards, Fabrizio
admin
2,930 Posts
January 29, 2024, 4:43 pmQuote from admin on January 29, 2024, 4:43 pmi recommend you check the ceph info available online regarding this issue.
i recommend you check the ceph info available online regarding this issue.
f.cuseo
66 Posts
January 29, 2024, 4:49 pmQuote from f.cuseo on January 29, 2024, 4:49 pmI have done it, tried some fix, but without any result...
I have done it, tried some fix, but without any result...
f.cuseo
66 Posts
January 30, 2024, 4:33 pmQuote from f.cuseo on January 30, 2024, 4:33 pmFixed with a
ceph config set osd osd_deep_scrub_large_omap_object_key_threshold 400000
but I think that rgw bucket index should be automatically resharded...
Fixed with a
ceph config set osd osd_deep_scrub_large_omap_object_key_threshold 400000
but I think that rgw bucket index should be automatically resharded...
large objects found in pool 'default.rgw.buckets.index'
rhamon
30 Posts
Quote from rhamon on December 20, 2022, 10:54 pmI've just started heavily using S3 with PetaSAN 3.1.0 and getting warnings about "large objects found in pool 'default.rgw.buckets.index'" in `ceph health detail`.
I was expecting dynamic resharding (https://docs.ceph.com/en/latest/radosgw/dynamicresharding/) to prevent or automatically fix my issue above...
What am I missing or need to tune for this to work as expected?
ceph --admin-daemon /var/run/ceph/ceph-client.rgw.*.asok config show | grep resharding
"rgw_dynamic_resharding": "true",Thanks
I've just started heavily using S3 with PetaSAN 3.1.0 and getting warnings about "large objects found in pool 'default.rgw.buckets.index'" in `ceph health detail`.
I was expecting dynamic resharding (https://docs.ceph.com/en/latest/radosgw/dynamicresharding/) to prevent or automatically fix my issue above...
What am I missing or need to tune for this to work as expected?
ceph --admin-daemon /var/run/ceph/ceph-client.rgw.*.asok config show | grep resharding
"rgw_dynamic_resharding": "true",
Thanks
admin
2,930 Posts
Quote from admin on December 21, 2022, 5:23 pmroughly how many objects do you have in your large buckets ?
are you in a multi-site setup ?
roughly how many objects do you have in your large buckets ?
are you in a multi-site setup ?
rhamon
30 Posts
Quote from rhamon on December 22, 2022, 9:38 pmNot using multi-site, here's radosgw-admin stats for the bucket:
{
"bucket": "storage",
"num_shards": 197,
"placement_rule": "default-placement",
"index_type": "Normal",
"usage": {
"rgw.none": {
"size": 0,
"size_actual": 0,
"size_utilized": 0,
"size_kb": 0,
"size_kb_actual": 0,
"size_kb_utilized": 0,
"num_objects": 18446744073709551615
},
"rgw.main": {
"size": 170558800117025,
"size_actual": 170575466446848,
"size_utilized": 170558800117025,
"size_kb": 166561328240,
"size_kb_actual": 166577603952,
"size_kb_utilized": 166561328240,
"num_objects": 15782176
},
"rgw.multimeta": {
"size": 0,
"size_actual": 0,
"size_utilized": 61290,
"size_kb": 0,
"size_kb_actual": 0,
"size_kb_utilized": 60,
"num_objects": 2270
}
},
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
}
}And exemple of log entry found for large object:
log_channel(cluster) log [WRN] : Large omap object found. Object: 6:5d7bee93:::.dir.65b393bb-9d7d-4c08-a55e-7934f0064560.23002980.1.23:head PG: 6.c977deba (6.a) Key count: 207663 Size (bytes): 63900535Thanks
Not using multi-site, here's radosgw-admin stats for the bucket:
{
"bucket": "storage",
"num_shards": 197,
"placement_rule": "default-placement",
"index_type": "Normal",
"usage": {
"rgw.none": {
"size": 0,
"size_actual": 0,
"size_utilized": 0,
"size_kb": 0,
"size_kb_actual": 0,
"size_kb_utilized": 0,
"num_objects": 18446744073709551615
},
"rgw.main": {
"size": 170558800117025,
"size_actual": 170575466446848,
"size_utilized": 170558800117025,
"size_kb": 166561328240,
"size_kb_actual": 166577603952,
"size_kb_utilized": 166561328240,
"num_objects": 15782176
},
"rgw.multimeta": {
"size": 0,
"size_actual": 0,
"size_utilized": 61290,
"size_kb": 0,
"size_kb_actual": 0,
"size_kb_utilized": 60,
"num_objects": 2270
}
},
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
}
}
And exemple of log entry found for large object:
log_channel(cluster) log [WRN] : Large omap object found. Object: 6:5d7bee93:::.dir.65b393bb-9d7d-4c08-a55e-7934f0064560.23002980.1.23:head PG: 6.c977deba (6.a) Key count: 207663 Size (bytes): 63900535
Thanks
admin
2,930 Posts
Quote from admin on January 4, 2023, 2:10 pmdo you still see this error or have you resolved it ?
do you still see this error or have you resolved it ?
rhamon
30 Posts
Quote from rhamon on January 9, 2023, 9:11 pmStill getting warning about large objects, now at 31 up from the initial 8 large objects found.
# ceph health detail
HEALTH_WARN 31 large omap objects; 1 pgs not deep-scrubbed in time
[WRN] LARGE_OMAP_OBJECTS: 31 large omap objects
31 large objects found in pool 'default.rgw.buckets.index'
Search the cluster log for 'Large omap object found' for more details.
[WRN] PG_NOT_DEEP_SCRUBBED: 1 pgs not deep-scrubbed in time
pg 17.76 not deep-scrubbed since 2022-12-28T05:09:22.700534-0500
# ceph -s
cluster:
health: HEALTH_WARN
31 large omap objects
1 pgs not deep-scrubbed in timeservices:
mon: 3 daemons, quorum [...] (age 8d)
mgr: [...](active, since 5w), standbys: [...]
mds: 3 up:standby
osd: 50 osds: 50 up (since 2w), 50 in (since 5w)
rgw: 3 daemons active ([...])task status:
data:
pools: 11 pools, 1009 pgs
objects: 75.89M objects, 225 TiB
usage: 373 TiB used, 249 TiB / 622 TiB avail
pgs: 1007 active+clean
1 active+clean+scrubbing
1 active+clean+scrubbing+deep
# grep ' Large omap object found.' /var/log/ceph/ceph.log
[...]
2023-01-09T06:21:07.121852-0500 osd.33 (osd.33) 11432 : cluster [WRN] Large omap object found. Object: 6:09fda159:::.dir.65b393bb-9d7d-4c08-a55e-7934f0064560.21438367.3.3:head PG: 6.9a85bf90 (6.0) Key count: 214960 Size (bytes): 66443716
2023-01-09T14:34:01.315247-0500 osd.33 (osd.33) 11453 : cluster [WRN] Large omap object found. Object: 6:90fa0c6c:::.dir.65b393bb-9d7d-4c08-a55e-7934f0064560.21438367.3.88:head PG: 6.36305f09 (6.9) Key count: 222468 Size (bytes): 68741290
2023-01-09T14:34:12.276768-0500 osd.33 (osd.33) 11454 : cluster [WRN] Large omap object found. Object: 6:9642e5e8:::.dir.65b393bb-9d7d-4c08-a55e-7934f0064560.21438367.3.186:head PG: 6.17a74269 (6.9) Key count: 239416 Size (bytes): 73845551
2023-01-09T14:34:29.206175-0500 osd.33 (osd.33) 11455 : cluster [WRN] Large omap object found. Object: 6:9f9be708:::.dir.65b393bb-9d7d-4c08-a55e-7934f0064560.21438367.3.75:head PG: 6.10e7d9f9 (6.9) Key count: 337766 Size (bytes): 104379073[...]
# radosgw-admin bucket stats
[
{
"bucket": "s3-bucket",
"num_shards": 197,
"tenant": "",
"zonegroup": "[...]",
"placement_rule": "default-placement",
"explicit_placement": {
"data_pool": "",
"data_extra_pool": "",
"index_pool": ""
},
"index_type": "Normal",
"owner": "owner",
"ver": "0#74980,1#18081,2#188267,3#221442,4#123725,5#34059,6#106611,7#39907,8#179701,9#51548,10#20305,11#200272,12#30211,13#37878,14#22721,15#35845,16#167822,17#23097,18#35721,19#32512,20#144586,21#20189,22#87688,23#22479,24#34607,25#6692,26#35015,27#38888,28#31121,29#55014,30#42483,31#36938,32#202931,33#44493,34#29178,35#11588,36#47592,37#42085,38#27340,39#36008,40#11077,41#115142,42#22737,43#17950,44#266533,45#30256,46#100858,47#144545,48#37577,49#90390,50#58125,51#237849,52#53828,53#164441,54#33349,55#81632,56#73355,57#29466,58#81869,59#34558,60#54234,61#27455,62#235497,63#36500,64#76102,65#34690,66#10612,67#18718,68#33150,69#21702,70#65150,71#126144,72#22914,73#30838,74#250221,75#225236,76#34287,77#120700,78#59226,79#163612,80#30444,81#38140,82#11584,83#68199,84#36880,85#254883,86#28060,87#26871,88#29290,89#95864,90#44233,91#33201,92#18208,93#11541,94#27171,95#36662,96#32895,97#17693,98#119338,99#15044,100#118795,101#44310,102#30669,103#17068,104#32161,105#32920,106#16041,107#94520,108#27135,109#35399,110#38444,111#53650,112#20285,113#17607,114#11519,115#93718,116#36972,117#25299,118#44019,119#5346,120#63197,121#122723,122#92030,123#111157,124#46502,125#113354,126#15074,127#81594,128#18503,129#13313,130#21200,131#127973,132#41494,133#18115,134#17392,135#27703,136#22478,137#9787,138#64426,139#20955,140#64377,141#6789,142#123420,143#25533,144#16474,145#4011,146#337290,147#209891,148#39412,149#144821,150#46543,151#18178,152#22976,153#42171,154#14750,155#46559,156#12148,157#18006,158#14712,159#77078,160#24254,161#57777,162#126363,163#25048,164#21486,165#28449,166#50098,167#82983,168#19755,169#91077,170#20769,171#20005,172#13542,173#3762,174#19423,175#76019,176#37772,177#43040,178#8584,179#20538,180#28992,181#15110,182#16846,183#127675,184#56701,185#33299,186#129077,187#38940,188#108275,189#28650,190#21171,191#77984,192#5239,193#34959,194#14433,195#55604,196#120194",
"master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0,30#0,31#0,32#0,33#0,34#0,35#0,36#0,37#0,38#0,39#0,40#0,41#0,42#0,43#0,44#0,45#0,46#0,47#0,48#0,49#0,50#0,51#0,52#0,53#0,54#0,55#0,56#0,57#0,58#0,59#0,60#0,61#0,62#0,63#0,64#0,65#0,66#0,67#0,68#0,69#0,70#0,71#0,72#0,73#0,74#0,75#0,76#0,77#0,78#0,79#0,80#0,81#0,82#0,83#0,84#0,85#0,86#0,87#0,88#0,89#0,90#0,91#0,92#0,93#0,94#0,95#0,96#0,97#0,98#0,99#0,100#0,101#0,102#0,103#0,104#0,105#0,106#0,107#0,108#0,109#0,110#0,111#0,112#0,113#0,114#0,115#0,116#0,117#0,118#0,119#0,120#0,121#0,122#0,123#0,124#0,125#0,126#0,127#0,128#0,129#0,130#0,131#0,132#0,133#0,134#0,135#0,136#0,137#0,138#0,139#0,140#0,141#0,142#0,143#0,144#0,145#0,146#0,147#0,148#0,149#0,150#0,151#0,152#0,153#0,154#0,155#0,156#0,157#0,158#0,159#0,160#0,161#0,162#0,163#0,164#0,165#0,166#0,167#0,168#0,169#0,170#0,171#0,172#0,173#0,174#0,175#0,176#0,177#0,178#0,179#0,180#0,181#0,182#0,183#0,184#0,185#0,186#0,187#0,188#0,189#0,190#0,191#0,192#0,193#0,194#0,195#0,196#0",
"mtime": "2022-12-19T23:20:42.208563Z",
"creation_time": "2022-07-04T18:43:39.231681Z",
"max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#,11#,12#,13#,14#,15#,16#,17#,18#,19#,20#,21#,22#,23#,24#,25#,26#,27#,28#,29#,30#,31#,32#,33#,34#,35#,36#,37#,38#,39#,40#,41#,42#,43#,44#,45#,46#,47#,48#,49#,50#,51#,52#,53#,54#,55#,56#,57#,58#,59#,60#,61#,62#,63#,64#,65#,66#,67#,68#,69#,70#,71#,72#,73#,74#,75#,76#,77#,78#,79#,80#,81#,82#,83#,84#,85#,86#,87#,88#,89#,90#,91#,92#,93#,94#,95#,96#,97#,98#,99#,100#,101#,102#,103#,104#,105#,106#,107#,108#,109#,110#,111#,112#,113#,114#,115#,116#,117#,118#,119#,120#,121#,122#,123#,124#,125#,126#,127#,128#,129#,130#,131#,132#,133#,134#,135#,136#,137#,138#,139#,140#,141#,142#,143#,144#,145#,146#,147#,148#,149#,150#,151#,152#,153#,154#,155#,156#,157#,158#,159#,160#,161#,162#,163#,164#,165#,166#,167#,168#,169#,170#,171#,172#,173#,174#,175#,176#,177#,178#,179#,180#,181#,182#,183#,184#,185#,186#,187#,188#,189#,190#,191#,192#,193#,194#,195#,196#",
"usage": {
"rgw.main": {
"size": 206156753500892,
"size_actual": 206174454808576,
"size_utilized": 206156753500892,
"size_kb": 201324954591,
"size_kb_actual": 201342241024,
"size_kb_utilized": 201324954591,
"num_objects": 16944078
}}
Still getting warning about large objects, now at 31 up from the initial 8 large objects found.
# ceph health detail
HEALTH_WARN 31 large omap objects; 1 pgs not deep-scrubbed in time
[WRN] LARGE_OMAP_OBJECTS: 31 large omap objects
31 large objects found in pool 'default.rgw.buckets.index'
Search the cluster log for 'Large omap object found' for more details.
[WRN] PG_NOT_DEEP_SCRUBBED: 1 pgs not deep-scrubbed in time
pg 17.76 not deep-scrubbed since 2022-12-28T05:09:22.700534-0500
# ceph -s
cluster:
health: HEALTH_WARN
31 large omap objects
1 pgs not deep-scrubbed in time
services:
mon: 3 daemons, quorum [...] (age 8d)
mgr: [...](active, since 5w), standbys: [...]
mds: 3 up:standby
osd: 50 osds: 50 up (since 2w), 50 in (since 5w)
rgw: 3 daemons active ([...])
task status:
data:
pools: 11 pools, 1009 pgs
objects: 75.89M objects, 225 TiB
usage: 373 TiB used, 249 TiB / 622 TiB avail
pgs: 1007 active+clean
1 active+clean+scrubbing
1 active+clean+scrubbing+deep
# grep ' Large omap object found.' /var/log/ceph/ceph.log
[...]
2023-01-09T06:21:07.121852-0500 osd.33 (osd.33) 11432 : cluster [WRN] Large omap object found. Object: 6:09fda159:::.dir.65b393bb-9d7d-4c08-a55e-7934f0064560.21438367.3.3:head PG: 6.9a85bf90 (6.0) Key count: 214960 Size (bytes): 66443716
2023-01-09T14:34:01.315247-0500 osd.33 (osd.33) 11453 : cluster [WRN] Large omap object found. Object: 6:90fa0c6c:::.dir.65b393bb-9d7d-4c08-a55e-7934f0064560.21438367.3.88:head PG: 6.36305f09 (6.9) Key count: 222468 Size (bytes): 68741290
2023-01-09T14:34:12.276768-0500 osd.33 (osd.33) 11454 : cluster [WRN] Large omap object found. Object: 6:9642e5e8:::.dir.65b393bb-9d7d-4c08-a55e-7934f0064560.21438367.3.186:head PG: 6.17a74269 (6.9) Key count: 239416 Size (bytes): 73845551
2023-01-09T14:34:29.206175-0500 osd.33 (osd.33) 11455 : cluster [WRN] Large omap object found. Object: 6:9f9be708:::.dir.65b393bb-9d7d-4c08-a55e-7934f0064560.21438367.3.75:head PG: 6.10e7d9f9 (6.9) Key count: 337766 Size (bytes): 104379073
[...]
# radosgw-admin bucket stats
[
{
"bucket": "s3-bucket",
"num_shards": 197,
"tenant": "",
"zonegroup": "[...]",
"placement_rule": "default-placement",
"explicit_placement": {
"data_pool": "",
"data_extra_pool": "",
"index_pool": ""
},
"index_type": "Normal",
"owner": "owner",
"ver": "0#74980,1#18081,2#188267,3#221442,4#123725,5#34059,6#106611,7#39907,8#179701,9#51548,10#20305,11#200272,12#30211,13#37878,14#22721,15#35845,16#167822,17#23097,18#35721,19#32512,20#144586,21#20189,22#87688,23#22479,24#34607,25#6692,26#35015,27#38888,28#31121,29#55014,30#42483,31#36938,32#202931,33#44493,34#29178,35#11588,36#47592,37#42085,38#27340,39#36008,40#11077,41#115142,42#22737,43#17950,44#266533,45#30256,46#100858,47#144545,48#37577,49#90390,50#58125,51#237849,52#53828,53#164441,54#33349,55#81632,56#73355,57#29466,58#81869,59#34558,60#54234,61#27455,62#235497,63#36500,64#76102,65#34690,66#10612,67#18718,68#33150,69#21702,70#65150,71#126144,72#22914,73#30838,74#250221,75#225236,76#34287,77#120700,78#59226,79#163612,80#30444,81#38140,82#11584,83#68199,84#36880,85#254883,86#28060,87#26871,88#29290,89#95864,90#44233,91#33201,92#18208,93#11541,94#27171,95#36662,96#32895,97#17693,98#119338,99#15044,100#118795,101#44310,102#30669,103#17068,104#32161,105#32920,106#16041,107#94520,108#27135,109#35399,110#38444,111#53650,112#20285,113#17607,114#11519,115#93718,116#36972,117#25299,118#44019,119#5346,120#63197,121#122723,122#92030,123#111157,124#46502,125#113354,126#15074,127#81594,128#18503,129#13313,130#21200,131#127973,132#41494,133#18115,134#17392,135#27703,136#22478,137#9787,138#64426,139#20955,140#64377,141#6789,142#123420,143#25533,144#16474,145#4011,146#337290,147#209891,148#39412,149#144821,150#46543,151#18178,152#22976,153#42171,154#14750,155#46559,156#12148,157#18006,158#14712,159#77078,160#24254,161#57777,162#126363,163#25048,164#21486,165#28449,166#50098,167#82983,168#19755,169#91077,170#20769,171#20005,172#13542,173#3762,174#19423,175#76019,176#37772,177#43040,178#8584,179#20538,180#28992,181#15110,182#16846,183#127675,184#56701,185#33299,186#129077,187#38940,188#108275,189#28650,190#21171,191#77984,192#5239,193#34959,194#14433,195#55604,196#120194",
"master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0,30#0,31#0,32#0,33#0,34#0,35#0,36#0,37#0,38#0,39#0,40#0,41#0,42#0,43#0,44#0,45#0,46#0,47#0,48#0,49#0,50#0,51#0,52#0,53#0,54#0,55#0,56#0,57#0,58#0,59#0,60#0,61#0,62#0,63#0,64#0,65#0,66#0,67#0,68#0,69#0,70#0,71#0,72#0,73#0,74#0,75#0,76#0,77#0,78#0,79#0,80#0,81#0,82#0,83#0,84#0,85#0,86#0,87#0,88#0,89#0,90#0,91#0,92#0,93#0,94#0,95#0,96#0,97#0,98#0,99#0,100#0,101#0,102#0,103#0,104#0,105#0,106#0,107#0,108#0,109#0,110#0,111#0,112#0,113#0,114#0,115#0,116#0,117#0,118#0,119#0,120#0,121#0,122#0,123#0,124#0,125#0,126#0,127#0,128#0,129#0,130#0,131#0,132#0,133#0,134#0,135#0,136#0,137#0,138#0,139#0,140#0,141#0,142#0,143#0,144#0,145#0,146#0,147#0,148#0,149#0,150#0,151#0,152#0,153#0,154#0,155#0,156#0,157#0,158#0,159#0,160#0,161#0,162#0,163#0,164#0,165#0,166#0,167#0,168#0,169#0,170#0,171#0,172#0,173#0,174#0,175#0,176#0,177#0,178#0,179#0,180#0,181#0,182#0,183#0,184#0,185#0,186#0,187#0,188#0,189#0,190#0,191#0,192#0,193#0,194#0,195#0,196#0",
"mtime": "2022-12-19T23:20:42.208563Z",
"creation_time": "2022-07-04T18:43:39.231681Z",
"max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#,11#,12#,13#,14#,15#,16#,17#,18#,19#,20#,21#,22#,23#,24#,25#,26#,27#,28#,29#,30#,31#,32#,33#,34#,35#,36#,37#,38#,39#,40#,41#,42#,43#,44#,45#,46#,47#,48#,49#,50#,51#,52#,53#,54#,55#,56#,57#,58#,59#,60#,61#,62#,63#,64#,65#,66#,67#,68#,69#,70#,71#,72#,73#,74#,75#,76#,77#,78#,79#,80#,81#,82#,83#,84#,85#,86#,87#,88#,89#,90#,91#,92#,93#,94#,95#,96#,97#,98#,99#,100#,101#,102#,103#,104#,105#,106#,107#,108#,109#,110#,111#,112#,113#,114#,115#,116#,117#,118#,119#,120#,121#,122#,123#,124#,125#,126#,127#,128#,129#,130#,131#,132#,133#,134#,135#,136#,137#,138#,139#,140#,141#,142#,143#,144#,145#,146#,147#,148#,149#,150#,151#,152#,153#,154#,155#,156#,157#,158#,159#,160#,161#,162#,163#,164#,165#,166#,167#,168#,169#,170#,171#,172#,173#,174#,175#,176#,177#,178#,179#,180#,181#,182#,183#,184#,185#,186#,187#,188#,189#,190#,191#,192#,193#,194#,195#,196#",
"usage": {
"rgw.main": {
"size": 206156753500892,
"size_actual": 206174454808576,
"size_utilized": 206156753500892,
"size_kb": 201324954591,
"size_kb_actual": 201342241024,
"size_kb_utilized": 201324954591,
"num_objects": 16944078
}}
f.cuseo
66 Posts
Quote from f.cuseo on January 29, 2024, 11:41 amHave you fixed this issue ?
I am experiencing the same problem...
Regards, Fabrizio
Have you fixed this issue ?
I am experiencing the same problem...
Regards, Fabrizio
admin
2,930 Posts
Quote from admin on January 29, 2024, 4:43 pmi recommend you check the ceph info available online regarding this issue.
i recommend you check the ceph info available online regarding this issue.
f.cuseo
66 Posts
Quote from f.cuseo on January 29, 2024, 4:49 pmI have done it, tried some fix, but without any result...
I have done it, tried some fix, but without any result...
f.cuseo
66 Posts
Quote from f.cuseo on January 30, 2024, 4:33 pmFixed with a
ceph config set osd osd_deep_scrub_large_omap_object_key_threshold 400000 but I think that rgw bucket index should be automatically resharded...
Fixed with a
ceph config set osd osd_deep_scrub_large_omap_object_key_threshold 400000
but I think that rgw bucket index should be automatically resharded...