Forums

Home / Forums

You need to log in to create posts and topics. Login · Register

After nautilus upgrade - cluster stays in health warning


alienn
37 Posts
October 30, 2019, 11:44 pm

Hi,

I'm in the process of upgrading the petasan cluster from 2.3.0 to 2.3.1.1 with upgrade to nautilus.

While the main upgrade went well I now have some messages that I don't quite know how to silence.

  1. BlueFS spillover detected on 24 OSD(s)
    I know that the journaling ssd partition I initially created for the osd is quite small (20GB for 10TB OSD), but for the traffic on the cluster id should be enough. Has anyone an idea on how to fix this?
  2. 44 pgs not deep-scrubbed in time
    The number of not deep-scrubed placement groups increases and I have no idea why. Does anyone have a good advice on this?

 

Cheers,

 

#1

admin
2,930 Posts
October 31, 2019, 12:57 pm

1  You can disable the warning:

In conf file add in osd section add:

bluestore_warn_on_bluefs_spillover=false

+ inject the val in all running OSDs :

ceph tell osd.* injectargs '--bluestore_warn_on_bluefs_spillover=false'

This will disable the warning, you can do this if you wish. If you wish to correct it, expand it to 60 GB which is what we use. This partition is used for both journal and metadata db. You can use the ceph-bluestore-tool to expand the size:

https://docs.ceph.com/docs/master/man/8/ceph-bluestore-tool/

 

2- For the deep scrub issue, please see

http://www.petasan.org/forums/?view=thread&id=511

 

 

Last edited on October 31, 2019, 12:58 pm by admin · #2

alienn
37 Posts
November 5, 2019, 2:18 pm
  1. BlueFS Spillover
    I don't get this message really. The message says that it spilled over about 3GiB  while only 3GiB are used of 20GiB. See screenshot
    BlueFS spillover
  2. Deep scrub
    I now raised the limit to 0.6. When I understand http://bit.ly/34HzvFx correctly this value is calculated as <current load>/<num cpus of host>. Right now the load on the nodes is (according to top) at between 2.5 and 3 (see screenshot). So the normalized load should be 3/8 or 3/16 (with HT), right?
    Top screenshot
#3

alienn
37 Posts
November 6, 2019, 2:00 pm

By upping the load threshold I was able to reduce the not deep scrubbed pgs to 8. I will wait some time to see wether these will go away or not.

Do you have any good advice on (deep) scrubbing in general? Is it possible to move the heavy lifting to the weekend and do the (deep) scrubs e.g. on sunday?

#4

admin
2,930 Posts
November 6, 2019, 8:50 pm

it depends a lot on your hardware and load, search the forum here for pointers. example

The recommended way is to control sleep time and threshold:
osd_scrub_sleep = 1
osd_scrub_load_threshold = 0.3
you can change the value depending on impact to your client io, increasing the osd_scrub_sleep and decreasing osd_scrub_load_threshold will lower scrub processing. You cannot lower it too much as scrubs should be done every day and deep scrubs every week.

You can also use:
osd_scrub_begin_week_day
osd_scrub_end_week_day

To take full control, you can run
ceph pg deep-scrub PG
yourself in a script via cron job, you can loop through all pgs via
ceph pg dump
and sort them via DEEP_SCRUB_STAMP

#5