Deep Scrubbing killing cluster performance
wailer
75 Posts
May 5, 2019, 12:38 amQuote from wailer on May 5, 2019, 12:38 amHi ,
Since 1 week ago our cluster started dying every weekend , I suspect this is being caused by deep-scrubbing. Here are our current scrub settings:
osd_max_scrubs = 1
osd_scrub_during_recovery = false
osd_scrub_priority = 1
osd_scrub_sleep = 20
osd_scrub_chunk_min = 1
osd_scrub_chunk_max = 1
osd_scrub_load_threshold = 0.15
osd_scrub_begin_hour = 23
osd_scrub_end_hour = 6
Any advise on this ?
Our cluster details:
3x HP G8 Node with SSD Journal and 6x6TB HD each
Thanks!
Hi ,
Since 1 week ago our cluster started dying every weekend , I suspect this is being caused by deep-scrubbing. Here are our current scrub settings:
osd_max_scrubs = 1
osd_scrub_during_recovery = false
osd_scrub_priority = 1
osd_scrub_sleep = 20
osd_scrub_chunk_min = 1
osd_scrub_chunk_max = 1
osd_scrub_load_threshold = 0.15
osd_scrub_begin_hour = 23
osd_scrub_end_hour = 6
Any advise on this ?
Our cluster details:
3x HP G8 Node with SSD Journal and 6x6TB HD each
Thanks!
admin
2,930 Posts
May 5, 2019, 8:55 amQuote from admin on May 5, 2019, 8:55 amYou can turn off scrub and deep scrub from maintenance tab and see the problem goes away or not. Also to double check did you restart the osd services when you change the config file for the settings to take effect.
Also if they happen only on certain days, it could be some other issues: running backup workloads or maybe network maintenance. Look at the load charts for cpu, disk, network % busy it may give you some more info on when this happens and what resources are involved.
In many cases the cluster could die if the osd cannot heartbeat each other on time, could be system under too much load, also your backend network needs to be 10G or more so heartbeat traffic will not be interrupted by other traffic.
You can turn off scrub and deep scrub from maintenance tab and see the problem goes away or not. Also to double check did you restart the osd services when you change the config file for the settings to take effect.
Also if they happen only on certain days, it could be some other issues: running backup workloads or maybe network maintenance. Look at the load charts for cpu, disk, network % busy it may give you some more info on when this happens and what resources are involved.
In many cases the cluster could die if the osd cannot heartbeat each other on time, could be system under too much load, also your backend network needs to be 10G or more so heartbeat traffic will not be interrupted by other traffic.
wailer
75 Posts
May 5, 2019, 3:05 pmQuote from wailer on May 5, 2019, 3:05 pmThanks,
I applied the config changeswith inject_args command, so I have restarted all OSD daemons, just in case.
Our backend is currently 4 x1GBit (2 bonds) , and iSCSI is 2x1Gbit , checking network graphs I don't see any overload there, only a 200Mbits peak in one node's backend. Maybe in the future.
I will disable deep-scrub for now, so we make sure that this is the source of issue.
Thanks,
I applied the config changeswith inject_args command, so I have restarted all OSD daemons, just in case.
Our backend is currently 4 x1GBit (2 bonds) , and iSCSI is 2x1Gbit , checking network graphs I don't see any overload there, only a 200Mbits peak in one node's backend. Maybe in the future.
I will disable deep-scrub for now, so we make sure that this is the source of issue.
wailer
75 Posts
May 6, 2019, 3:17 pmQuote from wailer on May 6, 2019, 3:17 pmJust a quick update on this:
Yesterday one of the hard drives was marked out of the cluster. After doing some more tests it turned out to be defective because of bad sectors. So this brings some doubts to me:
- Is it that easy ? I mean A single failing or an about-to-fail drive can bring a whole ceph cluster down ? Maybe there are some tweaks I could do so it gets marked out before starts creating real troubles?
Thanks!
Just a quick update on this:
Yesterday one of the hard drives was marked out of the cluster. After doing some more tests it turned out to be defective because of bad sectors. So this brings some doubts to me:
- Is it that easy ? I mean A single failing or an about-to-fail drive can bring a whole ceph cluster down ? Maybe there are some tweaks I could do so it gets marked out before starts creating real troubles?
Thanks!
admin
2,930 Posts
May 6, 2019, 9:28 pmQuote from admin on May 6, 2019, 9:28 pmNo, it should not happen.
If you are unlucky and the drive dies slowly, the worst thing i can think of is it will degrade performance based on how slow it is and its ratio of how many other good disks you have. But i guess such slow die-ing cases will impact any applications not just PetaSAN. You can spot them from the charts, such as osd latency and disk % busy, typically all osds should be stressed the same, but for these cases will stand out as being much slower. Also we report SMART data and send email notification if a disk fails it, so it should detect a large percentage of disks before going bad.
No, it should not happen.
If you are unlucky and the drive dies slowly, the worst thing i can think of is it will degrade performance based on how slow it is and its ratio of how many other good disks you have. But i guess such slow die-ing cases will impact any applications not just PetaSAN. You can spot them from the charts, such as osd latency and disk % busy, typically all osds should be stressed the same, but for these cases will stand out as being much slower. Also we report SMART data and send email notification if a disk fails it, so it should detect a large percentage of disks before going bad.
Last edited on May 6, 2019, 9:34 pm by admin · #5
Deep Scrubbing killing cluster performance
wailer
75 Posts
Quote from wailer on May 5, 2019, 12:38 amHi ,
Since 1 week ago our cluster started dying every weekend , I suspect this is being caused by deep-scrubbing. Here are our current scrub settings:
osd_max_scrubs = 1
osd_scrub_during_recovery = false
osd_scrub_priority = 1
osd_scrub_sleep = 20
osd_scrub_chunk_min = 1
osd_scrub_chunk_max = 1
osd_scrub_load_threshold = 0.15
osd_scrub_begin_hour = 23
osd_scrub_end_hour = 6Any advise on this ?
Our cluster details:
3x HP G8 Node with SSD Journal and 6x6TB HD each
Thanks!
Hi ,
Since 1 week ago our cluster started dying every weekend , I suspect this is being caused by deep-scrubbing. Here are our current scrub settings:
osd_max_scrubs = 1
osd_scrub_during_recovery = false
osd_scrub_priority = 1
osd_scrub_sleep = 20
osd_scrub_chunk_min = 1
osd_scrub_chunk_max = 1
osd_scrub_load_threshold = 0.15
osd_scrub_begin_hour = 23
osd_scrub_end_hour = 6
Any advise on this ?
Our cluster details:
3x HP G8 Node with SSD Journal and 6x6TB HD each
Thanks!
admin
2,930 Posts
Quote from admin on May 5, 2019, 8:55 amYou can turn off scrub and deep scrub from maintenance tab and see the problem goes away or not. Also to double check did you restart the osd services when you change the config file for the settings to take effect.
Also if they happen only on certain days, it could be some other issues: running backup workloads or maybe network maintenance. Look at the load charts for cpu, disk, network % busy it may give you some more info on when this happens and what resources are involved.
In many cases the cluster could die if the osd cannot heartbeat each other on time, could be system under too much load, also your backend network needs to be 10G or more so heartbeat traffic will not be interrupted by other traffic.
You can turn off scrub and deep scrub from maintenance tab and see the problem goes away or not. Also to double check did you restart the osd services when you change the config file for the settings to take effect.
Also if they happen only on certain days, it could be some other issues: running backup workloads or maybe network maintenance. Look at the load charts for cpu, disk, network % busy it may give you some more info on when this happens and what resources are involved.
In many cases the cluster could die if the osd cannot heartbeat each other on time, could be system under too much load, also your backend network needs to be 10G or more so heartbeat traffic will not be interrupted by other traffic.
wailer
75 Posts
Quote from wailer on May 5, 2019, 3:05 pmThanks,
I applied the config changeswith inject_args command, so I have restarted all OSD daemons, just in case.
Our backend is currently 4 x1GBit (2 bonds) , and iSCSI is 2x1Gbit , checking network graphs I don't see any overload there, only a 200Mbits peak in one node's backend. Maybe in the future.
I will disable deep-scrub for now, so we make sure that this is the source of issue.
Thanks,
I applied the config changeswith inject_args command, so I have restarted all OSD daemons, just in case.
Our backend is currently 4 x1GBit (2 bonds) , and iSCSI is 2x1Gbit , checking network graphs I don't see any overload there, only a 200Mbits peak in one node's backend. Maybe in the future.
I will disable deep-scrub for now, so we make sure that this is the source of issue.
wailer
75 Posts
Quote from wailer on May 6, 2019, 3:17 pmJust a quick update on this:
Yesterday one of the hard drives was marked out of the cluster. After doing some more tests it turned out to be defective because of bad sectors. So this brings some doubts to me:
- Is it that easy ? I mean A single failing or an about-to-fail drive can bring a whole ceph cluster down ? Maybe there are some tweaks I could do so it gets marked out before starts creating real troubles?
Thanks!
Just a quick update on this:
Yesterday one of the hard drives was marked out of the cluster. After doing some more tests it turned out to be defective because of bad sectors. So this brings some doubts to me:
- Is it that easy ? I mean A single failing or an about-to-fail drive can bring a whole ceph cluster down ? Maybe there are some tweaks I could do so it gets marked out before starts creating real troubles?
Thanks!
admin
2,930 Posts
Quote from admin on May 6, 2019, 9:28 pmNo, it should not happen.
If you are unlucky and the drive dies slowly, the worst thing i can think of is it will degrade performance based on how slow it is and its ratio of how many other good disks you have. But i guess such slow die-ing cases will impact any applications not just PetaSAN. You can spot them from the charts, such as osd latency and disk % busy, typically all osds should be stressed the same, but for these cases will stand out as being much slower. Also we report SMART data and send email notification if a disk fails it, so it should detect a large percentage of disks before going bad.
No, it should not happen.
If you are unlucky and the drive dies slowly, the worst thing i can think of is it will degrade performance based on how slow it is and its ratio of how many other good disks you have. But i guess such slow die-ing cases will impact any applications not just PetaSAN. You can spot them from the charts, such as osd latency and disk % busy, typically all osds should be stressed the same, but for these cases will stand out as being much slower. Also we report SMART data and send email notification if a disk fails it, so it should detect a large percentage of disks before going bad.