OSD latency problems (VMware stuck)
Pages: 1 2
trexman
60 Posts
October 23, 2019, 11:02 amQuote from trexman on October 23, 2019, 11:02 amafter reboot you see normal OSD latency, but increase with time ?
Yes. This is what we see since the 3 last reboots.
% utilization also increases with time ?
I would say no.
Here are some graphs with IOPS, OSD latency and Disk utilization for the last:
6h: https://imgur.com/Go5jTZ9
24h: https://imgur.com/x0glmJ5 (reboot was yesterday at around 21:00)
week: https://imgur.com/FjXJQWx
after reboot you see normal OSD latency, but increase with time ?
Yes. This is what we see since the 3 last reboots.
% utilization also increases with time ?
I would say no.
Here are some graphs with IOPS, OSD latency and Disk utilization for the last:
6h: https://imgur.com/Go5jTZ9
24h: https://imgur.com/x0glmJ5 (reboot was yesterday at around 21:00)
week: https://imgur.com/FjXJQWx
therm
121 Posts
October 23, 2019, 11:14 amQuote from therm on October 23, 2019, 11:14 amhi trexman,
you are using consumer ssds. We did also (Samsung 850/860 pro). Those SSDs collapse at a specific amount of write load (due to their internal cache beeing filled). We ended up in an setup using intels 2TB DC P4600 as journal devices with filestore and 4TB hgst disks. We use an all flash pool only for those data were it is necessary(on those P4600), that keeps it affordable.
Regards,
Dennis
hi trexman,
you are using consumer ssds. We did also (Samsung 850/860 pro). Those SSDs collapse at a specific amount of write load (due to their internal cache beeing filled). We ended up in an setup using intels 2TB DC P4600 as journal devices with filestore and 4TB hgst disks. We use an all flash pool only for those data were it is necessary(on those P4600), that keeps it affordable.
Regards,
Dennis
admin
2,930 Posts
October 23, 2019, 11:22 amQuote from admin on October 23, 2019, 11:22 amYou could be experiencing this
https://tracker.ceph.com/issues/38738
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-March/033476.html
The default allocator in Luminous and Mimic (called stupid allocator) may experience this under some cases. A new allocator in Nautilus bitmap allocator was backported in Luminous 12.2.12 which is PetaSAN v 2.3.0 and 2.3.1 before Nautilus upgrade. If you are using 12.2.12, switch to this allocator, in the conf file in [osd] section:
[osd]
bluestore_allocator = bitmap
bluefs_allocator = bitmap
then restart all osd on node
systemctl restart ceph-osd.target
i suggest you do this on 1 node first and observe with time. you can check if it applied via:
ceph daemon osd.X config show --cluster XX | grep allocator
You could be experiencing this
https://tracker.ceph.com/issues/38738
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-March/033476.html
The default allocator in Luminous and Mimic (called stupid allocator) may experience this under some cases. A new allocator in Nautilus bitmap allocator was backported in Luminous 12.2.12 which is PetaSAN v 2.3.0 and 2.3.1 before Nautilus upgrade. If you are using 12.2.12, switch to this allocator, in the conf file in [osd] section:
[osd]
bluestore_allocator = bitmap
bluefs_allocator = bitmap
then restart all osd on node
systemctl restart ceph-osd.target
i suggest you do this on 1 node first and observe with time. you can check if it applied via:
ceph daemon osd.X config show --cluster XX | grep allocator
trexman
60 Posts
October 25, 2019, 6:03 pmQuote from trexman on October 25, 2019, 6:03 pm@therm:
Thanks for sharing your experience. We would have to think about this.
@admin:
Thanks so much for the research.
If I see this right, than an update of our cluster from PetaSAN 2.3.0 to 2.3.1 (Ceph Nautilus) would also change the allocator to bitmap right?
And I can check it with the last command.
Thank you
@therm:
Thanks for sharing your experience. We would have to think about this.
@admin:
Thanks so much for the research.
If I see this right, than an update of our cluster from PetaSAN 2.3.0 to 2.3.1 (Ceph Nautilus) would also change the allocator to bitmap right?
And I can check it with the last command.
Thank you
admin
2,930 Posts
October 25, 2019, 10:07 pmQuote from admin on October 25, 2019, 10:07 pmYes many consumer grade SSDs will cause issues,
If you upgrade to 2.3.1 and run the Nautilus upgrade steps, yes it will be built it, you do not need the --cluster XX in the commands with 2.3.1. If you want to test the new allocator with 2.3.0, you can use the steps posted earlier.
Yes many consumer grade SSDs will cause issues,
If you upgrade to 2.3.1 and run the Nautilus upgrade steps, yes it will be built it, you do not need the --cluster XX in the commands with 2.3.1. If you want to test the new allocator with 2.3.0, you can use the steps posted earlier.
trexman
60 Posts
November 19, 2019, 1:29 pmQuote from trexman on November 19, 2019, 1:29 pmJust as a short summery:
After the PetaSAN upgrade to V 2.3.1 with the allocator bitmap, our problem did not occur again as you can see here:
Thanks for your great help!
Just as a short summery:
After the PetaSAN upgrade to V 2.3.1 with the allocator bitmap, our problem did not occur again as you can see here:
Thanks for your great help!
admin
2,930 Posts
November 20, 2019, 9:39 pmQuote from admin on November 20, 2019, 9:39 pmExcellent, thanks for posting..
Excellent, thanks for posting..
mfrater
1 Post
November 25, 2019, 2:07 amQuote from mfrater on November 25, 2019, 2:07 amWe had the same issue recently. HDD commit latency was fine. SSD commit latency was awful. Latency increased over a number of hours to > 100-200ms.
We have (only) 3 SSDs split across 3 hosts. We changed bluestore_allocator to "bitmap" on each host one at a time as per process above. Worked a treat. Here are the results.
We had the same issue recently. HDD commit latency was fine. SSD commit latency was awful. Latency increased over a number of hours to > 100-200ms.
We have (only) 3 SSDs split across 3 hosts. We changed bluestore_allocator to "bitmap" on each host one at a time as per process above. Worked a treat. Here are the results.
Pages: 1 2
OSD latency problems (VMware stuck)
trexman
60 Posts
Quote from trexman on October 23, 2019, 11:02 amafter reboot you see normal OSD latency, but increase with time ?
Yes. This is what we see since the 3 last reboots.
% utilization also increases with time ?
I would say no.
Here are some graphs with IOPS, OSD latency and Disk utilization for the last:
6h: https://imgur.com/Go5jTZ9
24h: https://imgur.com/x0glmJ5 (reboot was yesterday at around 21:00)
week: https://imgur.com/FjXJQWx
after reboot you see normal OSD latency, but increase with time ?
Yes. This is what we see since the 3 last reboots.
% utilization also increases with time ?
I would say no.
Here are some graphs with IOPS, OSD latency and Disk utilization for the last:
6h: https://imgur.com/Go5jTZ9
24h: https://imgur.com/x0glmJ5 (reboot was yesterday at around 21:00)
week: https://imgur.com/FjXJQWx
therm
121 Posts
Quote from therm on October 23, 2019, 11:14 amhi trexman,
you are using consumer ssds. We did also (Samsung 850/860 pro). Those SSDs collapse at a specific amount of write load (due to their internal cache beeing filled). We ended up in an setup using intels 2TB DC P4600 as journal devices with filestore and 4TB hgst disks. We use an all flash pool only for those data were it is necessary(on those P4600), that keeps it affordable.
Regards,
Dennis
hi trexman,
you are using consumer ssds. We did also (Samsung 850/860 pro). Those SSDs collapse at a specific amount of write load (due to their internal cache beeing filled). We ended up in an setup using intels 2TB DC P4600 as journal devices with filestore and 4TB hgst disks. We use an all flash pool only for those data were it is necessary(on those P4600), that keeps it affordable.
Regards,
Dennis
admin
2,930 Posts
Quote from admin on October 23, 2019, 11:22 amYou could be experiencing this
https://tracker.ceph.com/issues/38738
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-March/033476.html
The default allocator in Luminous and Mimic (called stupid allocator) may experience this under some cases. A new allocator in Nautilus bitmap allocator was backported in Luminous 12.2.12 which is PetaSAN v 2.3.0 and 2.3.1 before Nautilus upgrade. If you are using 12.2.12, switch to this allocator, in the conf file in [osd] section:
[osd]
bluestore_allocator = bitmap
bluefs_allocator = bitmapthen restart all osd on node
systemctl restart ceph-osd.target
i suggest you do this on 1 node first and observe with time. you can check if it applied via:
ceph daemon osd.X config show --cluster XX | grep allocator
You could be experiencing this
https://tracker.ceph.com/issues/38738
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-March/033476.html
The default allocator in Luminous and Mimic (called stupid allocator) may experience this under some cases. A new allocator in Nautilus bitmap allocator was backported in Luminous 12.2.12 which is PetaSAN v 2.3.0 and 2.3.1 before Nautilus upgrade. If you are using 12.2.12, switch to this allocator, in the conf file in [osd] section:
[osd]
bluestore_allocator = bitmap
bluefs_allocator = bitmap
then restart all osd on node
systemctl restart ceph-osd.target
i suggest you do this on 1 node first and observe with time. you can check if it applied via:
ceph daemon osd.X config show --cluster XX | grep allocator
trexman
60 Posts
Quote from trexman on October 25, 2019, 6:03 pm@therm:
Thanks for sharing your experience. We would have to think about this.@admin:
Thanks so much for the research.If I see this right, than an update of our cluster from PetaSAN 2.3.0 to 2.3.1 (Ceph Nautilus) would also change the allocator to bitmap right?
And I can check it with the last command.Thank you
@therm:
Thanks for sharing your experience. We would have to think about this.
@admin:
Thanks so much for the research.
If I see this right, than an update of our cluster from PetaSAN 2.3.0 to 2.3.1 (Ceph Nautilus) would also change the allocator to bitmap right?
And I can check it with the last command.
Thank you
admin
2,930 Posts
Quote from admin on October 25, 2019, 10:07 pmYes many consumer grade SSDs will cause issues,
If you upgrade to 2.3.1 and run the Nautilus upgrade steps, yes it will be built it, you do not need the --cluster XX in the commands with 2.3.1. If you want to test the new allocator with 2.3.0, you can use the steps posted earlier.
Yes many consumer grade SSDs will cause issues,
If you upgrade to 2.3.1 and run the Nautilus upgrade steps, yes it will be built it, you do not need the --cluster XX in the commands with 2.3.1. If you want to test the new allocator with 2.3.0, you can use the steps posted earlier.
trexman
60 Posts
Quote from trexman on November 19, 2019, 1:29 pmJust as a short summery:
After the PetaSAN upgrade to V 2.3.1 with the allocator bitmap, our problem did not occur again as you can see here:
Thanks for your great help!
Just as a short summery:
After the PetaSAN upgrade to V 2.3.1 with the allocator bitmap, our problem did not occur again as you can see here:
Thanks for your great help!
admin
2,930 Posts
Quote from admin on November 20, 2019, 9:39 pmExcellent, thanks for posting..
Excellent, thanks for posting..
mfrater
1 Post
Quote from mfrater on November 25, 2019, 2:07 amWe had the same issue recently. HDD commit latency was fine. SSD commit latency was awful. Latency increased over a number of hours to > 100-200ms.
We have (only) 3 SSDs split across 3 hosts. We changed bluestore_allocator to "bitmap" on each host one at a time as per process above. Worked a treat. Here are the results.
We had the same issue recently. HDD commit latency was fine. SSD commit latency was awful. Latency increased over a number of hours to > 100-200ms.
We have (only) 3 SSDs split across 3 hosts. We changed bluestore_allocator to "bitmap" on each host one at a time as per process above. Worked a treat. Here are the results.