Injectargs for osd_memory_target needs a restart?
trexman
60 Posts
January 13, 2020, 9:37 amQuote from trexman on January 13, 2020, 9:37 amHello,
we increased the memory usage per OSD from 2GB to 4GB.
http://www.petasan.org/forums/?view=thread&id=498&part=1#postid-2928
Referring to your answer we changed the ceph.conf on all nodes and used the ceph injectargs to change it online:
ceph tell osd.* injectargs '--osd_memory_target 4294967296'
If i check the configuration for a few OSDs with "ceph daemon osd.0 config show |grep osd_memory_target" i see the new set value is "4294967296".
But it seams that the OSD are still consuming only 2GB of RAM, if I check it via "ps".
If i do a reboot of one node the memory consumption of an OSD is at 4GB.
So it seams that you can't change it online without a reboot.
Or do I have to restart a service? (or of all OSDs?)
Thanks.
Hello,
we increased the memory usage per OSD from 2GB to 4GB.
http://www.petasan.org/forums/?view=thread&id=498&part=1#postid-2928
Referring to your answer we changed the ceph.conf on all nodes and used the ceph injectargs to change it online:
ceph tell osd.* injectargs '--osd_memory_target 4294967296'
If i check the configuration for a few OSDs with "ceph daemon osd.0 config show |grep osd_memory_target" i see the new set value is "4294967296".
But it seams that the OSD are still consuming only 2GB of RAM, if I check it via "ps".
If i do a reboot of one node the memory consumption of an OSD is at 4GB.
So it seams that you can't change it online without a reboot.
Or do I have to restart a service? (or of all OSDs?)
Thanks.
Last edited on January 13, 2020, 9:39 am by trexman · #1
admin
2,930 Posts
January 13, 2020, 11:15 amQuote from admin on January 13, 2020, 11:15 amit should work without restart, note that a restart will read from conf file and any prev injectargs will be lost. also config help command shows it can be set runtime:
ceph config help osd_memory_target
Can update at runtime: true
i say should because in some cases this help command may not be accurate, but many users online do to set osd_memory_target using injectargs, so it should work.
It could be that you need to read/write the osds so it will slowly use the extra cache, so it may take some time.
Of course if you can afford it, the certain method is to update you conf file on all osds nodes, you can restart the osds via systemctl restart ceph-osd.target on each node, you do not need a full reboot.
In v 2.5 we are adding centralized ceph configuration, which saves the config on ceph monitor database rather than separate conf files on nodes, this was added in Nautilus.
it should work without restart, note that a restart will read from conf file and any prev injectargs will be lost. also config help command shows it can be set runtime:
ceph config help osd_memory_target
Can update at runtime: true
i say should because in some cases this help command may not be accurate, but many users online do to set osd_memory_target using injectargs, so it should work.
It could be that you need to read/write the osds so it will slowly use the extra cache, so it may take some time.
Of course if you can afford it, the certain method is to update you conf file on all osds nodes, you can restart the osds via systemctl restart ceph-osd.target on each node, you do not need a full reboot.
In v 2.5 we are adding centralized ceph configuration, which saves the config on ceph monitor database rather than separate conf files on nodes, this was added in Nautilus.
Last edited on January 13, 2020, 11:16 am by admin · #2
Injectargs for osd_memory_target needs a restart?
trexman
60 Posts
Quote from trexman on January 13, 2020, 9:37 amHello,
we increased the memory usage per OSD from 2GB to 4GB.
http://www.petasan.org/forums/?view=thread&id=498&part=1#postid-2928
Referring to your answer we changed the ceph.conf on all nodes and used the ceph injectargs to change it online:
ceph tell osd.* injectargs '--osd_memory_target 4294967296'
If i check the configuration for a few OSDs with "ceph daemon osd.0 config show |grep osd_memory_target" i see the new set value is "4294967296".
But it seams that the OSD are still consuming only 2GB of RAM, if I check it via "ps".
If i do a reboot of one node the memory consumption of an OSD is at 4GB.So it seams that you can't change it online without a reboot.
Or do I have to restart a service? (or of all OSDs?)Thanks.
Hello,
we increased the memory usage per OSD from 2GB to 4GB.
http://www.petasan.org/forums/?view=thread&id=498&part=1#postid-2928
Referring to your answer we changed the ceph.conf on all nodes and used the ceph injectargs to change it online:
ceph tell osd.* injectargs '--osd_memory_target 4294967296'
If i check the configuration for a few OSDs with "ceph daemon osd.0 config show |grep osd_memory_target" i see the new set value is "4294967296".
But it seams that the OSD are still consuming only 2GB of RAM, if I check it via "ps".
If i do a reboot of one node the memory consumption of an OSD is at 4GB.
So it seams that you can't change it online without a reboot.
Or do I have to restart a service? (or of all OSDs?)
Thanks.
admin
2,930 Posts
Quote from admin on January 13, 2020, 11:15 amit should work without restart, note that a restart will read from conf file and any prev injectargs will be lost. also config help command shows it can be set runtime:
ceph config help osd_memory_target
Can update at runtime: truei say should because in some cases this help command may not be accurate, but many users online do to set osd_memory_target using injectargs, so it should work.
It could be that you need to read/write the osds so it will slowly use the extra cache, so it may take some time.
Of course if you can afford it, the certain method is to update you conf file on all osds nodes, you can restart the osds via systemctl restart ceph-osd.target on each node, you do not need a full reboot.
In v 2.5 we are adding centralized ceph configuration, which saves the config on ceph monitor database rather than separate conf files on nodes, this was added in Nautilus.
it should work without restart, note that a restart will read from conf file and any prev injectargs will be lost. also config help command shows it can be set runtime:
ceph config help osd_memory_target
Can update at runtime: true
i say should because in some cases this help command may not be accurate, but many users online do to set osd_memory_target using injectargs, so it should work.
It could be that you need to read/write the osds so it will slowly use the extra cache, so it may take some time.
Of course if you can afford it, the certain method is to update you conf file on all osds nodes, you can restart the osds via systemctl restart ceph-osd.target on each node, you do not need a full reboot.
In v 2.5 we are adding centralized ceph configuration, which saves the config on ceph monitor database rather than separate conf files on nodes, this was added in Nautilus.