no data points returned from metric query
Pages: 1 2
Yipkaiwing
18 Posts
September 13, 2018, 5:09 amQuote from Yipkaiwing on September 13, 2018, 5:09 amDear Admin
I found that the ram utilization reaches at about 75 - 95 % in ALL nodes. ( 3 management + 4 storage)
The iscsi function is fine. The cluster is still normal.
But the management nodes are not stable. It will be down for 10 - 20 seconds.
Now, there is a problem : no data points returned from metric query
How to solve it?
Thanks!
Ben
Dear Admin
I found that the ram utilization reaches at about 75 - 95 % in ALL nodes. ( 3 management + 4 storage)
The iscsi function is fine. The cluster is still normal.
But the management nodes are not stable. It will be down for 10 - 20 seconds.
Now, there is a problem : no data points returned from metric query
How to solve it?
Thanks!
Ben
admin
2,930 Posts
September 13, 2018, 5:36 amQuote from admin on September 13, 2018, 5:36 amHow much ram do you have per node ? how may osds per node ? do you use hdd or ssd ?
How much ram do you have per node ? how may osds per node ? do you use hdd or ssd ?
Yipkaiwing
18 Posts
September 13, 2018, 6:17 amQuote from Yipkaiwing on September 13, 2018, 6:17 amNode 1-3 are running in VM server with 2GB ram only
Node 4 and Node 7 with 40 GB Ram, having 6 OSD, all are SSD, no journal disk, each disk is 1TB
Node 5 and Node 6 with 32GB Ram, having 6 OSD, all are SSD, no journal disk, each disk is 1TB
The cluster is connecting to 2 VM server with 5 TB respectively.
No of PG is 1024, replicas is 2.
Now, the cluster storage and VM servers are still running normally.
Thanks, for your help and answer.
Ben
Node 1-3 are running in VM server with 2GB ram only
Node 4 and Node 7 with 40 GB Ram, having 6 OSD, all are SSD, no journal disk, each disk is 1TB
Node 5 and Node 6 with 32GB Ram, having 6 OSD, all are SSD, no journal disk, each disk is 1TB
The cluster is connecting to 2 VM server with 5 TB respectively.
No of PG is 1024, replicas is 2.
Now, the cluster storage and VM servers are still running normally.
Thanks, for your help and answer.
Ben
admin
2,930 Posts
September 13, 2018, 6:44 amQuote from admin on September 13, 2018, 6:44 amMost likely the stats function failed due to memory, but as you stated the entire cluster will not be stable as well. You should try to avoid high ram % usage. You may want to increase your ram on your management nodes to 4M
You should also considering lowering your osd cache memory via
bluestore_cache_size_ssd = 1GB
(the default value is 3G for bluestore) in your /etc/ceph/xx.conf file and restart osd services manually or reboot.
if this still does not solve it let me know
Most likely the stats function failed due to memory, but as you stated the entire cluster will not be stable as well. You should try to avoid high ram % usage. You may want to increase your ram on your management nodes to 4M
You should also considering lowering your osd cache memory via
bluestore_cache_size_ssd = 1GB
(the default value is 3G for bluestore) in your /etc/ceph/xx.conf file and restart osd services manually or reboot.
if this still does not solve it let me know
Last edited on September 13, 2018, 6:48 am by admin · #4
Yipkaiwing
18 Posts
September 13, 2018, 6:59 amQuote from Yipkaiwing on September 13, 2018, 6:59 amAdmin
Noted, I try now, thanks for your solution given
Ben
Admin
Noted, I try now, thanks for your solution given
Ben
Yipkaiwing
18 Posts
September 13, 2018, 7:11 amQuote from Yipkaiwing on September 13, 2018, 7:11 amAdmin
I just added the RAM to 4G on ALL management nodes.
Afterwards, I reboot all servers, the problem of "no data points returned from metric query' has been solved.
Thanks for your help.
By the way, I will try to add more RAM on the OSD data storage server. It seems that 32 GB is not enough.
Ben
Admin
I just added the RAM to 4G on ALL management nodes.
Afterwards, I reboot all servers, the problem of "no data points returned from metric query' has been solved.
Thanks for your help.
By the way, I will try to add more RAM on the OSD data storage server. It seems that 32 GB is not enough.
Ben
Yipkaiwing
18 Posts
September 13, 2018, 8:47 amQuote from Yipkaiwing on September 13, 2018, 8:47 amHi Admin
The storage and iscsi server has 32GB ram. There is 1 OS disk and 7 OSD SSD disks.
According to your hardware recommendation document, 2GB per disk and 16GB for iscsi service. So, my sever meets the hardware requirement.
8 * 2GB + 16GB + 32GB
But now, the RAM utilization is almost 100%. Is it normal? Shall I add more RAM?
Ben
Hi Admin
The storage and iscsi server has 32GB ram. There is 1 OS disk and 7 OSD SSD disks.
According to your hardware recommendation document, 2GB per disk and 16GB for iscsi service. So, my sever meets the hardware requirement.
8 * 2GB + 16GB + 32GB
But now, the RAM utilization is almost 100%. Is it normal? Shall I add more RAM?
Ben
admin
2,930 Posts
September 13, 2018, 9:28 amQuote from admin on September 13, 2018, 9:28 amRe using EVO disks, please see my post on SSD performance
Re using EVO disks, please see my post on SSD performance
Yipkaiwing
18 Posts
September 13, 2018, 10:50 amQuote from Yipkaiwing on September 13, 2018, 10:50 amThanks, I will try to replace the enterprises SSD or HDD
Thanks, I will try to replace the enterprises SSD or HDD
admin
2,930 Posts
September 13, 2018, 1:19 pmQuote from admin on September 13, 2018, 1:19 pmRe adding RAM, yes it is better. else set
bluestore_cache_size_ssd = 1GB
as indicated earlier.
You are correct, there are recent bluestore OSD tunings for SSDs that do use more memory than what the Ceph and PetaSAN docs state.
https://www.spinics.net/lists/ceph-devel/msg40592.html
We will change it
Re adding RAM, yes it is better. else set
bluestore_cache_size_ssd = 1GB
as indicated earlier.
You are correct, there are recent bluestore OSD tunings for SSDs that do use more memory than what the Ceph and PetaSAN docs state.
https://www.spinics.net/lists/ceph-devel/msg40592.html
We will change it
Last edited on September 13, 2018, 1:19 pm by admin · #10
Pages: 1 2
no data points returned from metric query
Yipkaiwing
18 Posts
Quote from Yipkaiwing on September 13, 2018, 5:09 amDear Admin
I found that the ram utilization reaches at about 75 - 95 % in ALL nodes. ( 3 management + 4 storage)
The iscsi function is fine. The cluster is still normal.
But the management nodes are not stable. It will be down for 10 - 20 seconds.
Now, there is a problem : no data points returned from metric query
How to solve it?
Thanks!
Ben
Dear Admin
I found that the ram utilization reaches at about 75 - 95 % in ALL nodes. ( 3 management + 4 storage)
The iscsi function is fine. The cluster is still normal.
But the management nodes are not stable. It will be down for 10 - 20 seconds.
Now, there is a problem : no data points returned from metric query
How to solve it?
Thanks!
Ben
admin
2,930 Posts
Quote from admin on September 13, 2018, 5:36 amHow much ram do you have per node ? how may osds per node ? do you use hdd or ssd ?
How much ram do you have per node ? how may osds per node ? do you use hdd or ssd ?
Yipkaiwing
18 Posts
Quote from Yipkaiwing on September 13, 2018, 6:17 amNode 1-3 are running in VM server with 2GB ram only
Node 4 and Node 7 with 40 GB Ram, having 6 OSD, all are SSD, no journal disk, each disk is 1TB
Node 5 and Node 6 with 32GB Ram, having 6 OSD, all are SSD, no journal disk, each disk is 1TB
The cluster is connecting to 2 VM server with 5 TB respectively.
No of PG is 1024, replicas is 2.
Now, the cluster storage and VM servers are still running normally.
Thanks, for your help and answer.
Ben
Node 1-3 are running in VM server with 2GB ram only
Node 4 and Node 7 with 40 GB Ram, having 6 OSD, all are SSD, no journal disk, each disk is 1TB
Node 5 and Node 6 with 32GB Ram, having 6 OSD, all are SSD, no journal disk, each disk is 1TB
The cluster is connecting to 2 VM server with 5 TB respectively.
No of PG is 1024, replicas is 2.
Now, the cluster storage and VM servers are still running normally.
Thanks, for your help and answer.
Ben
admin
2,930 Posts
Quote from admin on September 13, 2018, 6:44 amMost likely the stats function failed due to memory, but as you stated the entire cluster will not be stable as well. You should try to avoid high ram % usage. You may want to increase your ram on your management nodes to 4M
You should also considering lowering your osd cache memory via
bluestore_cache_size_ssd = 1GB
(the default value is 3G for bluestore) in your /etc/ceph/xx.conf file and restart osd services manually or reboot.
if this still does not solve it let me know
Most likely the stats function failed due to memory, but as you stated the entire cluster will not be stable as well. You should try to avoid high ram % usage. You may want to increase your ram on your management nodes to 4M
You should also considering lowering your osd cache memory via
bluestore_cache_size_ssd = 1GB
(the default value is 3G for bluestore) in your /etc/ceph/xx.conf file and restart osd services manually or reboot.
if this still does not solve it let me know
Yipkaiwing
18 Posts
Quote from Yipkaiwing on September 13, 2018, 6:59 amAdmin
Noted, I try now, thanks for your solution given
Ben
Admin
Noted, I try now, thanks for your solution given
Ben
Yipkaiwing
18 Posts
Quote from Yipkaiwing on September 13, 2018, 7:11 amAdmin
I just added the RAM to 4G on ALL management nodes.
Afterwards, I reboot all servers, the problem of "no data points returned from metric query' has been solved.
Thanks for your help.
By the way, I will try to add more RAM on the OSD data storage server. It seems that 32 GB is not enough.
Ben
Admin
I just added the RAM to 4G on ALL management nodes.
Afterwards, I reboot all servers, the problem of "no data points returned from metric query' has been solved.
Thanks for your help.
By the way, I will try to add more RAM on the OSD data storage server. It seems that 32 GB is not enough.
Ben
Yipkaiwing
18 Posts
Quote from Yipkaiwing on September 13, 2018, 8:47 amHi Admin
The storage and iscsi server has 32GB ram. There is 1 OS disk and 7 OSD SSD disks.
According to your hardware recommendation document, 2GB per disk and 16GB for iscsi service. So, my sever meets the hardware requirement.
8 * 2GB + 16GB + 32GB
But now, the RAM utilization is almost 100%. Is it normal? Shall I add more RAM?
Ben
Hi Admin
The storage and iscsi server has 32GB ram. There is 1 OS disk and 7 OSD SSD disks.
According to your hardware recommendation document, 2GB per disk and 16GB for iscsi service. So, my sever meets the hardware requirement.
8 * 2GB + 16GB + 32GB
But now, the RAM utilization is almost 100%. Is it normal? Shall I add more RAM?
Ben
admin
2,930 Posts
Quote from admin on September 13, 2018, 9:28 amRe using EVO disks, please see my post on SSD performance
Re using EVO disks, please see my post on SSD performance
Yipkaiwing
18 Posts
Quote from Yipkaiwing on September 13, 2018, 10:50 amThanks, I will try to replace the enterprises SSD or HDD
Thanks, I will try to replace the enterprises SSD or HDD
admin
2,930 Posts
Quote from admin on September 13, 2018, 1:19 pmRe adding RAM, yes it is better. else set
bluestore_cache_size_ssd = 1GB
as indicated earlier.
You are correct, there are recent bluestore OSD tunings for SSDs that do use more memory than what the Ceph and PetaSAN docs state.
https://www.spinics.net/lists/ceph-devel/msg40592.html
We will change it
Re adding RAM, yes it is better. else set
bluestore_cache_size_ssd = 1GB
as indicated earlier.
You are correct, there are recent bluestore OSD tunings for SSDs that do use more memory than what the Ceph and PetaSAN docs state.
https://www.spinics.net/lists/ceph-devel/msg40592.html
We will change it