Forums

Home / Forums

You need to log in to create posts and topics. Login · Register

Empty charts for Node Statistics after reboot


zidex
6 Posts
February 13, 2022, 12:48 pm

I have a test cluster of 3 nodes. After the reboot of the first node (petasan1), the charts (Node Statisics) for the second and third disappeared. Charts for Cluster Statistics remained. 

Log from petasan1:

13/02/2022 14:58:02 INFO update_node_info completed
13/02/2022 14:58:02 INFO node info has been updated successfully.
13/02/2022 14:58:01 INFO update_node_info starting
13/02/2022 14:58:01 INFO Starting petasan-update-node-info
13/02/2022 14:58:01 INFO Starting petasan-qperf server
13/02/2022 14:58:01 INFO Starting Config Upload service
13/02/2022 14:58:01 INFO Starting petasan tuning service
13/02/2022 14:58:01 INFO Starting OSDs
13/02/2022 14:57:59 INFO Starting activating PetaSAN lvs
13/02/2022 14:57:32 INFO Activating ps-873f4c7c-08f7-4458-8687-f8070ec5e7f6-wc-osd.3
13/02/2022 14:57:24 INFO Cleaning unused ips.
13/02/2022 14:57:24 INFO Cleaning unused rbd images.
13/02/2022 14:57:24 INFO Cleaning all mapped disks
13/02/2022 14:57:24 INFO Cleaning unused configurations.
13/02/2022 14:57:24 INFO iSCSI Service successfully dropped old sessions
13/02/2022 14:57:18 INFO LeaderElectionBase successfully dropped old sessions
13/02/2022 14:57:09 INFO iSCSI Service dropping old sessions
13/02/2022 14:57:09 INFO Service is starting.
13/02/2022 14:57:08 INFO checking uptime of node: petasan3 ip:10.10.31.243
13/02/2022 14:57:08 INFO checking uptime of node: petasan2 ip:10.10.31.242
13/02/2022 14:57:08 INFO checking uptime of node: petasan1 is starting
13/02/2022 14:57:08 INFO checking uptime of node: petasan1 ip:10.10.31.241
13/02/2022 14:57:08 INFO checking uptime
13/02/2022 14:57:08 INFO Activating ps-873f4c7c-08f7-4458-8687-f8070ec5e7f6-wc-osd.0
13/02/2022 14:57:08 INFO LeaderElectionBase dropping old sessions
13/02/2022 14:57:08 INFO Starting activating PetaSAN lvs
13/02/2022 14:57:08 INFO Starting Node Stats Service
13/02/2022 14:57:08 INFO Starting Cluster Management application
13/02/2022 14:57:08 INFO Starting iSCSI Service
13/02/2022 14:57:07 INFO Starting cluster file sync service
13/02/2022 14:57:05 INFO str_start_command: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consul agent -raft-protocol 2 -config-dir /opt/petasan/config/etc/consul.d/server -bind 10.10.20.1 -retry-join 10.10.20.2 -retry-join 10.10.20.3
13/02/2022 14:57:04 INFO GlusterFS mount attempt
13/02/2022 14:57:04 INFO Start settings IPs
13/02/2022 14:55:49 INFO LeaderElectionBase successfully dropped old sessions
13/02/2022 14:55:39 INFO iSCSI Service dropping old sessions
13/02/2022 14:55:39 INFO Service is starting.
13/02/2022 14:55:39 INFO checking uptime of node: petasan3 ip:10.10.31.243
13/02/2022 14:55:39 INFO checking uptime of node: petasan2 ip:10.10.31.242
13/02/2022 14:55:39 INFO checking uptime of node: petasan1 is starting
13/02/2022 14:55:39 INFO checking uptime of node: petasan1 ip:10.10.31.241
13/02/2022 14:55:39 INFO checking uptime
13/02/2022 14:55:38 INFO LeaderElectionBase dropping old sessions
13/02/2022 14:55:38 INFO Activating ps-873f4c7c-08f7-4458-8687-f8070ec5e7f6-wc-osd.0
13/02/2022 14:55:38 INFO Starting activating PetaSAN lvs
13/02/2022 14:55:38 INFO Starting Node Stats Service
13/02/2022 14:55:38 INFO Starting Cluster Management application
13/02/2022 14:55:38 INFO Starting iSCSI Service
13/02/2022 14:55:38 INFO Starting cluster file sync service
13/02/2022 14:55:35 INFO str_start_command: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consul agent -raft-protocol 2 -config-dir /opt/petasan/config/etc/consul.d/server -bind 10.10.20.1 -retry-join 10.10.20.2 -retry-join 10.10.20.3
13/02/2022 14:55:35 INFO GlusterFS mount attempt
13/02/2022 14:55:34 INFO Start settings IPs

Log from petasan2:

13/02/2022 14:56:09 INFO Path 00001/2 acquired successfully
13/02/2022 14:56:09 INFO LIO add_target() disk wwn is 873f4c7c00001
13/02/2022 14:56:05 INFO Image image-00001 mapped successfully.
13/02/2022 14:55:59 INFO This node will stop node petasan1/10.10.31.241.
13/02/2022 14:55:59 INFO The path 00001/2 was locked by petasan1.
13/02/2022 14:55:59 INFO Found pool:rbd1 for disk:00001 via consul
13/02/2022 14:55:52 INFO Could not lock path 00001/1 with session 4d96cc0a-c855-8755-d503-40c3e0c6b355.
13/02/2022 14:55:52 INFO Found pool:rbd1 for disk:00001 via consul
13/02/2022 14:55:45 INFO Could not lock path 00001/1 with session 4d96cc0a-c855-8755-d503-40c3e0c6b355.
13/02/2022 14:55:45 INFO Found pool:rbd1 for disk:00001 via consul

Log from petasan3:

13/02/2022 14:56:12 INFO Could not lock path 00001/2 with session 21527e7c-8887-d30a-3a19-1c84f92a41a0.
13/02/2022 14:56:11 INFO Found pool:rbd1 for disk:00001 via consul
13/02/2022 14:55:57 INFO ClusterLeader starting services
13/02/2022 14:55:57 INFO ClusterLeader start action
13/02/2022 14:55:52 INFO Path 00001/1 acquired successfully
13/02/2022 14:55:52 INFO This node will stop node petasan1/10.10.31.241.
13/02/2022 14:55:52 INFO The path 00001/1 was locked by petasan1.
13/02/2022 14:55:52 INFO Found pool:rbd1 for disk:00001 via consul

 

 

#1

admin
2,918 Posts
February 14, 2022, 7:52 am

You mean the cluster stats are working but not the node stats ?

Can show the output of following command on all nodes

/opt/petasan/scripts/util/get_cluster_leader.py

#2

zidex
6 Posts
February 14, 2022, 1:27 pm

Yes. Cluster stats work, but host stats empty for some nodes. After restarting the node, statistics for it appear, but it may disappear for another / others.

/opt/petasan/scripts/util/get_cluster_leader.py from nodes:

petasan1, stats work:

{'petasan3': '10.10.31.243'}

petasan2, stats work:

{'petasan3': '10.10.31.243'}

petasan3, stats empty:

{'petasan3': '10.10.31.243'}

Last edited on February 14, 2022, 1:28 pm by zidex · #3

admin
2,918 Posts
February 14, 2022, 1:50 pm

so i understand only petasan3 is not showing stats. on that node,  try :

systemctl restart petasan-node-stats

also do you see any error in its PetaSAN.log file ?

 

Last edited on February 14, 2022, 1:50 pm by admin · #4

zidex
6 Posts
February 14, 2022, 2:05 pm

root@petasan3:~# systemctl status petasan-node-stats
● petasan-node-stats.service - PetaSAN Node Stats Service
Loaded: loaded (/lib/systemd/system/petasan-node-stats.service; static; vendor preset: enabled)
Active: active (running) since Sun 2022-02-13 23:58:58 MSK; 17h ago
Main PID: 1351 (node_stats.py)
Tasks: 1 (limit: 19171)
Memory: 32.2M
CGroup: /system.slice/petasan-node-stats.service
└─1351 /usr/bin/python3 /opt/petasan/scripts/node_stats.py

Feb 13 23:58:58 petasan3 systemd[1]: Started PetaSAN Node Stats Service.

 

There are no errors in the logs. INFO only.

 

Last edited on February 14, 2022, 2:06 pm by zidex · #5

admin
2,918 Posts
February 14, 2022, 3:53 pm

systemctl restart petasan-node-stats

#6

zidex
6 Posts
February 14, 2022, 7:22 pm

Thank you. After that, the data appeared on the charts.

#7