Forums

Home / Forums

You need to log in to create posts and topics. Login · Register

continuously monitor leader election


f.cuseo
66 Posts
February 8, 2024, 8:46 am

Good morning.

I still have some problems with monitor that calls monitor election; someone can help me ? Cluster is a 10 nodes with 12x8Tbyte (linear sas), 4x10Gbit bond (with separate vlans), 2 cpu, 64 giga, with NFS (for backup use) and S3 gateway.

root@petasan-01:~# grep "is new leader" /var/log/ceph/ceph.log

2024-02-08T07:48:01.631122+0100 mon.petasan-03 (mon.0) 31180 : cluster [INF] mon.petasan-03 is new leader, mons petasan-03,petasan-01,petasan-02 in quorum (ranks 0,1,2)
2024-02-08T07:48:42.294234+0100 mon.petasan-01 (mon.1) 13568 : cluster [INF] mon.petasan-01 is new leader, mons petasan-01,petasan-02 in quorum (ranks 1,2)
2024-02-08T07:48:49.041100+0100 mon.petasan-03 (mon.0) 31219 : cluster [INF] mon.petasan-03 is new leader, mons petasan-03,petasan-01,petasan-02 in quorum (ranks 0,1,2)
2024-02-08T07:49:35.092889+0100 mon.petasan-03 (mon.0) 31268 : cluster [INF] mon.petasan-03 is new leader, mons petasan-03,petasan-01,petasan-02 in quorum (ranks 0,1,2)
2024-02-08T07:54:33.936636+0100 mon.petasan-03 (mon.0) 31566 : cluster [INF] mon.petasan-03 is new leader, mons petasan-03,petasan-01,petasan-02 in quorum (ranks 0,1,2)
2024-02-08T07:56:40.975585+0100 mon.petasan-03 (mon.0) 31674 : cluster [INF] mon.petasan-03 is new leader, mons petasan-03,petasan-01 in quorum (ranks 0,1)
2024-02-08T07:56:49.167292+0100 mon.petasan-03 (mon.0) 31694 : cluster [INF] mon.petasan-03 is new leader, mons petasan-03,petasan-01,petasan-02 in quorum (ranks 0,1,2)
2024-02-08T07:57:01.070843+0100 mon.petasan-03 (mon.0) 31712 : cluster [INF] mon.petasan-03 is new leader, mons petasan-03,petasan-01,petasan-02 in quorum (ranks 0,1,2)
2024-02-08T07:57:52.419200+0100 mon.petasan-03 (mon.0) 31763 : cluster [INF] mon.petasan-03 is new leader, mons petasan-03,petasan-01,petasan-02 in quorum (ranks 0,1,2)
2024-02-08T07:58:45.521576+0100 mon.petasan-03 (mon.0) 31805 : cluster [INF] mon.petasan-03 is new leader, mons petasan-03,petasan-01,petasan-02 in quorum (ranks 0,1,2)
2024-02-08T08:14:34.649614+0100 mon.petasan-03 (mon.0) 32731 : cluster [INF] mon.petasan-03 is new leader, mons petasan-03,petasan-01,petasan-02 in quorum (ranks 0,1,2)
2024-02-08T08:15:43.315153+0100 mon.petasan-03 (mon.0) 32791 : cluster [INF] mon.petasan-03 is new leader, mons petasan-03,petasan-01,petasan-02 in quorum (ranks 0,1,2)
2024-02-08T08:17:39.069558+0100 mon.petasan-03 (mon.0) 32891 : cluster [INF] mon.petasan-03 is new leader, mons petasan-03,petasan-01,petasan-02 in quorum (ranks 0,1,2)
2024-02-08T08:19:00.664183+0100 mon.petasan-03 (mon.0) 32964 : cluster [INF] mon.petasan-03 is new leader, mons petasan-03,petasan-01,petasan-02 in quorum (ranks 0,1,2)
2024-02-08T08:19:46.151951+0100 mon.petasan-03 (mon.0) 33008 : cluster [INF] mon.petasan-03 is new leader, mons petasan-03,petasan-01,petasan-02 in quorum (ranks 0,1,2)
2024-02-08T08:24:40.820055+0100 mon.petasan-03 (mon.0) 33300 : cluster [INF] mon.petasan-03 is new leader, mons petasan-03,petasan-01,petasan-02 in quorum (ranks 0,1,2)
2024-02-08T08:25:32.550846+0100 mon.petasan-03 (mon.0) 33353 : cluster [INF] mon.petasan-03 is new leader, mons petasan-03,petasan-01,petasan-02 in quorum (ranks 0,1,2)
2024-02-08T08:25:48.720672+0100 mon.petasan-03 (mon.0) 33366 : cluster [INF] mon.petasan-03 is new leader, mons petasan-03,petasan-01,petasan-02 in quorum (ranks 0,1,2)
2024-02-08T08:26:34.767811+0100 mon.petasan-03 (mon.0) 33415 : cluster [INF] mon.petasan-03 is new leader, mons petasan-03,petasan-01,petasan-02 in quorum (ranks 0,1,2)
2024-02-08T08:26:59.452638+0100 mon.petasan-01 (mon.1) 13645 : cluster [INF] mon.petasan-01 is new leader, mons petasan-01,petasan-02 in quorum (ranks 1,2)
2024-02-08T08:27:01.733524+0100 mon.petasan-03 (mon.0) 33437 : cluster [INF] mon.petasan-03 is new leader, mons petasan-03,petasan-01,petasan-02 in quorum (ranks 0,1,2)
2024-02-08T08:27:45.558127+0100 mon.petasan-03 (mon.0) 33478 : cluster [INF] mon.petasan-03 is new leader, mons petasan-03,petasan-01,petasan-02 in quorum (ranks 0,1,2)
2024-02-08T08:28:37.529833+0100 mon.petasan-03 (mon.0) 33530 : cluster [INF] mon.petasan-03 is new leader, mons petasan-03,petasan-01,petasan-02 in quorum (ranks 0,1,2)
2024-02-08T08:29:47.624525+0100 mon.petasan-03 (mon.0) 33596 : cluster [INF] mon.petasan-03 is new leader, mons petasan-03,petasan-01,petasan-02 in quorum (ranks 0,1,2)
2024-02-08T08:33:34.280160+0100 mon.petasan-03 (mon.0) 33830 : cluster [INF] mon.petasan-03 is new leader, mons petasan-03,petasan-01,petasan-02 in quorum (ranks 0,1,2)
2024-02-08T08:44:48.920604+0100 mon.petasan-03 (mon.0) 34500 : cluster [INF] mon.petasan-03 is new leader, mons petasan-03,petasan-01,petasan-02 in quorum (ranks 0,1,2)
2024-02-08T08:45:35.954200+0100 mon.petasan-03 (mon.0) 34546 : cluster [INF] mon.petasan-03 is new leader, mons petasan-03,petasan-01,petasan-02 in quorum (ranks 0,1,2)
2024-02-08T08:45:52.499676+0100 mon.petasan-03 (mon.0) 34564 : cluster [INF] mon.petasan-03 is new leader, mons petasan-03,petasan-01,petasan-02 in quorum (ranks 0,1,2)
2024-02-08T08:46:46.103291+0100 mon.petasan-03 (mon.0) 34613 : cluster [INF] mon.petasan-03 is new leader, mons petasan-03,petasan-01,petasan-02 in quorum (ranks 0,1,2)
2024-02-08T08:47:41.486807+0100 mon.petasan-03 (mon.0) 34669 : cluster [INF] mon.petasan-03 is new leader, mons petasan-03,petasan-01,petasan-02 in quorum (ranks 0,1,2)
2024-02-08T08:48:57.485952+0100 mon.petasan-03 (mon.0) 34739 : cluster [INF] mon.petasan-03 is new leader, mons petasan-03,petasan-01,petasan-02 in quorum (ranks 0,1,2)
2024-02-08T08:49:39.649533+0100 mon.petasan-03 (mon.0) 34781 : cluster [INF] mon.petasan-03 is new leader, mons petasan-03,petasan-01,petasan-02 in quorum (ranks 0,1,2)
2024-02-08T08:54:32.568480+0100 mon.petasan-03 (mon.0) 35075 : cluster [INF] mon.petasan-03 is new leader, mons petasan-03,petasan-01,petasan-02 in quorum (ranks 0,1,2)
2024-02-08T08:55:33.667944+0100 mon.petasan-03 (mon.0) 35130 : cluster [INF] mon.petasan-03 is new leader, mons petasan-03,petasan-01,petasan-02 in quorum (ranks 0,1,2)
2024-02-08T08:56:37.813987+0100 mon.petasan-03 (mon.0) 35193 : cluster [INF] mon.petasan-03 is new leader, mons petasan-03,petasan-01,petasan-02 in quorum (ranks 0,1,2)
2024-02-08T08:57:35.275792+0100 mon.petasan-03 (mon.0) 35247 : cluster [INF] mon.petasan-03 is new leader, mons petasan-03,petasan-01,petasan-02 in quorum (ranks 0,1,2)
2024-02-08T08:58:45.200363+0100 mon.petasan-03 (mon.0) 35324 : cluster [INF] mon.petasan-03 is new leader, mons petasan-03,petasan-01,petasan-02 in quorum (ranks 0,1,2)
2024-02-08T08:59:39.996455+0100 mon.petasan-03 (mon.0) 35379 : cluster [INF] mon.petasan-03 is new leader, mons petasan-03,petasan-01,petasan-02 in quorum (ranks 0,1,2)
2024-02-08T09:17:38.951598+0100 mon.petasan-03 (mon.0) 36441 : cluster [INF] mon.petasan-03 is new leader, mons petasan-03,petasan-01,petasan-02 in quorum (ranks 0,1,2)
2024-02-08T09:19:35.170115+0100 mon.petasan-03 (mon.0) 36557 : cluster [INF] mon.petasan-03 is new leader, mons petasan-03,petasan-01,petasan-02 in quorum (ranks 0,1,2)
2024-02-08T09:25:36.335062+0100 mon.petasan-03 (mon.0) 36912 : cluster [INF] mon.petasan-03 is new leader, mons petasan-03,petasan-01,petasan-02 in quorum (ranks 0,1,2)
2024-02-08T09:26:36.574228+0100 mon.petasan-03 (mon.0) 36973 : cluster [INF] mon.petasan-03 is new leader, mons petasan-03,petasan-01,petasan-02 in quorum (ranks 0,1,2)
2024-02-08T09:39:53.554203+0100 mon.petasan-03 (mon.0) 37769 : cluster [INF] mon.petasan-03 is new leader, mons petasan-03,petasan-01,petasan-02 in quorum (ranks 0,1,2)

Log from: ceph-mon.petasan-01.log

2024-02-08T09:39:48.275+0100 7f5d8a7bb700 1 mon.petasan-01@1(peon).paxos(paxos updating c 9259644..9260300) lease_timeout -- calling new election
2024-02-08T09:39:48.275+0100 7f5d87fb6700 0 log_channel(cluster) log [INF] : mon.petasan-01 calling monitor election
2024-02-08T09:39:48.275+0100 7f5d87fb6700 1 paxos.1).electionLogic(407124) init, last seen epoch 407124
2024-02-08T09:39:52.691+0100 7f5d85fb2700 1 mon.petasan-01@1(electing) e4 handle_auth_request failed to assign global_id
2024-02-08T09:39:53.315+0100 7f5d8a7bb700 1 paxos.1).electionLogic(407125) init, last seen epoch 407125, mid-election, bumping
2024-02-08T09:39:53.403+0100 7f5d87fb6700 0 log_channel(cluster) log [INF] : mon.petasan-01 calling monitor election
2024-02-08T09:39:53.403+0100 7f5d87fb6700 1 paxos.1).electionLogic(407129) init, last seen epoch 407129, mid-election, bumping

ceph -w:

024-02-08T09:39:48.278039+0100 mon.petasan-01 [INF] mon.petasan-01 calling monitor election
2024-02-08T09:39:53.366584+0100 mon.petasan-02 [INF] mon.petasan-02 calling monitor election
2024-02-08T09:39:53.367688+0100 mon.petasan-03 [INF] mon.petasan-03 calling monitor election
2024-02-08T09:39:53.408924+0100 mon.petasan-01 [INF] mon.petasan-01 calling monitor election
2024-02-08T09:39:53.463841+0100 mon.petasan-03 [INF] mon.petasan-03 calling monitor election
2024-02-08T09:39:53.554203+0100 mon.petasan-03 [INF] mon.petasan-03 is new leader, mons petasan-03,petasan-01,petasan-02 in quorum (ranks 0,1,2)
2024-02-08T09:39:53.599956+0100 mon.petasan-03 [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum petasan-03,petasan-02)
2024-02-08T09:39:53.599971+0100 mon.petasan-03 [INF] Cluster is now healthy
2024-02-08T09:39:53.677515+0100 mon.petasan-03 [INF] overall HEALTH_OK
2024-02-08T09:40:00.000075+0100 mon.petasan-03 [INF] overall HEALTH_OK

#1

admin
2,930 Posts
February 12, 2024, 10:02 pm

If you have a monitor service giving you issues, you can re-install it from command line.

#2

f.cuseo
66 Posts
February 14, 2024, 9:11 am
Quote from admin on February 12, 2024, 10:02 pm

If you have a monitor service giving you issues, you can re-install it from command line.

Do you think that is better that i wait for the new release in the next days ?

If not, which software I need to reinstall ? (which is the correct procedure ?)

 

#3