Forums

Home / Forums

You need to log in to create posts and topics. Login · Register

mon out of quorum permission denied


ghbiz
76 Posts
September 24, 2024, 11:41 pm

This Cluster is running Version 2.8.1

At this point, i am trying to redeploy the mon service but i am getting following issue. is there a specific way to run create_mon.py ....

oot@ceph-public1:~# /opt/petasan/scripts/create_mon.py
/opt/petasan/scripts/create_mon.py: line 14:
Copyright (C) 2020 Maged Mokhtar <mmokhtar <at> petasan.org>
Copyright (C) 2020 PetaSAN http://www.petasan.org

This program is free software; you can redistribute it and/or
modify it under the terms of the GNU Affero General Public License
as published by the Free Software Foundation

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
: No such file or directory
/opt/petasan/scripts/create_mon.py: line 16: import: command not found
/opt/petasan/scripts/create_mon.py: line 17: import: command not found
/opt/petasan/scripts/create_mon.py: line 18: from: command not found
/opt/petasan/scripts/create_mon.py: line 19: from: command not found
/opt/petasan/scripts/create_mon.py: line 20: from: command not found
/opt/petasan/scripts/create_mon.py: line 21: from: command not found
/opt/petasan/scripts/create_mon.py: line 24: syntax error near unexpected token `('
/opt/petasan/scripts/create_mon.py: line 24: `cluster_name = configuration().get_cluster_name()'
root@ceph-public1:~#

 

 

status shows == mon: 3 daemons, quorum ceph-public2,ceph-public3 (age 3d), out of quorum: ceph-public1

Rebooted / restarted mon service and no change.

## tried running monitor service manually with following command.
/usr/bin/ceph-mon -d --cluster ceph --id ceph-public1 --setuser ceph --setgroup ceph --keyring /var/lib/ceph/mon/ceph-ceph-public1/keyring

### standard output shows ###

Standby daemons:

[mds.ceph-public1{-1:1013691799} state up:standby seq 2 addr [v2:10.255.254.11:6800/954759959,v1:10.255.254.11:6801/954759959]]
[mds.ceph-public2{-1:1035398777} state up:standby seq 1 addr [v2:10.255.254.12:6800/1402808735,v1:10.255.254.12:6801/1402808735]]

2024-09-24T19:35:34.611-0400 7fd1b0885540 0 mon.ceph-public1@-1(???).osd e7985904 crush map has features 432629308056666112, adjusting msgr requires
2024-09-24T19:35:34.611-0400 7fd1b0885540 0 mon.ceph-public1@-1(???).osd e7985904 crush map has features 432629308056666112, adjusting msgr requires
2024-09-24T19:35:34.611-0400 7fd1b0885540 0 mon.ceph-public1@-1(???).osd e7985904 crush map has features 3314933069573799936, adjusting msgr requires
2024-09-24T19:35:34.611-0400 7fd1b0885540 0 mon.ceph-public1@-1(???).osd e7985904 crush map has features 432629308056666112, adjusting msgr requires
2024-09-24T19:35:34.611-0400 7fd1b0885540 1 mon.ceph-public1@-1(???).paxosservice(auth 150002..150036) refresh upgraded, format 0 -> 3
2024-09-24T19:35:34.615-0400 7fd1b0885540 10 mon.ceph-public1@-1(???).mgrstat 75910772
2024-09-24T19:35:34.615-0400 7fd1b0885540 10 mon.ceph-public1@-1(???).mgrstat update_from_paxos v75910772 service_map e1469367 0 progress events
2024-09-24T19:35:34.615-0400 7fd1b0885540 10 mon.ceph-public1@-1(???).mgrstat check_subs
2024-09-24T19:35:34.615-0400 7fd1b0885540 10 mon.ceph-public1@-1(???).health update_from_paxos
2024-09-24T19:35:34.615-0400 7fd1b0885540 10 mon.ceph-public1@-1(???).config update_from_paxos 406
2024-09-24T19:35:34.615-0400 7fd1b0885540 10 mon.ceph-public1@-1(???).config load_config got 68 keys
2024-09-24T19:35:34.615-0400 7fd1b0885540 10 mon.ceph-public1@-1(???).config check_all_subs
2024-09-24T19:35:34.615-0400 7fd1b0885540 -1 compacting monitor store ...
2024-09-24T19:35:34.615-0400 7fd1b0885540 4 rocksdb: [db/db_impl_compaction_flush.cc:1403] [default] Manual compaction starting
2024-09-24T19:35:34.615-0400 7fd1b0885540 4 rocksdb: [db/db_impl_compaction_flush.cc:1403] [default] Manual compaction starting
2024-09-24T19:35:34.615-0400 7fd1b0885540 4 rocksdb: [db/db_impl_compaction_flush.cc:1403] [default] Manual compaction starting
2024-09-24T19:35:34.615-0400 7fd1b0885540 4 rocksdb: [db/db_impl_compaction_flush.cc:1403] [default] Manual compaction starting
2024-09-24T19:35:34.615-0400 7fd1b0885540 4 rocksdb: [db/db_impl_compaction_flush.cc:1403] [default] Manual compaction starting
2024-09-24T19:35:34.615-0400 7fd1b0885540 4 rocksdb: [db/db_impl_compaction_flush.cc:1403] [default] Manual compaction starting
2024-09-24T19:35:34.615-0400 7fd1b0885540 -1 done compacting
2024-09-24T19:35:34.615-0400 7fd1b0885540 2 mon.ceph-public1@-1(???) e10 init
2024-09-24T19:35:34.615-0400 7fd1b0885540 10 mon.ceph-public1@-1(???).mgr e326 prime_mgr_client
2024-09-24T19:35:34.619-0400 7fd19ea65700 10 mon.ceph-public1@-1(???) e10 get_authorizer for mgr
2024-09-24T19:35:34.619-0400 7fd19ea65700 -1 mon.ceph-public1@-1(???) e10 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
2024-09-24T19:35:34.619-0400 7fd1b0885540 10 mon.ceph-public1@-1(probing) e10 bootstrap
2024-09-24T19:35:34.619-0400 7fd1b0885540 10 mon.ceph-public1@-1(probing) e10 sync_reset_requester
2024-09-24T19:35:34.619-0400 7fd1b0885540 10 mon.ceph-public1@-1(probing) e10 unregister_cluster_logger - not registered
2024-09-24T19:35:34.619-0400 7fd1b0885540 10 mon.ceph-public1@-1(probing) e10 cancel_probe_timeout (none scheduled)
2024-09-24T19:35:34.619-0400 7fd1b0885540 10 mon.ceph-public1@-1(probing) e10 monmap e10: 3 mons at {ceph-public1=[v2:10.255.254.11:3300/0,v1:10.255.254.11:6789/0],ceph-public2=[v2:10.255.254.12:3300/0,v1:10.255.254.12:6789/0],ceph-public3=[v2:10.255.254.13:3300/0,v1:10.255.254.13:6789/0]}
2024-09-24T19:35:34.619-0400 7fd1b0885540 0 mon.ceph-public1@-1(probing) e10 my rank is now 2 (was -1)
2024-09-24T19:35:34.619-0400 7fd1b0885540 10 mon.ceph-public1@2(probing) e10 _reset
2024-09-24T19:35:34.619-0400 7fd1b0885540 10 mon.ceph-public1@2(probing).auth v150036 _set_mon_num_rank num 0 rank 0
2024-09-24T19:35:34.619-0400 7fd1b0885540 10 mon.ceph-public1@2(probing) e10 cancel_probe_timeout (none scheduled)
2024-09-24T19:35:34.619-0400 7fd1b0885540 10 mon.ceph-public1@2(probing) e10 timecheck_finish
2024-09-24T19:35:34.619-0400 7fd1b0885540 10 mon.ceph-public1@2(probing) e10 scrub_event_cancel
2024-09-24T19:35:34.619-0400 7fd1b0885540 10 mon.ceph-public1@2(probing) e10 scrub_reset
2024-09-24T19:35:34.619-0400 7fd1b0885540 10 mon.ceph-public1@2(probing) e10 cancel_probe_timeout (none scheduled)
2024-09-24T19:35:34.619-0400 7fd1b0885540 10 mon.ceph-public1@2(probing) e10 reset_probe_timeout 0x55a52fcd5fa0 after 2 seconds
2024-09-24T19:35:34.619-0400 7fd1b0885540 10 mon.ceph-public1@2(probing) e10 probing other monitors
2024-09-24T19:35:34.619-0400 7fd19f266700 10 mon.ceph-public1@2(probing) e10 get_authorizer for mon
2024-09-24T19:35:34.619-0400 7fd199a5b700 10 mon.ceph-public1@2(probing) e10 get_authorizer for mon
2024-09-24T19:35:34.631-0400 7fd19c260700 10 osdmap epoch 7985904 mapping took 0.021209 seconds
2024-09-24T19:35:34.631-0400 7fd19c260700 10 mon.ceph-public1@2(probing).osd e7985904 update_creating_pgs 0 pgs creating, 0 pools in queue
2024-09-24T19:35:34.823-0400 7fd19ea65700 10 mon.ceph-public1@2(probing) e10 get_authorizer for mgr
2024-09-24T19:35:34.823-0400 7fd199a5b700 10 mon.ceph-public1@2(probing) e10 get_authorizer for mon
2024-09-24T19:35:34.823-0400 7fd19f266700 10 mon.ceph-public1@2(probing) e10 get_authorizer for mon
2024-09-24T19:35:34.823-0400 7fd19ea65700 -1 mon.ceph-public1@2(probing) e10 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
2024-09-24T19:35:35.227-0400 7fd19f266700 10 mon.ceph-public1@2(probing) e10 get_authorizer for mon
2024-09-24T19:35:35.227-0400 7fd199a5b700 10 mon.ceph-public1@2(probing) e10 get_authorizer for mon
2024-09-24T19:35:35.227-0400 7fd19ea65700 10 mon.ceph-public1@2(probing) e10 get_authorizer for mgr
2024-09-24T19:35:35.227-0400 7fd19ea65700 -1 mon.ceph-public1@2(probing) e10 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied
^C2024-09-24T19:35:35.743-0400 7fd1a126a700 -1 received signal: Interrupt, si_code : 128, si_value (int): 801628160, si_value (ptr): 0x55a52fc7e000, si_errno: 0, si_pid : 0, si_uid : 0, si_addr0, si_status801628160
2024-09-24T19:35:35.743-0400 7fd1a126a700 -1 mon.ceph-public1@2(probing) e10 *** Got Signal Interrupt ***
2024-09-24T19:35:35.743-0400 7fd1a126a700 1 mon.ceph-public1@2(probing) e10 shutdown
2024-09-24T19:35:35.743-0400 7fd1a126a700 10 mon.ceph-public1@2(probing).auth v150036 _set_mon_num_rank num 0 rank 0
2024-09-24T19:35:35.743-0400 7fd1a126a700 10 mon.ceph-public1@2(shutdown).osd e7985904 on_shutdown
2024-09-24T19:35:35.743-0400 7fd1a126a700 10 mon.ceph-public1@2(shutdown).osd e7985904 on_shutdown canceling previous mapping_job 0x55a52ebc2800
2024-09-24T19:35:35.743-0400 7fd1a126a700 10 mon.ceph-public1@2(shutdown).osd e7985904 take_all_failures on 0 osds
2024-09-24T19:35:35.743-0400 7fd19aa5d700 10 mon.ceph-public1@2(shutdown) e10 ms_handle_reset 0x55a52ec1e800 v2:10.255.254.13:3300/0
2024-09-24T19:35:35.743-0400 7fd19aa5d700 10 mon.ceph-public1@2(shutdown) e10 ms_handle_reset 0x55a52ec1ec00 v2:10.255.254.12:3300/0
2024-09-24T19:35:35.743-0400 7fd1b0885540 4 rocksdb: [db/db_impl.cc:390] Shutdown: canceling all background work
2024-09-24T19:35:35.747-0400 7fd1b0885540 4 rocksdb: [db/db_impl.cc:563] Shutdown complete

 

 

#1

ghbiz
76 Posts
September 25, 2024, 6:00 pm

As an Update....

Cluster is NOW HEALTH_OK....

ran the following two commands....

ceph mon remove ceph-public1

ceph mon add ceph-public1 10.255.254.11

 

#2