Forums

Home / Forums

You need to log in to create posts and topics. Login · Register

[SOLVED] How to import OSD's

Pages: 1 2

tntbizhub
14 Posts
June 14, 2024, 3:04 pm

Hi all,

Thanks in advance for the help.

I had a cluster with 3 machines running PetaSAN 3.2.1, I have reinstalled them with 3.3.0 version, didn't marked the OSD's as per the instructions during setup of the nodes, there was a mention saying that disks migrated from another install would be "detected" but the fact was that they weren't and I have the cluster empty of OSD's.

Is there a procedure for this? For re-adding the disks?

Best regards from Portugal,
Tiago Matias

Last edited on July 15, 2024, 11:03 am by tntbizhub · #1

admin
2,930 Posts
June 14, 2024, 3:47 pm

What is supported: if you upgrade the cluster to new version or if you replace a failed node within the cluster.

What is not supported by us: if you re-install a new cluster: new monitors, new uuids, keys and use OSD drives from a previous cluster and expect it to work in new cluster. It can be done with manual cli commands but it is not straightforward, you will have to extract from existing cluster maps from the OSDs and inject them into the new monitors. You need to make sure you use the most recent  OSD maps in case there is any discrepancy among the maps in different OSDs. Search online and you should see examples.

Last edited on June 14, 2024, 3:51 pm by admin · #2

tntbizhub
14 Posts
June 15, 2024, 1:51 pm

Hi!

I have searched a lot on this subject, didn't find some procedure that helps, only a bit here and a bit there.

I am testing with node-3 but a bit affraid to loose the data, I have a doubt, as long as I don't mess around with the other disks/osd's from the other 2 nodes is still possible to recover access to the data?

I have done a ceph-volume lvm activate --all and now I have when I run ceph-volume lvm list:

====== osd.1 =======

[block] /dev/ceph-106a710d-94ad-411c-998a-c7e922b657a2/osd-block-f3725738-3e97-4cf0-bbdd-aa331ade68b4

block device /dev/ceph-106a710d-94ad-411c-998a-c7e922b657a2/osd-block-f3725738-3e97-4cf0-bbdd-aa331ade68b4
block uuid sfAmoI-6Q4A-Jn7s-y12q-p8tM-Y8dY-iTsiBt
cephx lockbox secret
cluster fsid 81acea00-fc7b-4131-a038-43ef1aa2efcc
cluster name ceph
crush device class
encrypted 0
osd fsid f3725738-3e97-4cf0-bbdd-aa331ade68b4
osd id 1
osdspec affinity
type block
vdo 0
devices /dev/sdb1

====== osd.2 =======

[block] /dev/ceph-a18e227e-b96b-4c1d-8451-8e4dc9c8474c/osd-block-c1afbb22-9d9f-45fb-a35d-2fbe05b538d8

block device /dev/ceph-a18e227e-b96b-4c1d-8451-8e4dc9c8474c/osd-block-c1afbb22-9d9f-45fb-a35d-2fbe05b538d8
block uuid bOIOAn-odo7-gHOX-MYU4-9uTJ-Tgjt-wWKcR2
cephx lockbox secret
cluster fsid 81acea00-fc7b-4131-a038-43ef1aa2efcc
cluster name ceph
crush device class
encrypted 0
osd fsid c1afbb22-9d9f-45fb-a35d-2fbe05b538d8
osd id 2
osdspec affinity
type block
vdo 0
devices /dev/sdc1

====== osd.4 =======

[block] /dev/ceph-bc854676-5560-497f-8d57-1058a7b29629/osd-block-d827adbd-e6d1-4561-bc58-1776674ecb82

block device /dev/ceph-bc854676-5560-497f-8d57-1058a7b29629/osd-block-d827adbd-e6d1-4561-bc58-1776674ecb82
block uuid MUuRZf-UDzK-3VgT-d3MP-4Cne-raE6-gmegMc
cephx lockbox secret
cluster fsid 81acea00-fc7b-4131-a038-43ef1aa2efcc
cluster name ceph
crush device class
encrypted 0
osd fsid d827adbd-e6d1-4561-bc58-1776674ecb82
osd id 4
osdspec affinity
type block
vdo 0
devices /dev/sdd1

I cannot start OSD's with sudo systemctl start ceph-osd@1 I get an error... this is the status running systemctl list-units --failed

UNIT LOAD ACTIVE SUB DESCRIPTION
● ceph-osd@1.service loaded failed failed Ceph object storage daemon osd.1
● ceph-osd@2.service loaded failed failed Ceph object storage daemon osd.2
● ceph-osd@4.service loaded failed failed Ceph object storage daemon osd.4
● logrotate.service loaded failed failed Rotate log files

I should be able to start the services for the OSD's right? Or the cluster/OSD key are different as they were from previous cluster and I need to make them match the ones on the OSD's?

Trying to use documentation from https://docs.ceph.com/en/mimic/rados/operations/add-or-rm-osds/ but from what i understood much of the documentation is to add new OSD's not for add OSD's with data, if you could point me in the right direction, what I need to solve first I would appreciate.

Thanks again,
Tiago Matias

 

 

#3

admin
2,930 Posts
June 15, 2024, 4:35 pm

As per last post, you need to populate the monitors with cluster data you read from the OSDs. In normal case the monitor stores authoritative cluster data: pools/crush/pgs/osds/pg mapping osds/osd authorization keys...monitors inform OSD of state. In current case, you have fresh monitors with empty data. You need to do the reverse, use the state data in your current OSDs and force this/inject them into the monitors.
Some links:
https://docs.ceph.com/en/reef/rados/troubleshooting/troubleshooting-mon/#mon-store-recovery-using-osds
if you were using CephFS
https://docs.ceph.com/en/reef/cephfs/recover-fs-after-mon-store-loss/

You will also need to update /etc/ceph/ceph.conf with your old cluster fsid 81acea00-fc7b-4131-a038-43ef1aa2efcc

Some risks are involved, if the stored cluster data on the OSDs was inconsistent or the cluster state was unhealthily or unknown at the time the old monitors were deleted.
It s recommended you run the previous steps on a test/lab cluster before applying to you existing cluster.
As you had pointed, you do not want to re-add OSDs as this will initialise them and erase their data.

I hope this helps...Good luck.

Last edited on June 15, 2024, 4:36 pm by admin · #4

tntbizhub
14 Posts
June 15, 2024, 4:47 pm

I have executed:

https://docs.ceph.com/en/latest/rados/operations/add-or-rm-osds/
ceph auth add osd.1 osd 'allow *' mon 'allow rwx' -i /var/lib/ceph/osd/ceph-1/keyring
ceph auth add osd.2 osd 'allow *' mon 'allow rwx' -i /var/lib/ceph/osd/ceph-2/keyring
ceph auth add osd.4 osd 'allow *' mon 'allow rwx' -i /var/lib/ceph/osd/ceph-4/keyring

and now I can start the OSD services without errors:
sudo systemctl start ceph-osd@1
sudo systemctl start ceph-osd@2
sudo systemctl start ceph-osd@4

but.... I get 3 wierd stuff...
-> 3 disks added to the cluster but offline
-> on node list, opening the disks of node-3, I have the disks to add with a + sign plus the osd.1, osd.2 e osd.4 without name, size, with usage on each line saying OSD1, OSD2, OSD4... !?
-> Doing systemctl status ceph-osd@1.service I see...
Jun 15 12:31:36 san-3 ceph-osd[1892]: 2024-06-15T12:31:36.981+0100 7efd0547c3c0 -1 Falling back to public interface
Jun 15 12:31:55 san-3 ceph-osd[1892]: 2024-06-15T12:31:55.722+0100 7efd0547c3c0 -1 osd.1 19837 log_to_monitors true
Jun 15 12:31:56 san-3 ceph-osd[1892]: 2024-06-15T12:31:56.242+0100 7efcfd9e3700 -1 osd.1 19837 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory

Could it be the cause of the disk not "coming online" because they are in wrong IP/interface?

Any hint will be appreciated....

#5

tntbizhub
14 Posts
June 16, 2024, 1:17 pm

Another challenge...

Message on /var/log/ceph/ceph-mon.san-3.log

2024-06-16T08:40:13.392+0100 7f875ee2da00 0 starting mon.san-3 rank 2 at public addrs [v2:10.0.110.3:3300/0,v1:10.0.110.3:6789/0] at bind addrs [v2:10.0.110.3:3300/0,v1:10.0.110.3:6789/0] mon_data /var/lib/ceph/mon/ceph-san-3 fsid f090a6c1-eb64-45e8-9873-ac109361ebbc
2024-06-16T08:40:13.396+0100 7f875ee2da00 1 mon.san-3@-1(???) e3 preinit fsid f090a6c1-eb64-45e8-9873-ac109361ebbc
2024-06-16T08:40:13.396+0100 7f875ee2da00 -1 mon.san-3@-1(???) e3 error: cluster_uuid file exists with value 81acea00-fc7b-4131-a038-43ef1aa2efcc, != our uuid f090a6c1-eb64-45e8-9873-ac109361ebbc
2024-06-16T08:40:13.396+0100 7f875ee2da00 -1 failed to initialize

I have used documentation from https://mattventura.net/2024/06/08/how-to-fix-ceph-error-cluster_uuid-file-exists-with-value-x-our-uuid-y/comment-page-1/#comment-638883

systemctl stop ceph-mon
ceph-mon -i san-3 –extract-monmap /tmp/monmap
monmaptool –clobber –fsid f090a6c1-eb64-45e8-9873-ac109361ebbc /tmp/monmap
ceph-mon -i san-3 –inject-monmap /tmp/monmap
systemctl start ceph-mon

Keep getting same error 🙁

Then I searched for the "old cluster id" with

grep -Rnw '/var/lib/ceph/mon/ceph-san-3/' -e '81acea00-fc7b-4131-a038-43ef1aa2efcc'  #(old ID)

and found it on Binary file /var/lib/ceph/mon/ceph-san-3/store.db/000321.sst

Trying to find the logic on this... the only way that the old  cluster ID appears on the config of the "new cluster" is "coming" from the OSD's I am trying to add to the new cluster... even after changing with monmaptool --clobber --fsid f090a6c1-eb64-45e8-9873-ac109361ebbc /tmp/monmap and importing it back with ceph-mon -i san-3 –inject-monmap /tmp/monmap !?!?

I hope this somewhat helps others in the future not to do have the same idiotic idea I had to reinstall the servers...

Last edited on June 16, 2024, 1:18 pm by tntbizhub · #6

admin
2,930 Posts
June 16, 2024, 4:53 pm

the monitors seems to be still see-ing the new fsid, have you followed steps outlined earlier? also as recommended, test it on test/lab first.

#7

tntbizhub
14 Posts
June 17, 2024, 11:47 am

I had missed your indication of changing the fsid to the old one... now i have done it... that improved a lot, less errors etc...

I am trying to follow this docs -> https://www.reddit.com/r/ceph/comments/eah4v6/recreate_cluster_with_existing_osds/ to see if it works.

#8

admin
2,930 Posts
June 17, 2024, 2:31 pm

Please do follow the steps in order, setup osd authorization after the mons have the correct/old data.

#9

tntbizhub
14 Posts
June 17, 2024, 10:33 pm
Quote from admin on June 17, 2024, 2:31 pm

Please do follow the steps in order, setup osd authorization after the mons have the correct/old data.

Thanks again, I think I got stuck because of what you just wrote.... ok... will start over and do it "the right way" 🙂

#10
Pages: 1 2