Forums

Home / Forums

You need to log in to create posts and topics. Login · Register

Update Issue


neiltorda
98 Posts
October 14, 2023, 6:58 pm

Trying to update my cluster today following the online upgrade guide.

Current version is 3.1.0 on all nodes

Health Is OK and all disks are up and in.

Pasting full output of command at end of this message, but when trying to run the update.sh script, i get:

----

root@petasan6:~# /opt/petasan/scripts/online-updates/update.sh
Incorrect require_osd_release nautilus found. Before upgrading first set:
ceph osd require-osd-release octopus

This isn't covered in the guide. Do I just run that command and then rerun the update.sh script? does anything have to happen before running that command?

 

Thanks,
Neil

 

 

 

 

root@petasan6:~# apt update

Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease

Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]

Get:3 https://repo.45drives.com/debian focal InRelease [6,857 B]                                             

Get:4 http://archive.ubuntu.com/ubuntu focal-security InRelease [114 kB]                                               

Get:5 https://repo.45drives.com/debian focal/main amd64 Packages [52.2 kB]

Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2,888 kB]

Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [473 kB]

Get:9 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,123 kB]

Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [268 kB]

Get:11 http://archive.ubuntu.com/ubuntu focal-security/main amd64 Packages [2,500 kB] 

Get:12 http://archive.ubuntu.com/ubuntu focal-security/main Translation-en [390 kB]

Get:13 http://archive.ubuntu.com/ubuntu focal-security/universe amd64 Packages [891 kB]

Get:7 https://archive.petasan.org/repo_v3 petasan-v3 InRelease [1,659 B]       

Get:14 http://archive.ubuntu.com/ubuntu focal-security/universe Translation-en [186 kB]     

Get:15 https://archive.petasan.org/repo_v3 petasan-v3/updates amd64 Packages [15.8 kB]

Fetched 9,023 kB in 3s (3,500 kB/s)                               

Reading package lists... Done

Building dependency tree       

Reading state information... Done

196 packages can be upgraded. Run 'apt list --upgradable' to see them.

root@petasan6:~# apt install ca-certificates

Reading package lists... Done

Building dependency tree       

Reading state information... Done

The following packages will be upgraded:

  ca-certificates

1 upgraded, 0 newly installed, 0 to remove and 195 not upgraded.

Need to get 152 kB of archives.

After this operation, 23.6 kB of additional disk space will be used.

Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 ca-certificates all 20230311ubuntu0.20.04.1 [152 kB]

Fetched 152 kB in 1s (288 kB/s)         

Preconfiguring packages ...

(Reading database ... 50509 files and directories currently installed.)

Preparing to unpack .../ca-certificates_20230311ubuntu0.20.04.1_all.deb ...

Unpacking ca-certificates (20230311ubuntu0.20.04.1) over (20211016ubuntu0.20.04.1) ...

Setting up ca-certificates (20230311ubuntu0.20.04.1) ...

Updating certificates in /etc/ssl/certs...

rehash: warning: skipping ca-certificates.crt,it does not contain exactly one certificate or CRL

19 added, 6 removed; done.

Processing triggers for ca-certificates (20230311ubuntu0.20.04.1) ...

Updating certificates in /etc/ssl/certs...

0 added, 0 removed; done.

Running hooks in /etc/ca-certificates/update.d...

done.

root@petasan6:~# /opt/petasan/scripts/online-updates/update.sh

Incorrect require_osd_release nautilus found. Before upgrading first set:

ceph osd require-osd-release octopus

root@petasan6:~# 

#1

admin
2,930 Posts
October 14, 2023, 9:10 pm

Yes please run
ceph osd require-osd-release octopus

Additionally if your cluster was initially built using PetaSAN 2.3.0 or earlier, we also recommend to first run
ceph config set mon mon_mds_skip_sanity true

#2

neiltorda
98 Posts
October 14, 2023, 9:21 pm

so to confirm, should run in the following order:

ceph config set mon mon_mds_skip_sanity true (does this need to be run on one or all nodes?)

ceph osd require-osd-release octopus (does this need to be run on one or all nodes?)

 

then:

apt update

apt install ca-certificates

/opt/petasan/scripts/online-updates/update.sh

 

then once all systems have had the update script ran, run:

ceph osd require-osd-release quincy

 

Are those first 2 steps in the correct order?

Also, do there need to be any reboots throughout this process?

#3

admin
2,930 Posts
October 14, 2023, 11:03 pm

Yes it is correct.

The update script will tell you if you need to reboot or not.

The ceph command can run on any single node, they affect the ceph cluster not the local host.

Last edited on October 14, 2023, 11:04 pm by admin · #4

neiltorda
98 Posts
October 16, 2023, 1:34 pm

Thanks so much for the clarification.

This process worked fine on the first 2 nodes I updated, the 3rd node came up, but had a health warn that it needed to fix misplaced objects. It has been running that process for over 30 hours… started out fast but the closer it get to the end, the slower it goes. I assume it will be done this morning and I can move on to the next node to update:

 

cluster:

    id:     1da111ec-ffe8-4029-9834-e0988079925b

    health: HEALTH_WARN
            1/3 mons down, quorum petasan3,petasan2

  services:
    mon: 3 daemons, quorum petasan3,petasan2 (age 35h), out of quorum: petasan1
    mgr: petasan2(active, since 35h), standbys: petasan3, petasan1
    mds: 1/1 daemons up, 1 standby
    osd: 155 osds: 155 up (since 35h), 155 in (since 35h); 1 remapped pgs

  data:
    volumes: 1/1 healthy
    pools:   6 pools, 3489 pgs
    objects: 54.87M objects, 208 TiB
    usage:   423 TiB used, 694 TiB / 1.1 PiB avail

    pgs:     9187/109739348 objects misplaced (0.008%)
             3472 active+clean
             9    active+clean+scrubbing
             7    active+clean+scrubbing+deep
             1    active+remapped+backfilling

  io:
    client:   3.5 KiB/s rd, 357 KiB/s wr, 1 op/s rd, 3 op/s wr
    recovery: 13 MiB/s, 3 objects/s

 

#5