Forums

Home / Forums

You need to log in to create posts and topics. Login · Register

Client compat after upgrade to 2.0 / upmap


therm
121 Posts
January 4, 2019, 10:06 am

Hi,

As in https://ceph.com/community/new-luminous-upgrade-complete/ the command

ceph osd set-require-min-compat-client luminous

is needed to use the new upmap feature. This is needed to get an perfect distribution of pgs over the cluster.

The upgrade-guide does not mention that command, so I am asking: Is it save to use it when all servers were updated? (Output is within the upgrade from 2.0 to 2.1)

root@ceph-node-mru-2:~# ceph versions
{
 "mon": {
 "ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable)": 2,
 "ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous (stable)": 1
 },
 "mgr": {
 "ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable)": 2,
 "ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous (stable)": 1
 },
 "osd": {
 "ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable)": 168,
 "ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous (stable)": 24
 },
 "mds": {},
 "overall": {
 "ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable)": 172,
 "ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous (stable)": 26
 }
}


ceph features
{
 "mon": {
 "group": {
 "features": "0x1ffddff8eea4fffb",
 "release": "luminous",
 "num": 2
 },
 "group": {
 "features": "0x3ffddff8eea4fffb",
 "release": "luminous",
 "num": 1
 }
 },
 "osd": {
 "group": {
 "features": "0x1ffddff8eea4fffb",
 "release": "luminous",
 "num": 187
 },
 "group": {
 "features": "0x3ffddff8eea4fffb",
 "release": "luminous",
 "num": 5
 }
 },
 "client": {
 "group": {
 "features": "0x7010eb84aa42a52",
 "release": "jewel",
 "num": 7
 },
 "group": {
 "features": "0x7fddff8ee84bffb",
 "release": "jewel",
 "num": 5
 },
 "group": {
 "features": "0x1ffddff8eea4fffb",
 "release": "luminous",
 "num": 8
 }
 }
}

I am a bit confused why 'ceph features' mentions jewel. I have 9 servers in production while one management-node has no osds and the other 8 do have.
My goal is to enable the balancer plugin with upmap mode.

Regards,
Dennis
#1

admin
2,930 Posts
January 4, 2019, 3:33 pm

Our guide lists the command for OSDs

ceph osd require-osd-release luminous –-cluster CLUSTER_NAME

As for clients, we use the kernel client which does not have all features to support luminous, this is why it shows it your output as jewel. The kernel does have support for upmap, and some Ceph users would go ahead and bump the client version as in

https://ceph-users.ceph.narkive.com/zborbmhi/cephfs-kernel-client-versions-pg-upmap

It is not something we tested and i would not recommend it until the kernel has full luminous feature support. The balancer module :

http://docs.ceph.com/docs/mimic/mgr/balancer/

does give good results in the crush-compat mode (its default mode).

Last edited on January 4, 2019, 3:34 pm by admin · #2

therm
121 Posts
January 22, 2019, 4:27 pm

Ok, I learned that we have jewel clients. Is it save to use

ceph osd set-require-min-compat-client jewel 
ceph osd crush set-all-straw-buckets-to-straw2
ceph osd crush tunables jewel

like in

http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-September/029708.html  ?

It seems to be needed to get the balancer working. Current Setting is:

root@ceph-node-mru-1:~# ceph balancer status
{
"active": true,
"plans": [],
"mode": "crush-compat"
}

root@ceph-node-mru-1:~# ceph osd dump | grep min_compat_client
require_min_compat_client firefly
min_compat_client firefly

root@ceph-node-mru-1:~# ceph osd crush dump | grep straw
"alg": "straw",

...

root@ceph-node-mru-1:~# ceph osd crush dump | grep profile
"profile": "firefly",

This leads to:

root@ceph-node-mru-1:~# ceph balancer optimize myplan
Error EAGAIN: compat weight-set not available

Regards,

Dennis

 

Last edited on January 22, 2019, 4:57 pm by therm · #3

admin
2,930 Posts
January 22, 2019, 10:24 pm

Thanks for sharing this info.

Installation of PetaSAN 2.0 onward will already have min_compat_client jewel and straw2 algorithm. The value you see are most probably due to having upgraded from earlier versions.

Typically the algorithm change would move most of your data, but as per the link you shared, going from straw to straw2 only moved 5% of data. We have not tested this so i cannot say if this is applicable, i did look at the code and both use the same hashing function but straw2 applies the weight using a different logarithmic function, the hashing function being the same should give some comfort that not all data will be moved around. I would suggest you test this in a virtual environment yourself as a simulation, you may also try the crushtool utility with the "--test" parameter simulation could also help.

Last edited on January 22, 2019, 10:27 pm by admin · #4

therm
121 Posts
January 28, 2020, 8:37 am

Hi,

can someone confirm if balancer`s upmap mode is working in petasan 2.4?

Regards,

Dennis

#5

admin
2,930 Posts
January 28, 2020, 3:51 pm

yes you can use upmap balancer in 2.4

#6