Forums

Home / Forums

You need to log in to create posts and topics. Login · Register

Journal not showing on OSD's

Pages: 1 2 3 4

neiltorda
98 Posts
January 15, 2021, 9:18 pm

We have a petasan cluster with 4 nodes.

We were having slow access speeds: ceph -s showing client io at around 150MiB/s rd, 8MiB/s wr

Each of the 4 nodes have 16 OSD's each, for a total of 64 OSD.

The OSD's are 10T disks (spinning) but I configured them for Journal and Cache using 1T ssd disks.

On 2 of the 4 systems, the OSD's are showing that they have Cache and Journal, but the other 2 nodes are only showing cache.. the journal disk is missing from the list.

Below are nodes 1 and 2. 1 doesn't show the journal, but 2 does.

If you look at disk sdf on node1 pic, you can see it is at 3%. Earlier today I removed that OSD so I could re-add it with both the Journal and the Cache… but it is still only showing the cache, the journal did not add in even though I selected it when I recreated the OSD.

What would be preventing the OSD's from from having journals?
Are they there and it is an issue in the interface not showing them?

 

Node 1:

Node 2:

node 2

Last edited on January 15, 2021, 9:19 pm by neiltorda · #1

admin
2,930 Posts
January 16, 2021, 1:21 pm

on node 1, what is the ouptput of:

ls -l /var/lib/ceph/osd/ceph-9
ceph-volume lvm list --format json

#2

neiltorda
98 Posts
January 16, 2021, 2:24 pm

root@petasan1:~# ls -l /var/lib/ceph/osd/ceph-9
total 52
-rw-r--r-- 1 ceph ceph 436 Jan 15 09:30 activate.monmap
lrwxrwxrwx 1 ceph ceph 58 Jan 15 09:31 block -> /dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.9/main
lrwxrwxrwx 1 ceph ceph 10 Jan 15 09:31 block.db -> /dev/sdad1
-rw------- 1 ceph ceph 2 Jan 15 09:30 bluefs
-rw------- 1 ceph ceph 37 Jan 15 09:31 ceph_fsid
-rw-r--r-- 1 ceph ceph 37 Jan 15 09:31 fsid
-rw------- 1 ceph ceph 55 Jan 15 09:31 keyring
-rw------- 1 ceph ceph 8 Jan 15 09:30 kv_backend
-rw------- 1 ceph ceph 21 Jan 15 09:31 magic
-rw------- 1 ceph ceph 4 Jan 15 09:31 mkfs_done
-rw------- 1 ceph ceph 41 Jan 15 09:31 osd_key
-rw------- 1 ceph ceph 6 Jan 15 09:31 ready
-rw------- 1 ceph ceph 3 Jan 15 09:31 require_osd_release
-rw------- 1 ceph ceph 10 Jan 15 09:31 type
-rw------- 1 ceph ceph 2 Jan 15 09:31 whoami

 

 

root@petasan1:~# ceph-volume lvm list --format json
{
"10": [
{
"devices": [],
"lv_name": "main",
"lv_path": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.10/main",
"lv_size": "<9.10t",
"lv_tags": "ceph.block_device=/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.10/main,ceph.block_uuid=uH2FpA-QkVa-Hkkt-qLWM-132b-Rpda-wR5z2m,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=1da111ec-ffe8-4029-9834-e0988079925b,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/sdae1,ceph.db_uuid=cb5b5825-41fe-4c7d-8fa0-a1f0cd61f9f2,ceph.encrypted=0,ceph.osd_fsid=4de2b3cb-9c47-4a06-b33b-8bb909c7422f,ceph.osd_id=10,ceph.type=block,ceph.vdo=0",
"lv_uuid": "uH2FpA-QkVa-Hkkt-qLWM-132b-Rpda-wR5z2m",
"name": "main",
"path": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.10/main",
"tags": {
"ceph.block_device": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.10/main",
"ceph.block_uuid": "uH2FpA-QkVa-Hkkt-qLWM-132b-Rpda-wR5z2m",
"ceph.cephx_lockbox_secret": "",
"ceph.cluster_fsid": "1da111ec-ffe8-4029-9834-e0988079925b",
"ceph.cluster_name": "ceph",
"ceph.crush_device_class": "None",
"ceph.db_device": "/dev/sdae1",
"ceph.db_uuid": "cb5b5825-41fe-4c7d-8fa0-a1f0cd61f9f2",
"ceph.encrypted": "0",
"ceph.osd_fsid": "4de2b3cb-9c47-4a06-b33b-8bb909c7422f",
"ceph.osd_id": "10",
"ceph.type": "block",
"ceph.vdo": "0"
},
"type": "block",
"vg_name": "ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.10"
},
{
"path": "/dev/sdae1",
"tags": {
"PARTUUID": "cb5b5825-41fe-4c7d-8fa0-a1f0cd61f9f2"
},
"type": "db"
}
],
"11": [
{
"devices": [],
"lv_name": "main",
"lv_path": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.11/main",
"lv_size": "<9.10t",
"lv_tags": "ceph.block_device=/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.11/main,ceph.block_uuid=UECYBx-WnIA-pqUP-leiU-C1Eg-pGxS-041MdQ,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=1da111ec-ffe8-4029-9834-e0988079925b,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/sdaf1,ceph.db_uuid=04a232ec-3731-44ab-8c0f-e771f0b680a1,ceph.encrypted=0,ceph.osd_fsid=15973941-f3da-40e2-8e9f-460f2d5bc20f,ceph.osd_id=11,ceph.type=block,ceph.vdo=0",
"lv_uuid": "UECYBx-WnIA-pqUP-leiU-C1Eg-pGxS-041MdQ",
"name": "main",
"path": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.11/main",
"tags": {
"ceph.block_device": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.11/main",
"ceph.block_uuid": "UECYBx-WnIA-pqUP-leiU-C1Eg-pGxS-041MdQ",
"ceph.cephx_lockbox_secret": "",
"ceph.cluster_fsid": "1da111ec-ffe8-4029-9834-e0988079925b",
"ceph.cluster_name": "ceph",
"ceph.crush_device_class": "None",
"ceph.db_device": "/dev/sdaf1",
"ceph.db_uuid": "04a232ec-3731-44ab-8c0f-e771f0b680a1",
"ceph.encrypted": "0",
"ceph.osd_fsid": "15973941-f3da-40e2-8e9f-460f2d5bc20f",
"ceph.osd_id": "11",
"ceph.type": "block",
"ceph.vdo": "0"
},
"type": "block",
"vg_name": "ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.11"
},
{
"path": "/dev/sdaf1",
"tags": {
"PARTUUID": "04a232ec-3731-44ab-8c0f-e771f0b680a1"
},
"type": "db"
}
],
"12": [
{
"devices": [],
"lv_name": "main",
"lv_path": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.12/main",
"lv_size": "<9.10t",
"lv_tags": "ceph.block_device=/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.12/main,ceph.block_uuid=4V39uj-YkXj-ifeu-X3Q1-qaoX-c330-jbe01i,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=1da111ec-ffe8-4029-9834-e0988079925b,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/sdad2,ceph.db_uuid=5c9e231e-c084-4e94-a642-c8c378e58a71,ceph.encrypted=0,ceph.osd_fsid=213641e6-e2d3-4424-b8b8-6fe78b64fa8f,ceph.osd_id=12,ceph.type=block,ceph.vdo=0",
"lv_uuid": "4V39uj-YkXj-ifeu-X3Q1-qaoX-c330-jbe01i",
"name": "main",
"path": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.12/main",
"tags": {
"ceph.block_device": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.12/main",
"ceph.block_uuid": "4V39uj-YkXj-ifeu-X3Q1-qaoX-c330-jbe01i",
"ceph.cephx_lockbox_secret": "",
"ceph.cluster_fsid": "1da111ec-ffe8-4029-9834-e0988079925b",
"ceph.cluster_name": "ceph",
"ceph.crush_device_class": "None",
"ceph.db_device": "/dev/sdad2",
"ceph.db_uuid": "5c9e231e-c084-4e94-a642-c8c378e58a71",
"ceph.encrypted": "0",
"ceph.osd_fsid": "213641e6-e2d3-4424-b8b8-6fe78b64fa8f",
"ceph.osd_id": "12",
"ceph.type": "block",
"ceph.vdo": "0"
},
"type": "block",
"vg_name": "ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.12"
},
{
"path": "/dev/sdad2",
"tags": {
"PARTUUID": "5c9e231e-c084-4e94-a642-c8c378e58a71"
},
"type": "db"
}
],
"13": [
{
"devices": [],
"lv_name": "main",
"lv_path": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.13/main",
"lv_size": "<9.10t",
"lv_tags": "ceph.block_device=/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.13/main,ceph.block_uuid=AYClMv-qK3V-sGcb-oQ39-VYb8-aRAQ-Ghu0vu,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=1da111ec-ffe8-4029-9834-e0988079925b,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/sdae2,ceph.db_uuid=e8c48f9e-dad9-449c-a60b-b6dc508040ed,ceph.encrypted=0,ceph.osd_fsid=f8bdc18b-8862-4117-8bdf-b2ac50f4f3b2,ceph.osd_id=13,ceph.type=block,ceph.vdo=0",
"lv_uuid": "AYClMv-qK3V-sGcb-oQ39-VYb8-aRAQ-Ghu0vu",
"name": "main",
"path": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.13/main",
"tags": {
"ceph.block_device": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.13/main",
"ceph.block_uuid": "AYClMv-qK3V-sGcb-oQ39-VYb8-aRAQ-Ghu0vu",
"ceph.cephx_lockbox_secret": "",
"ceph.cluster_fsid": "1da111ec-ffe8-4029-9834-e0988079925b",
"ceph.cluster_name": "ceph",
"ceph.crush_device_class": "None",
"ceph.db_device": "/dev/sdae2",
"ceph.db_uuid": "e8c48f9e-dad9-449c-a60b-b6dc508040ed",
"ceph.encrypted": "0",
"ceph.osd_fsid": "f8bdc18b-8862-4117-8bdf-b2ac50f4f3b2",
"ceph.osd_id": "13",
"ceph.type": "block",
"ceph.vdo": "0"
},
"type": "block",
"vg_name": "ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.13"
},
{
"path": "/dev/sdae2",
"tags": {
"PARTUUID": "e8c48f9e-dad9-449c-a60b-b6dc508040ed"
},
"type": "db"
}
],
"14": [
{
"devices": [],
"lv_name": "main",
"lv_path": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.14/main",
"lv_size": "<9.10t",
"lv_tags": "ceph.block_device=/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.14/main,ceph.block_uuid=ql6IWc-dknJ-Jn5c-VJH4-Us8I-6q6K-KuCutJ,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=1da111ec-ffe8-4029-9834-e0988079925b,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/sdaf2,ceph.db_uuid=91d54f53-95ed-46ce-bb32-d52ef59ae031,ceph.encrypted=0,ceph.osd_fsid=49e0e055-901c-42fd-a74f-de213c6d63b3,ceph.osd_id=14,ceph.type=block,ceph.vdo=0",
"lv_uuid": "ql6IWc-dknJ-Jn5c-VJH4-Us8I-6q6K-KuCutJ",
"name": "main",
"path": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.14/main",
"tags": {
"ceph.block_device": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.14/main",
"ceph.block_uuid": "ql6IWc-dknJ-Jn5c-VJH4-Us8I-6q6K-KuCutJ",
"ceph.cephx_lockbox_secret": "",
"ceph.cluster_fsid": "1da111ec-ffe8-4029-9834-e0988079925b",
"ceph.cluster_name": "ceph",
"ceph.crush_device_class": "None",
"ceph.db_device": "/dev/sdaf2",
"ceph.db_uuid": "91d54f53-95ed-46ce-bb32-d52ef59ae031",
"ceph.encrypted": "0",
"ceph.osd_fsid": "49e0e055-901c-42fd-a74f-de213c6d63b3",
"ceph.osd_id": "14",
"ceph.type": "block",
"ceph.vdo": "0"
},
"type": "block",
"vg_name": "ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.14"
},
{
"path": "/dev/sdaf2",
"tags": {
"PARTUUID": "91d54f53-95ed-46ce-bb32-d52ef59ae031"
},
"type": "db"
}
],
"15": [
{
"devices": [],
"lv_name": "main",
"lv_path": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.15/main",
"lv_size": "<9.10t",
"lv_tags": "ceph.block_device=/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.15/main,ceph.block_uuid=1cUekS-e5RS-J8tw-xPhg-6MSF-JNjX-mSQbQq,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=1da111ec-ffe8-4029-9834-e0988079925b,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/sdad3,ceph.db_uuid=9f37df57-cf67-45b0-97e7-4b7723f776b5,ceph.encrypted=0,ceph.osd_fsid=bfda9d74-f68e-41dd-8735-3e9b9fd7fd61,ceph.osd_id=15,ceph.type=block,ceph.vdo=0",
"lv_uuid": "1cUekS-e5RS-J8tw-xPhg-6MSF-JNjX-mSQbQq",
"name": "main",
"path": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.15/main",
"tags": {
"ceph.block_device": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.15/main",
"ceph.block_uuid": "1cUekS-e5RS-J8tw-xPhg-6MSF-JNjX-mSQbQq",
"ceph.cephx_lockbox_secret": "",
"ceph.cluster_fsid": "1da111ec-ffe8-4029-9834-e0988079925b",
"ceph.cluster_name": "ceph",
"ceph.crush_device_class": "None",
"ceph.db_device": "/dev/sdad3",
"ceph.db_uuid": "9f37df57-cf67-45b0-97e7-4b7723f776b5",
"ceph.encrypted": "0",
"ceph.osd_fsid": "bfda9d74-f68e-41dd-8735-3e9b9fd7fd61",
"ceph.osd_id": "15",
"ceph.type": "block",
"ceph.vdo": "0"
},
"type": "block",
"vg_name": "ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.15"
},
{
"path": "/dev/sdad3",
"tags": {
"PARTUUID": "9f37df57-cf67-45b0-97e7-4b7723f776b5"
},
"type": "db"
}
],
"16": [
{
"devices": [],
"lv_name": "main",
"lv_path": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.16/main",
"lv_size": "<9.10t",
"lv_tags": "ceph.block_device=/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.16/main,ceph.block_uuid=ZYS3Uv-kdKQ-t308-qVKl-1bjv-XNsu-CrB16g,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=1da111ec-ffe8-4029-9834-e0988079925b,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/sdae3,ceph.db_uuid=19a64ba9-b79c-484d-a04e-1cee213061e9,ceph.encrypted=0,ceph.osd_fsid=3da8c954-81c1-430f-905e-6ed35f27c0a6,ceph.osd_id=16,ceph.type=block,ceph.vdo=0",
"lv_uuid": "ZYS3Uv-kdKQ-t308-qVKl-1bjv-XNsu-CrB16g",
"name": "main",
"path": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.16/main",
"tags": {
"ceph.block_device": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.16/main",
"ceph.block_uuid": "ZYS3Uv-kdKQ-t308-qVKl-1bjv-XNsu-CrB16g",
"ceph.cephx_lockbox_secret": "",
"ceph.cluster_fsid": "1da111ec-ffe8-4029-9834-e0988079925b",
"ceph.cluster_name": "ceph",
"ceph.crush_device_class": "None",
"ceph.db_device": "/dev/sdae3",
"ceph.db_uuid": "19a64ba9-b79c-484d-a04e-1cee213061e9",
"ceph.encrypted": "0",
"ceph.osd_fsid": "3da8c954-81c1-430f-905e-6ed35f27c0a6",
"ceph.osd_id": "16",
"ceph.type": "block",
"ceph.vdo": "0"
},
"type": "block",
"vg_name": "ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.16"
},
{
"path": "/dev/sdae3",
"tags": {
"PARTUUID": "19a64ba9-b79c-484d-a04e-1cee213061e9"
},
"type": "db"
}
],
"17": [
{
"devices": [],
"lv_name": "main",
"lv_path": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.17/main",
"lv_size": "<9.10t",
"lv_tags": "ceph.block_device=/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.17/main,ceph.block_uuid=1f8Tq1-pr08-1gXg-Df5s-ZCTn-TU7B-Ihm2bv,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=1da111ec-ffe8-4029-9834-e0988079925b,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/sdaf3,ceph.db_uuid=4843c967-893d-47a6-8624-43ccdf0c976e,ceph.encrypted=0,ceph.osd_fsid=6f236339-013d-4999-9b05-92200bf228d7,ceph.osd_id=17,ceph.type=block,ceph.vdo=0",
"lv_uuid": "1f8Tq1-pr08-1gXg-Df5s-ZCTn-TU7B-Ihm2bv",
"name": "main",
"path": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.17/main",
"tags": {
"ceph.block_device": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.17/main",
"ceph.block_uuid": "1f8Tq1-pr08-1gXg-Df5s-ZCTn-TU7B-Ihm2bv",
"ceph.cephx_lockbox_secret": "",
"ceph.cluster_fsid": "1da111ec-ffe8-4029-9834-e0988079925b",
"ceph.cluster_name": "ceph",
"ceph.crush_device_class": "None",
"ceph.db_device": "/dev/sdaf3",
"ceph.db_uuid": "4843c967-893d-47a6-8624-43ccdf0c976e",
"ceph.encrypted": "0",
"ceph.osd_fsid": "6f236339-013d-4999-9b05-92200bf228d7",
"ceph.osd_id": "17",
"ceph.type": "block",
"ceph.vdo": "0"
},
"type": "block",
"vg_name": "ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.17"
},
{
"path": "/dev/sdaf3",
"tags": {
"PARTUUID": "4843c967-893d-47a6-8624-43ccdf0c976e"
},
"type": "db"
}
],
"27": [
{
"devices": [],
"lv_name": "main",
"lv_path": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.27/main",
"lv_size": "<9.10t",
"lv_tags": "ceph.block_device=/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.27/main,ceph.block_uuid=WS8j2E-M3b2-AfGD-KgXQ-rQVY-dJ1o-rFZo8M,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=1da111ec-ffe8-4029-9834-e0988079925b,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/sdaj1,ceph.db_uuid=23a17727-034f-4e68-aae2-77a7b2e8ca4a,ceph.encrypted=0,ceph.osd_fsid=6a284929-01a2-4b3c-81c6-a321966372fc,ceph.osd_id=27,ceph.type=block,ceph.vdo=0",
"lv_uuid": "WS8j2E-M3b2-AfGD-KgXQ-rQVY-dJ1o-rFZo8M",
"name": "main",
"path": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.27/main",
"tags": {
"ceph.block_device": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.27/main",
"ceph.block_uuid": "WS8j2E-M3b2-AfGD-KgXQ-rQVY-dJ1o-rFZo8M",
"ceph.cephx_lockbox_secret": "",
"ceph.cluster_fsid": "1da111ec-ffe8-4029-9834-e0988079925b",
"ceph.cluster_name": "ceph",
"ceph.crush_device_class": "None",
"ceph.db_device": "/dev/sdaj1",
"ceph.db_uuid": "23a17727-034f-4e68-aae2-77a7b2e8ca4a",
"ceph.encrypted": "0",
"ceph.osd_fsid": "6a284929-01a2-4b3c-81c6-a321966372fc",
"ceph.osd_id": "27",
"ceph.type": "block",
"ceph.vdo": "0"
},
"type": "block",
"vg_name": "ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.27"
},
{
"path": "/dev/sdaj1",
"tags": {
"PARTUUID": "23a17727-034f-4e68-aae2-77a7b2e8ca4a"
},
"type": "db"
}
],
"28": [
{
"devices": [],
"lv_name": "main",
"lv_path": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.28/main",
"lv_size": "<9.10t",
"lv_tags": "ceph.block_device=/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.28/main,ceph.block_uuid=12FWfw-Yzp9-6M7h-m752-Q3yL-bDBn-wuCKU1,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=1da111ec-ffe8-4029-9834-e0988079925b,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/sdaj2,ceph.db_uuid=48fbbdce-0310-4125-8cbc-541a97873cf5,ceph.encrypted=0,ceph.osd_fsid=3bf2a4f2-47e4-45eb-ad10-30309693461e,ceph.osd_id=28,ceph.type=block,ceph.vdo=0",
"lv_uuid": "12FWfw-Yzp9-6M7h-m752-Q3yL-bDBn-wuCKU1",
"name": "main",
"path": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.28/main",
"tags": {
"ceph.block_device": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.28/main",
"ceph.block_uuid": "12FWfw-Yzp9-6M7h-m752-Q3yL-bDBn-wuCKU1",
"ceph.cephx_lockbox_secret": "",
"ceph.cluster_fsid": "1da111ec-ffe8-4029-9834-e0988079925b",
"ceph.cluster_name": "ceph",
"ceph.crush_device_class": "None",
"ceph.db_device": "/dev/sdaj2",
"ceph.db_uuid": "48fbbdce-0310-4125-8cbc-541a97873cf5",
"ceph.encrypted": "0",
"ceph.osd_fsid": "3bf2a4f2-47e4-45eb-ad10-30309693461e",
"ceph.osd_id": "28",
"ceph.type": "block",
"ceph.vdo": "0"
},
"type": "block",
"vg_name": "ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.28"
},
{
"path": "/dev/sdaj2",
"tags": {
"PARTUUID": "48fbbdce-0310-4125-8cbc-541a97873cf5"
},
"type": "db"
}
],
"29": [
{
"devices": [],
"lv_name": "main",
"lv_path": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.29/main",
"lv_size": "<9.10t",
"lv_tags": "ceph.block_device=/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.29/main,ceph.block_uuid=QDS45r-UFNd-FHrQ-KOWT-YU3c-sc2t-WHW6pR,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=1da111ec-ffe8-4029-9834-e0988079925b,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/sdaj3,ceph.db_uuid=cc20854f-3383-4b86-96b2-6f2d04a669ca,ceph.encrypted=0,ceph.osd_fsid=7146702b-9ca2-4cba-872a-1623b7db3fda,ceph.osd_id=29,ceph.type=block,ceph.vdo=0",
"lv_uuid": "QDS45r-UFNd-FHrQ-KOWT-YU3c-sc2t-WHW6pR",
"name": "main",
"path": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.29/main",
"tags": {
"ceph.block_device": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.29/main",
"ceph.block_uuid": "QDS45r-UFNd-FHrQ-KOWT-YU3c-sc2t-WHW6pR",
"ceph.cephx_lockbox_secret": "",
"ceph.cluster_fsid": "1da111ec-ffe8-4029-9834-e0988079925b",
"ceph.cluster_name": "ceph",
"ceph.crush_device_class": "None",
"ceph.db_device": "/dev/sdaj3",
"ceph.db_uuid": "cc20854f-3383-4b86-96b2-6f2d04a669ca",
"ceph.encrypted": "0",
"ceph.osd_fsid": "7146702b-9ca2-4cba-872a-1623b7db3fda",
"ceph.osd_id": "29",
"ceph.type": "block",
"ceph.vdo": "0"
},
"type": "block",
"vg_name": "ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.29"
},
{
"path": "/dev/sdaj3",
"tags": {
"PARTUUID": "cc20854f-3383-4b86-96b2-6f2d04a669ca"
},
"type": "db"
}
],
"36": [
{
"devices": [],
"lv_name": "main",
"lv_path": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.36/main",
"lv_size": "<9.10t",
"lv_tags": "ceph.block_device=/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.36/main,ceph.block_uuid=31cCLM-AxUh-mJ9x-uf7r-3dQy-88oy-dSmCbG,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=1da111ec-ffe8-4029-9834-e0988079925b,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/sdal1,ceph.db_uuid=9384ba36-8473-4864-a68c-c8890de9226e,ceph.encrypted=0,ceph.osd_fsid=8d4bd91b-dcc5-478d-8d14-11afff417c60,ceph.osd_id=36,ceph.type=block,ceph.vdo=0",
"lv_uuid": "31cCLM-AxUh-mJ9x-uf7r-3dQy-88oy-dSmCbG",
"name": "main",
"path": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.36/main",
"tags": {
"ceph.block_device": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.36/main",
"ceph.block_uuid": "31cCLM-AxUh-mJ9x-uf7r-3dQy-88oy-dSmCbG",
"ceph.cephx_lockbox_secret": "",
"ceph.cluster_fsid": "1da111ec-ffe8-4029-9834-e0988079925b",
"ceph.cluster_name": "ceph",
"ceph.crush_device_class": "None",
"ceph.db_device": "/dev/sdal1",
"ceph.db_uuid": "9384ba36-8473-4864-a68c-c8890de9226e",
"ceph.encrypted": "0",
"ceph.osd_fsid": "8d4bd91b-dcc5-478d-8d14-11afff417c60",
"ceph.osd_id": "36",
"ceph.type": "block",
"ceph.vdo": "0"
},
"type": "block",
"vg_name": "ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.36"
},
{
"path": "/dev/sdal1",
"tags": {
"PARTUUID": "9384ba36-8473-4864-a68c-c8890de9226e"
},
"type": "db"
}
],
"37": [
{
"devices": [],
"lv_name": "main",
"lv_path": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.37/main",
"lv_size": "<9.10t",
"lv_tags": "ceph.block_device=/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.37/main,ceph.block_uuid=yeL3mQ-H2rZ-IqyI-rweU-Xk6o-TQgk-vkHwTu,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=1da111ec-ffe8-4029-9834-e0988079925b,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/sdal2,ceph.db_uuid=ab5862f7-0f73-464b-ba64-42340a4243b8,ceph.encrypted=0,ceph.osd_fsid=c0e03edf-2eba-4ecc-8bb8-a8c9eabf3adb,ceph.osd_id=37,ceph.type=block,ceph.vdo=0",
"lv_uuid": "yeL3mQ-H2rZ-IqyI-rweU-Xk6o-TQgk-vkHwTu",
"name": "main",
"path": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.37/main",
"tags": {
"ceph.block_device": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.37/main",
"ceph.block_uuid": "yeL3mQ-H2rZ-IqyI-rweU-Xk6o-TQgk-vkHwTu",
"ceph.cephx_lockbox_secret": "",
"ceph.cluster_fsid": "1da111ec-ffe8-4029-9834-e0988079925b",
"ceph.cluster_name": "ceph",
"ceph.crush_device_class": "None",
"ceph.db_device": "/dev/sdal2",
"ceph.db_uuid": "ab5862f7-0f73-464b-ba64-42340a4243b8",
"ceph.encrypted": "0",
"ceph.osd_fsid": "c0e03edf-2eba-4ecc-8bb8-a8c9eabf3adb",
"ceph.osd_id": "37",
"ceph.type": "block",
"ceph.vdo": "0"
},
"type": "block",
"vg_name": "ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.37"
},
{
"path": "/dev/sdal2",
"tags": {
"PARTUUID": "ab5862f7-0f73-464b-ba64-42340a4243b8"
},
"type": "db"
}
],
"38": [
{
"devices": [],
"lv_name": "main",
"lv_path": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.38/main",
"lv_size": "<9.10t",
"lv_tags": "ceph.block_device=/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.38/main,ceph.block_uuid=N3WbD3-UFoH-w9jT-51VR-BnnI-m0eF-tGTkvD,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=1da111ec-ffe8-4029-9834-e0988079925b,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/sdal3,ceph.db_uuid=56dcb9c4-8598-483e-af1e-8e824eaed80c,ceph.encrypted=0,ceph.osd_fsid=3da9e17e-e1b3-4a79-aec5-edabbc992694,ceph.osd_id=38,ceph.type=block,ceph.vdo=0",
"lv_uuid": "N3WbD3-UFoH-w9jT-51VR-BnnI-m0eF-tGTkvD",
"name": "main",
"path": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.38/main",
"tags": {
"ceph.block_device": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.38/main",
"ceph.block_uuid": "N3WbD3-UFoH-w9jT-51VR-BnnI-m0eF-tGTkvD",
"ceph.cephx_lockbox_secret": "",
"ceph.cluster_fsid": "1da111ec-ffe8-4029-9834-e0988079925b",
"ceph.cluster_name": "ceph",
"ceph.crush_device_class": "None",
"ceph.db_device": "/dev/sdal3",
"ceph.db_uuid": "56dcb9c4-8598-483e-af1e-8e824eaed80c",
"ceph.encrypted": "0",
"ceph.osd_fsid": "3da9e17e-e1b3-4a79-aec5-edabbc992694",
"ceph.osd_id": "38",
"ceph.type": "block",
"ceph.vdo": "0"
},
"type": "block",
"vg_name": "ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.38"
},
{
"path": "/dev/sdal3",
"tags": {
"PARTUUID": "56dcb9c4-8598-483e-af1e-8e824eaed80c"
},
"type": "db"
}
],
"39": [
{
"devices": [],
"lv_name": "main",
"lv_path": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.39/main",
"lv_size": "<9.10t",
"lv_tags": "ceph.block_device=/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.39/main,ceph.block_uuid=z5Tt0m-wAU4-qpVP-Fdoa-hsTj-wzhi-TR9fzO,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=1da111ec-ffe8-4029-9834-e0988079925b,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/sdad4,ceph.db_uuid=4ac90e8e-6f45-4e86-a008-bf1962a96f7e,ceph.encrypted=0,ceph.osd_fsid=2f2e56b8-be6e-4ba6-8f3d-a877a8b867a6,ceph.osd_id=39,ceph.type=block,ceph.vdo=0",
"lv_uuid": "z5Tt0m-wAU4-qpVP-Fdoa-hsTj-wzhi-TR9fzO",
"name": "main",
"path": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.39/main",
"tags": {
"ceph.block_device": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.39/main",
"ceph.block_uuid": "z5Tt0m-wAU4-qpVP-Fdoa-hsTj-wzhi-TR9fzO",
"ceph.cephx_lockbox_secret": "",
"ceph.cluster_fsid": "1da111ec-ffe8-4029-9834-e0988079925b",
"ceph.cluster_name": "ceph",
"ceph.crush_device_class": "None",
"ceph.db_device": "/dev/sdad4",
"ceph.db_uuid": "4ac90e8e-6f45-4e86-a008-bf1962a96f7e",
"ceph.encrypted": "0",
"ceph.osd_fsid": "2f2e56b8-be6e-4ba6-8f3d-a877a8b867a6",
"ceph.osd_id": "39",
"ceph.type": "block",
"ceph.vdo": "0"
},
"type": "block",
"vg_name": "ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.39"
},
{
"path": "/dev/sdad4",
"tags": {
"PARTUUID": "4ac90e8e-6f45-4e86-a008-bf1962a96f7e"
},
"type": "db"
}
],
"9": [
{
"devices": [],
"lv_name": "main",
"lv_path": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.9/main",
"lv_size": "<9.10t",
"lv_tags": "ceph.block_device=/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.9/main,ceph.block_uuid=5fczlP-HpPQ-doNY-AfMd-S4Id-2Kuo-eXtdQq,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=1da111ec-ffe8-4029-9834-e0988079925b,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/sdad1,ceph.db_uuid=9ad5cec3-3511-4017-8e53-b4e9d6d9d2ac,ceph.encrypted=0,ceph.osd_fsid=61bb9d94-da8f-45b8-8818-b694737525d0,ceph.osd_id=9,ceph.osdspec_affinity=,ceph.type=block,ceph.vdo=0",
"lv_uuid": "5fczlP-HpPQ-doNY-AfMd-S4Id-2Kuo-eXtdQq",
"name": "main",
"path": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.9/main",
"tags": {
"ceph.block_device": "/dev/ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.9/main",
"ceph.block_uuid": "5fczlP-HpPQ-doNY-AfMd-S4Id-2Kuo-eXtdQq",
"ceph.cephx_lockbox_secret": "",
"ceph.cluster_fsid": "1da111ec-ffe8-4029-9834-e0988079925b",
"ceph.cluster_name": "ceph",
"ceph.crush_device_class": "None",
"ceph.db_device": "/dev/sdad1",
"ceph.db_uuid": "9ad5cec3-3511-4017-8e53-b4e9d6d9d2ac",
"ceph.encrypted": "0",
"ceph.osd_fsid": "61bb9d94-da8f-45b8-8818-b694737525d0",
"ceph.osd_id": "9",
"ceph.osdspec_affinity": "",
"ceph.type": "block",
"ceph.vdo": "0"
},
"type": "block",
"vg_name": "ps-1da111ec-ffe8-4029-9834-e0988079925b-wc-osd.9"
},
{
"path": "/dev/sdad1",
"tags": {
"PARTUUID": "9ad5cec3-3511-4017-8e53-b4e9d6d9d2ac"
},
"type": "db"
}
]
}

#3

admin
2,930 Posts
January 16, 2021, 4:03 pm

the journal is configured correctly. both commands show sdad1 as journal for osd 9.  It could be a ui display bug, what version are you using ? one thing  to try is maybe reduce the browser zoom it could be there is not enough space to display, just maybe.

as for performance, look at any high resource utilization under node stats, particularly % disk utilization.

 

Last edited on January 16, 2021, 4:04 pm by admin · #4

neiltorda
98 Posts
January 16, 2021, 4:48 pm

I am on 2.7.0 currently.

I changed the view and tried Chrome/Firefox/Safari and it only displays the cache disk on each of those.

But it is only on 2 of the 4 nodes.

Last edited on January 16, 2021, 4:49 pm by neiltorda · #5

neiltorda
98 Posts
January 16, 2021, 5:54 pm

I can no longer get the disk list to populate on any of the nodes.. just get the spinning loading icon.

I also have this repeating in the log on node1 (haven't checked the other nodes). this is one of the nodes that showed that there were no journals in the interface.

 

root@petasan1:~# tail -f /opt/petasan/log/PetaSAN.log 

 16/01/2021 12:48:07 INFO     partision rbd6p1 has no ptype.

 16/01/2021 12:48:08 INFO     partision sdad2 has no ptype.

 16/01/2021 12:48:08 INFO     partision sdad3 has no ptype.

 16/01/2021 12:48:08 INFO     partision sdad4 has no ptype.

 16/01/2021 12:48:09 ERROR    object of type 'NoneType' has no len()

 16/01/2021 12:48:23 INFO     partision rbd6p1 has no ptype.

 16/01/2021 12:48:24 INFO     partision sdad2 has no ptype.

 16/01/2021 12:48:24 INFO     partision sdad3 has no ptype.

 16/01/2021 12:48:24 INFO     partision sdad4 has no ptype.

 16/01/2021 12:49:04 INFO     call get assignments stats function.

 16/01/2021 12:49:23 INFO     partision rbd6p1 has no ptype.

 16/01/2021 12:49:23 INFO     partision sdad2 has no ptype.

 16/01/2021 12:49:23 INFO     partision sdad3 has no ptype.

 16/01/2021 12:49:23 INFO     partision sdad4 has no ptype.

 16/01/2021 12:49:24 ERROR    object of type 'NoneType' has no len()

 16/01/2021 12:49:56 INFO     partision rbd6p1 has no ptype.

 16/01/2021 12:49:56 INFO     partision sdad2 has no ptype.

 16/01/2021 12:49:56 INFO     partision sdad3 has no ptype.

 16/01/2021 12:49:56 INFO     partision sdad4 has no ptype.

 

Last edited on January 16, 2021, 5:54 pm by neiltorda · #6

admin
2,930 Posts
January 18, 2021, 6:39 pm

can you run the following disk_info_logs.py file

https://drive.google.com/file/d/1ImsQbQjUtDMANrjsexC7tBu2Hbkoxglt/view?usp=sharing

and email the output logs to contact-us at petasan.org

#7

neiltorda
98 Posts
January 19, 2021, 7:51 pm

I get this error when trying to run that script:

root@petasan1:~# python disk_info_logs.py 

Traceback (most recent call last):

  File "disk_info_logs.py", line 4, in <module>

    import PetaSAN.core.common.json2 as json

ImportError: No module named PetaSAN.core.common.json2

 

 

#8

admin
2,930 Posts
January 20, 2021, 11:28 am

can you check the first line in script is

#!/usr/bin/python3

to use python3

#9

neiltorda
98 Posts
January 20, 2021, 6:26 pm

it does say #!/usr/bin/python3

 

 

#10
Pages: 1 2 3 4