Forums

Home / Forums

You need to log in to create posts and topics. Login · Register

Pool Inactive


Andoni
7 Posts
November 3, 2020, 7:17 am

Hi,

I have a issue, i have only one pool, it's "Inactive" on pool GUI section, and no iSCSI disk appears.

How can i "reactivate" the pool without dataloss?

I have 5 nodes: 3 Management + 2 Storage, each one of this has 10 disk.

I have the last petasan version 2.6.2

Thanks to all

Here is the ceph status:

root@petasan01:~# ceph -s
cluster:
id:     644e37d5-bc41-4068-875c-ddb2be8f6e20
health: HEALTH_WARN
Reduced data availability: 1 pg inactive, 1 pg incomplete
148 slow ops, oldest one blocked for 59158 sec, osd.0 has slow ops

services:
mon: 3 daemons, quorum dev-petasan01,dev-petasan02,dev-petasan03 (age 18h)
mgr: dev-petasan02(active, since 23h), standbys: dev-petasan01, dev-petasan03
osd: 50 osds: 50 up (since 16h), 50 in (since 18h)

data:
pools:   1 pools, 1024 pgs
objects: 1.16M objects, 3.5 TiB
usage:   5.8 TiB used, 38 TiB / 44 TiB avail
pgs:     0.098% pgs not active
1023 active+clean
1    incomplete

io:
client:   2.0 MiB/s rd, 42 KiB/s wr, 21 op/s rd, 16 op/s wr

and osd dump:

root@petasan01:~# ceph osd dump
epoch 3840
fsid 644e37d5-bc41-4068-875c-ddb2be8f6e20
created 2019-07-05 06:20:17.113972
modified 2020-11-03 07:55:49.544837
flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit
crush_version 288
full_ratio 0.95
backfillfull_ratio 0.9
nearfull_ratio 0.85
require_min_compat_client jewel
min_compat_client jewel
require_osd_release nautilus
pool 4 'rdb' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1024 pgp_num 1024 autoscale_mode warn last_change 3840 flags hashpspool,selfmanaged_snaps stripe_width 0 compression_algorithm none compression_mode none application rbd
removed_snaps [1~3]
max_osd 60
osd.0 up   in  weight 1 up_from 3834 up_thru 3834 down_at 3832 last_clean_interval [3824,3831) [v2:10.0.1.11:6850/258531,v1:10.0.1.11:6851/258531] [v2:10.0.1.11:6854/258531,v1:10.0.1.11:6857/258531] exists,up a5c27cd3-6782-47b2-9e60-e85160802548
osd.1 up   in  weight 1 up_from 3626 up_thru 3819 down_at 3623 last_clean_interval [3167,3622) [v2:10.0.1.11:6852/2949,v1:10.0.1.11:6855/2949] [v2:10.0.1.11:6859/2949,v1:10.0.1.11:6862/2949] exists,up a3b60125-fad5-404d-b47f-fef71cc2cf54
osd.2 up   in  weight 1 up_from 3626 up_thru 3819 down_at 3623 last_clean_interval [3166,3622) [v2:10.0.1.11:6810/2948,v1:10.0.1.11:6812/2948] [v2:10.0.1.11:6814/2948,v1:10.0.1.11:6815/2948] exists,up 509b3e67-e8a5-42f7-9dd4-8d878e8617e0
osd.3 up   in  weight 1 up_from 3628 up_thru 3819 down_at 3623 last_clean_interval [3166,3622) [v2:10.0.1.11:6853/2951,v1:10.0.1.11:6856/2951] [v2:10.0.1.11:6860/2951,v1:10.0.1.11:6863/2951] exists,up 597dc432-209d-4b72-bdf7-8c7cf7bcaf4b
osd.4 up   in  weight 1 up_from 3627 up_thru 3819 down_at 3623 last_clean_interval [3167,3622) [v2:10.0.1.11:6832/2950,v1:10.0.1.11:6835/2950] [v2:10.0.1.11:6836/2950,v1:10.0.1.11:6839/2950] exists,up 9b665d17-e4d4-4950-925b-f983f8f26cb0
osd.5 up   in  weight 1 up_from 3627 up_thru 3819 down_at 3623 last_clean_interval [3167,3622) [v2:10.0.1.11:6811/2953,v1:10.0.1.11:6813/2953] [v2:10.0.1.11:6816/2953,v1:10.0.1.11:6818/2953] exists,up b1c3af24-c028-4279-a4ea-a94c0d0a38f4
osd.6 up   in  weight 1 up_from 3626 up_thru 3819 down_at 3623 last_clean_interval [3167,3622) [v2:10.0.1.11:6873/2954,v1:10.0.1.11:6875/2954] [v2:10.0.1.11:6876/2954,v1:10.0.1.11:6877/2954] exists,up f20b3a72-3b37-4dee-9c9d-90170e881f8b
osd.8 up   in  weight 1 up_from 3627 up_thru 3819 down_at 3623 last_clean_interval [3166,3622) [v2:10.0.1.11:6802/2952,v1:10.0.1.11:6803/2952] [v2:10.0.1.11:6804/2952,v1:10.0.1.11:6805/2952] exists,up 4dc9d0c9-1309-40cc-8c93-86d120733688
osd.9 up   in  weight 1 up_from 3625 up_thru 3819 down_at 3623 last_clean_interval [3167,3622) [v2:10.0.1.11:6822/2955,v1:10.0.1.11:6827/2955] [v2:10.0.1.11:6830/2955,v1:10.0.1.11:6833/2955] exists,up 0ef67167-4511-4019-9457-1e6824f0d829
osd.10 up   in  weight 1 up_from 3627 up_thru 3819 down_at 3623 last_clean_interval [3166,3622) [v2:10.0.1.11:6823/2947,v1:10.0.1.11:6828/2947] [v2:10.0.1.11:6831/2947,v1:10.0.1.11:6834/2947] exists,up 44890a7d-a720-4954-8c90-80a0ecdefab5
osd.11 up   in  weight 1 up_from 3187 up_thru 3822 down_at 3183 last_clean_interval [3125,3182) [v2:10.0.1.12:6842/2974,v1:10.0.1.12:6844/2974] [v2:10.0.1.12:6845/2974,v1:10.0.1.12:6847/2974] exists,up 5538d96e-7706-4086-bb81-a395cb802f65
osd.12 up   in  weight 1 up_from 3188 up_thru 3834 down_at 3183 last_clean_interval [3125,3182) [v2:10.0.1.12:6866/2969,v1:10.0.1.12:6868/2969] [v2:10.0.1.12:6870/2969,v1:10.0.1.12:6871/2969] exists,up a5f4449d-6110-4767-a225-b8522c46dc7e
osd.13 up   in  weight 1 up_from 3185 up_thru 3832 down_at 3183 last_clean_interval [3124,3182) [v2:10.0.1.12:6858/2976,v1:10.0.1.12:6859/2976] [v2:10.0.1.12:6860/2976,v1:10.0.1.12:6861/2976] exists,up 49efeb59-e3d0-48f0-aacd-840bac9f7915
osd.14 up   in  weight 1 up_from 3187 up_thru 3824 down_at 3183 last_clean_interval [3124,3182) [v2:10.0.1.12:6834/2971,v1:10.0.1.12:6835/2971] [v2:10.0.1.12:6836/2971,v1:10.0.1.12:6837/2971] exists,up f0eb6038-9318-4097-a592-f94853475b0a
osd.15 up   in  weight 1 up_from 3187 up_thru 3834 down_at 3183 last_clean_interval [3125,3182) [v2:10.0.1.12:6802/2979,v1:10.0.1.12:6803/2979] [v2:10.0.1.12:6804/2979,v1:10.0.1.12:6805/2979] exists,up d8751f3a-8809-4ada-ab7b-8aa7c64b5ca7
osd.16 up   in  weight 1 up_from 3185 up_thru 3834 down_at 3183 last_clean_interval [3124,3182) [v2:10.0.1.12:6814/2973,v1:10.0.1.12:6817/2973] [v2:10.0.1.12:6820/2973,v1:10.0.1.12:6823/2973] exists,up 72bad51e-82dd-467d-a81f-d55b41d33c62
osd.17 up   in  weight 1 up_from 3185 up_thru 3834 down_at 3183 last_clean_interval [3124,3182) [v2:10.0.1.12:6811/2972,v1:10.0.1.12:6813/2972] [v2:10.0.1.12:6816/2972,v1:10.0.1.12:6818/2972] exists,up 42a1e430-2723-40d6-a124-3767d850a54b
osd.18 up   in  weight 1 up_from 3187 up_thru 3819 down_at 3183 last_clean_interval [3124,3182) [v2:10.0.1.12:6867/2970,v1:10.0.1.12:6869/2970] [v2:10.0.1.12:6872/2970,v1:10.0.1.12:6874/2970] exists,up 637940b5-e687-489c-8b95-f1816b2c3557
osd.19 up   in  weight 1 up_from 3185 up_thru 3834 down_at 3183 last_clean_interval [3125,3182) [v2:10.0.1.12:6810/2977,v1:10.0.1.12:6812/2977] [v2:10.0.1.12:6815/2977,v1:10.0.1.12:6819/2977] exists,up a448fc93-aad4-4f42-86b1-261473be96d0
osd.20 up   in  weight 1 up_from 3186 up_thru 3834 down_at 3183 last_clean_interval [3125,3182) [v2:10.0.1.12:6843/2978,v1:10.0.1.12:6846/2978] [v2:10.0.1.12:6848/2978,v1:10.0.1.12:6850/2978] exists,up e6305397-0623-42d9-b943-be0145a5a9a4
osd.21 up   in  weight 1 up_from 3670 up_thru 3832 down_at 3667 last_clean_interval [3595,3666) [v2:10.0.1.13:6811/2994,v1:10.0.1.13:6818/2994] [v2:10.0.1.13:6826/2994,v1:10.0.1.13:6834/2994] exists,up d02dd2fb-ac12-4614-94fb-c914bde49978
osd.22 up   in  weight 1 up_from 3670 up_thru 3819 down_at 3667 last_clean_interval [3595,3666) [v2:10.0.1.13:6821/2990,v1:10.0.1.13:6829/2990] [v2:10.0.1.13:6840/2990,v1:10.0.1.13:6852/2990] exists,up c3ad5a14-0c5e-41af-966f-73b35472232e
osd.23 up   in  weight 1 up_from 3670 up_thru 3834 down_at 3667 last_clean_interval [3596,3666) [v2:10.0.1.13:6802/2996,v1:10.0.1.13:6803/2996] [v2:10.0.1.13:6804/2996,v1:10.0.1.13:6806/2996] exists,up b8d4efb9-bf1f-4364-ba5d-90ec5b26a717
osd.24 up   in  weight 1 up_from 3670 up_thru 3832 down_at 3667 last_clean_interval [3596,3666) [v2:10.0.1.13:6817/2991,v1:10.0.1.13:6824/2991] [v2:10.0.1.13:6831/2991,v1:10.0.1.13:6839/2991] exists,up 03d8f93a-204f-4827-b6d1-b691b815b48d
osd.25 up   in  weight 1 up_from 3670 up_thru 3819 down_at 3667 last_clean_interval [3595,3666) [v2:10.0.1.13:6830/2993,v1:10.0.1.13:6838/2993] [v2:10.0.1.13:6845/2993,v1:10.0.1.13:6854/2993] exists,up 7a068321-cced-48fe-8225-44e32ac7146e
osd.26 up   in  weight 1 up_from 3669 up_thru 3834 down_at 3667 last_clean_interval [3595,3666) [v2:10.0.1.13:6813/2995,v1:10.0.1.13:6816/2995] [v2:10.0.1.13:6822/2995,v1:10.0.1.13:6832/2995] exists,up b65ef557-8b29-4cd2-8fe7-0830be0d0a18
osd.27 up   in  weight 1 up_from 3669 up_thru 3824 down_at 3667 last_clean_interval [3596,3666) [v2:10.0.1.13:6874/2992,v1:10.0.1.13:6875/2992] [v2:10.0.1.13:6876/2992,v1:10.0.1.13:6877/2992] exists,up bc787a3a-5753-476a-a9ec-810f79f0f5a3
osd.28 up   in  weight 1 up_from 3669 up_thru 3822 down_at 3667 last_clean_interval [3596,3666) [v2:10.0.1.13:6810/2997,v1:10.0.1.13:6819/2997] [v2:10.0.1.13:6828/2997,v1:10.0.1.13:6837/2997] exists,up c1ffe1e9-a439-4551-b705-f9a37e53cbdf
osd.29 up   in  weight 1 up_from 3669 up_thru 3834 down_at 3667 last_clean_interval [3594,3666) [v2:10.0.1.13:6812/2988,v1:10.0.1.13:6820/2988] [v2:10.0.1.13:6827/2988,v1:10.0.1.13:6836/2988] exists,up a16b32b2-0be3-4803-ae48-7f6104528b72
osd.30 up   in  weight 1 up_from 3670 up_thru 3824 down_at 3667 last_clean_interval [3595,3666) [v2:10.0.1.13:6805/2989,v1:10.0.1.13:6807/2989] [v2:10.0.1.13:6809/2989,v1:10.0.1.13:6815/2989] exists,up 982d6f78-43ba-4173-9159-4e1c083ac3f3
osd.31 up   in  weight 1 up_from 3150 up_thru 3832 down_at 3148 last_clean_interval [3048,3147) [v2:10.0.1.14:6804/2733,v1:10.0.1.14:6811/2733] [v2:10.0.1.14:6819/2733,v1:10.0.1.14:6826/2733] exists,up 7217b228-0e4d-4027-8d1e-fc230c3ebdc3
osd.32 up   in  weight 1 up_from 3150 up_thru 3834 down_at 3148 last_clean_interval [3048,3147) [v2:10.0.1.14:6800/2737,v1:10.0.1.14:6803/2737] [v2:10.0.1.14:6808/2737,v1:10.0.1.14:6815/2737] exists,up 79d0e3d3-e23b-4439-8364-4e79ea6c3ed5
osd.33 up   in  weight 1 up_from 3152 up_thru 3834 down_at 3148 last_clean_interval [3049,3147) [v2:10.0.1.14:6807/2729,v1:10.0.1.14:6814/2729] [v2:10.0.1.14:6821/2729,v1:10.0.1.14:6828/2729] exists,up 0d177ac5-58c6-4e04-91f5-d3c676402322
osd.34 up   in  weight 1 up_from 3152 up_thru 3834 down_at 3148 last_clean_interval [3048,3147) [v2:10.0.1.14:6863/2735,v1:10.0.1.14:6865/2735] [v2:10.0.1.14:6866/2735,v1:10.0.1.14:6867/2735] exists,up 97476a60-28aa-4f8c-8b4b-a94db029aafb
osd.35 up   in  weight 1 up_from 3151 up_thru 3834 down_at 3148 last_clean_interval [3048,3147) [v2:10.0.1.14:6805/2715,v1:10.0.1.14:6813/2715] [v2:10.0.1.14:6824/2715,v1:10.0.1.14:6832/2715] exists,up 7d7fe12f-2ffd-414e-bfef-30d3a59f725a
osd.36 up   in  weight 1 up_from 3151 up_thru 3824 down_at 3148 last_clean_interval [3048,3147) [v2:10.0.1.14:6871/2734,v1:10.0.1.14:6873/2734] [v2:10.0.1.14:6874/2734,v1:10.0.1.14:6875/2734] exists,up 9a33b677-d532-4f05-94e7-51bd7676a327
osd.41 up   in  weight 1 up_from 3150 up_thru 3834 down_at 3148 last_clean_interval [3049,3147) [v2:10.0.1.14:6801/2732,v1:10.0.1.14:6806/2732] [v2:10.0.1.14:6812/2732,v1:10.0.1.14:6820/2732] exists,up edfc8014-5796-488b-b6a6-47a3f9ad8324
osd.43 up   in  weight 1 up_from 3151 up_thru 3824 down_at 3148 last_clean_interval [3048,3147) [v2:10.0.1.14:6802/2736,v1:10.0.1.14:6809/2736] [v2:10.0.1.14:6818/2736,v1:10.0.1.14:6831/2736] exists,up 9bd66b36-e1b1-4fef-9714-2fb9a2fac61c
osd.48 up   in  weight 1 up_from 3150 up_thru 3834 down_at 3148 last_clean_interval [3048,3147) [v2:10.0.1.14:6816/2738,v1:10.0.1.14:6825/2738] [v2:10.0.1.14:6834/2738,v1:10.0.1.14:6845/2738] exists,up 3b3baa1f-3525-4700-ae5e-ea4cc9560423
osd.49 up   in  weight 1 up_from 3152 up_thru 3834 down_at 3148 last_clean_interval [3048,3147) [v2:10.0.1.14:6810/2739,v1:10.0.1.14:6817/2739] [v2:10.0.1.14:6823/2739,v1:10.0.1.14:6830/2739] exists,up 483e0884-6bba-4f16-bb4c-e30b704c2908
osd.50 up   in  weight 1 up_from 3139 up_thru 3819 down_at 3134 last_clean_interval [3027,3133) [v2:10.0.1.15:6801/2715,v1:10.0.1.15:6803/2715] [v2:10.0.1.15:6807/2715,v1:10.0.1.15:6812/2715] exists,up 3c75e7b7-6185-4b07-a8bb-5a19801d92a1
osd.51 up   in  weight 1 up_from 3137 up_thru 3832 down_at 3134 last_clean_interval [3025,3133) [v2:10.0.1.15:6800/2718,v1:10.0.1.15:6804/2718] [v2:10.0.1.15:6808/2718,v1:10.0.1.15:6813/2718] exists,up 2957c684-db23-41df-b83a-a3f8832169ab
osd.52 up   in  weight 1 up_from 3139 up_thru 3832 down_at 3134 last_clean_interval [3025,3133) [v2:10.0.1.15:6868/2719,v1:10.0.1.15:6870/2719] [v2:10.0.1.15:6872/2719,v1:10.0.1.15:6874/2719] exists,up 0151b888-c04f-4ae3-aa34-8b9809839bce
osd.53 up   in  weight 1 up_from 3139 up_thru 3834 down_at 3134 last_clean_interval [3025,3133) [v2:10.0.1.15:6848/2713,v1:10.0.1.15:6850/2713] [v2:10.0.1.15:6852/2713,v1:10.0.1.15:6855/2713] exists,up 64ddf716-b143-4b06-acb9-0773d704d55f
osd.54 up   in  weight 1 up_from 3136 up_thru 3822 down_at 3134 last_clean_interval [3025,3133) [v2:10.0.1.15:6864/2702,v1:10.0.1.15:6865/2702] [v2:10.0.1.15:6866/2702,v1:10.0.1.15:6867/2702] exists,up 723181fe-f366-41dd-bc81-23f6a749fabf
osd.55 up   in  weight 1 up_from 3140 up_thru 3824 down_at 3134 last_clean_interval [3027,3133) [v2:10.0.1.15:6849/2714,v1:10.0.1.15:6851/2714] [v2:10.0.1.15:6853/2714,v1:10.0.1.15:6854/2714] exists,up 70dd6026-2c57-4da0-a685-152d632420be
osd.56 up   in  weight 1 up_from 3138 up_thru 3824 down_at 3134 last_clean_interval [3025,3133) [v2:10.0.1.15:6840/2712,v1:10.0.1.15:6841/2712] [v2:10.0.1.15:6842/2712,v1:10.0.1.15:6843/2712] exists,up aa4b5cd3-2636-40dd-9a30-90f754306ae2
osd.57 up   in  weight 1 up_from 3137 up_thru 3834 down_at 3134 last_clean_interval [3027,3133) [v2:10.0.1.15:6809/2717,v1:10.0.1.15:6818/2717] [v2:10.0.1.15:6825/2717,v1:10.0.1.15:6830/2717] exists,up c4953cce-8940-4886-8d0b-3b247bca9195
osd.58 up   in  weight 1 up_from 3137 up_thru 3832 down_at 3134 last_clean_interval [3025,3133) [v2:10.0.1.15:6805/2711,v1:10.0.1.15:6810/2711] [v2:10.0.1.15:6814/2711,v1:10.0.1.15:6820/2711] exists,up 4fd262af-70ec-4360-9e62-8495d9386070
osd.59 up   in  weight 1 up_from 3138 up_thru 3834 down_at 3134 last_clean_interval [3025,3133) [v2:10.0.1.15:6802/2716,v1:10.0.1.15:6806/2716] [v2:10.0.1.15:6811/2716,v1:10.0.1.15:6816/2716] exists,up 4bb3695f-c91a-4705-8e2b-a539807a28a3

Last edited on November 3, 2020, 7:27 am by Andoni · #1

admin
2,930 Posts
November 3, 2020, 8:22 am

This is a tougher Ceph error, i would recommend you search online on the Ceph forums and mailing list. It could be related to size=2/min_size=1 and no power loss protection if using consumer SSDs or controller with cache with no battery backing. Good luck.

#2