Forums

Home / Forums

You need to log in to create posts and topics. Login · Register

General Questions

Pages: 1 2

rickbharper
11 Posts
September 26, 2017, 11:19 pm

root@psNode01:/etc/ceph# ceph-disk activate /dev/sdb1
ceph-disk: Cannot discover filesystem type: device /dev/sdb1: Command '/sbin/blkid' returned non-zero exit status 8

I'll try swapping the problem disk tomorrow (I'm going home for the evening)

Thanks for the help!

#11

rickbharper
11 Posts
September 27, 2017, 4:01 pm

Ok... I swapped out 3 of the 4 disks this morning and the new disks added to the cluster with no issues...

I still have one disk in node03 that won't activate if you want me to test anything else...

#12

admin
2,930 Posts
September 27, 2017, 6:10 pm

Great you got it working with other disks 🙂

while looking at ceph-disk i did find a -v verbose flag that i overlooked which will print a lot of info

ceph-disk -v prepare --cluster CLUSTER  --zap-disk --fs-type xfs /dev/sdX

it will be interesting if you can run it on the last disk before you replace it and post the output so we have an idea what the issue was.

Last please remember to un-comment the 2 lines in the config file which we changed yesterday.

#13

rickbharper
11 Posts
September 27, 2017, 7:15 pm

Here is the output of the verbose disk prepare:

root@psNode03:~# ceph-disk -v prepare --cluster psCluster01 --zap-disk --fs-type xfs /dev/sdc
command: Running command: /usr/bin/ceph-osd --cluster=psCluster01 --show-config-value=fsid
command: Running command: /usr/bin/ceph-osd --check-allows-journal -i 0 --cluster psCluster01 --setuser ceph --setgroup ceph
command: Running command: /usr/bin/ceph-osd --check-wants-journal -i 0 --cluster psCluster01 --setuser ceph --setgroup ceph
command: Running command: /usr/bin/ceph-osd --check-needs-journal -i 0 --cluster psCluster01 --setuser ceph --setgroup ceph
get_dm_uuid: get_dm_uuid /dev/sdc uuid path is /sys/dev/block/8:32/dm/uuid
set_type: Will colocate journal with data on /dev/sdc
command: Running command: /usr/bin/ceph-osd --cluster=psCluster01 --show-config-value=osd_journal_size
get_dm_uuid: get_dm_uuid /dev/sdc uuid path is /sys/dev/block/8:32/dm/uuid
get_dm_uuid: get_dm_uuid /dev/sdc uuid path is /sys/dev/block/8:32/dm/uuid
get_dm_uuid: get_dm_uuid /dev/sdc uuid path is /sys/dev/block/8:32/dm/uuid
command: Running command: /usr/bin/ceph-conf --cluster=psCluster01 --name=osd. --lookup osd_mkfs_options_xfs
command: Running command: /usr/bin/ceph-conf --cluster=psCluster01 --name=osd. --lookup osd_mount_options_xfs
get_dm_uuid: get_dm_uuid /dev/sdc uuid path is /sys/dev/block/8:32/dm/uuid
zap: Zapping partition table on /dev/sdc
command_check_call: Running command: /sbin/sgdisk --zap-all -- /dev/sdc
Creating new GPT entries.
GPT data structures destroyed! You may now partition the disk using fdisk or
other utilities.
command_check_call: Running command: /sbin/sgdisk --clear --mbrtogpt -- /dev/sdc
Creating new GPT entries.
The operation has completed successfully.
update_partition: Calling partprobe on zapped device /dev/sdc
command_check_call: Running command: /sbin/udevadm settle --timeout=600
command: Running command: /usr/bin/flock -s /dev/sdc /sbin/partprobe /dev/sdc
command_check_call: Running command: /sbin/udevadm settle --timeout=600
get_dm_uuid: get_dm_uuid /dev/sdc uuid path is /sys/dev/block/8:32/dm/uuid
get_dm_uuid: get_dm_uuid /dev/sdc uuid path is /sys/dev/block/8:32/dm/uuid
ptype_tobe_for_name: name = journal
get_dm_uuid: get_dm_uuid /dev/sdc uuid path is /sys/dev/block/8:32/dm/uuid
create_partition: Creating journal partition num 2 size 5120 on /dev/sdc
command_check_call: Running command: /sbin/sgdisk --new=2:0:+5120M --change-name=2:ceph journal --partition-guid=2:39ac761b-55ac-4a76-becd-07c297dc3196 --typecode=2:45b0969e-9b03-4f30-b4c6-b4b80ceff106 --mbrtogpt -- /dev/sdc
Setting name!
partNum is 1
REALLY setting name!
The operation has completed successfully.
update_partition: Calling partprobe on created device /dev/sdc
command_check_call: Running command: /sbin/udevadm settle --timeout=600
command: Running command: /usr/bin/flock -s /dev/sdc /sbin/partprobe /dev/sdc
command_check_call: Running command: /sbin/udevadm settle --timeout=600
get_dm_uuid: get_dm_uuid /dev/sdc uuid path is /sys/dev/block/8:32/dm/uuid
get_dm_uuid: get_dm_uuid /dev/sdc uuid path is /sys/dev/block/8:32/dm/uuid
get_dm_uuid: get_dm_uuid /dev/sdc2 uuid path is /sys/dev/block/8:34/dm/uuid
prepare_device: Journal is GPT partition /dev/disk/by-partuuid/39ac761b-55ac-4a76-becd-07c297dc3196
prepare_device: Journal is GPT partition /dev/disk/by-partuuid/39ac761b-55ac-4a76-becd-07c297dc3196
get_dm_uuid: get_dm_uuid /dev/sdc uuid path is /sys/dev/block/8:32/dm/uuid
set_data_partition: Creating osd partition on /dev/sdc
get_dm_uuid: get_dm_uuid /dev/sdc uuid path is /sys/dev/block/8:32/dm/uuid
ptype_tobe_for_name: name = data
get_dm_uuid: get_dm_uuid /dev/sdc uuid path is /sys/dev/block/8:32/dm/uuid
create_partition: Creating data partition num 1 size 0 on /dev/sdc
command_check_call: Running command: /sbin/sgdisk --largest-new=1 --change-name=1:ceph data --partition-guid=1:c9e3741d-2be5-4815-bd94-cbaaf9fdea19 --typecode=1:89c57f98-2fe5-4dc0-89c1-f3ad0ceff2be --mbrtogpt -- /dev/sdc
Setting name!
partNum is 0
REALLY setting name!
The operation has completed successfully.
update_partition: Calling partprobe on created device /dev/sdc
command_check_call: Running command: /sbin/udevadm settle --timeout=600
command: Running command: /usr/bin/flock -s /dev/sdc /sbin/partprobe /dev/sdc
command_check_call: Running command: /sbin/udevadm settle --timeout=600
get_dm_uuid: get_dm_uuid /dev/sdc uuid path is /sys/dev/block/8:32/dm/uuid
get_dm_uuid: get_dm_uuid /dev/sdc uuid path is /sys/dev/block/8:32/dm/uuid
get_dm_uuid: get_dm_uuid /dev/sdc1 uuid path is /sys/dev/block/8:33/dm/uuid
populate_data_path_device: Creating xfs fs on /dev/sdc1
command_check_call: Running command: /sbin/mkfs -t xfs -f -i size=2048 -f -- /dev/sdc1
meta-data=/dev/sdc1 isize=2048 agcount=4, agsize=30196417 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=0
data = bsize=4096 blocks=120785665, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=58977, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
mount: Mounting /dev/sdc1 on /var/lib/ceph/tmp/mnt.pd5hJp with options rw,noatime,inode64,logbsize=256k,allocsize=4M
command_check_call: Running command: /bin/mount -t xfs -o rw,noatime,inode64,logbsize=256k,allocsize=4M -- /dev/sdc1 /var/lib/ceph/tmp/mnt.pd5hJp
populate_data_path: Preparing osd data dir /var/lib/ceph/tmp/mnt.pd5hJp
command: Running command: /bin/chown -R ceph:ceph /var/lib/ceph/tmp/mnt.pd5hJp/ceph_fsid.289852.tmp
command: Running command: /bin/chown -R ceph:ceph /var/lib/ceph/tmp/mnt.pd5hJp/fsid.289852.tmp
command: Running command: /bin/chown -R ceph:ceph /var/lib/ceph/tmp/mnt.pd5hJp/magic.289852.tmp
command: Running command: /bin/chown -R ceph:ceph /var/lib/ceph/tmp/mnt.pd5hJp/journal_uuid.289852.tmp
adjust_symlink: Creating symlink /var/lib/ceph/tmp/mnt.pd5hJp/journal -> /dev/disk/by-partuuid/39ac761b-55ac-4a76-becd-07c297dc3196
command: Running command: /bin/chown -R ceph:ceph /var/lib/ceph/tmp/mnt.pd5hJp
unmount: Unmounting /var/lib/ceph/tmp/mnt.pd5hJp
command_check_call: Running command: /bin/umount -- /var/lib/ceph/tmp/mnt.pd5hJp
get_dm_uuid: get_dm_uuid /dev/sdc uuid path is /sys/dev/block/8:32/dm/uuid
command_check_call: Running command: /sbin/sgdisk --typecode=1:4fbd7e29-9d25-41b8-afd0-062c0ceff05d -- /dev/sdc
The operation has completed successfully.
update_partition: Calling partprobe on prepared device /dev/sdc
command_check_call: Running command: /sbin/udevadm settle --timeout=600
command: Running command: /usr/bin/flock -s /dev/sdc /sbin/partprobe /dev/sdc
command_check_call: Running command: /sbin/udevadm settle --timeout=600
command_check_call: Running command: /sbin/udevadm trigger --action=add --sysname-match sdc1

As before the disk is still not added to the cluster:

root@psNode03:~# ceph osd tree --cluster psCluster01
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
-1 3.59753 root default
-2 0.89938 host psNode03
0 0.44969 osd.0 up 1.00000 1.00000
1 0.44969 osd.1 up 1.00000 1.00000
-3 1.34908 host psNode02
2 0.44969 osd.2 up 1.00000 1.00000
3 0.44969 osd.3 up 1.00000 1.00000
4 0.44969 osd.4 up 1.00000 1.00000
-5 1.34908 host psNode01
10 0.44969 osd.10 up 1.00000 1.00000
11 0.44969 osd.11 up 1.00000 1.00000
5 0.44969 osd.5 up 1.00000 1.00000

root@psNode03:~# ceph-disk list
/dev/rbd0 :
/dev/rbd0p1 other, ntfs
/dev/rbd1 other, unknown
/dev/sda :
/dev/sda2 other, ext4, mounted on /
/dev/sda1 other, ext4, mounted on /boot
/dev/sda4 other, ext4, mounted on /opt/petasan/config
/dev/sda3 other, ext4, mounted on /var/lib/ceph
/dev/sdb :
/dev/sdb2 ceph journal, for /dev/sdb1
/dev/sdb1 ceph data, active, cluster psCluster01, osd.0, journal /dev/sdb2
/dev/sdc :
/dev/sdc1 other
/dev/sdc2 ceph journal
/dev/sdd :
/dev/sdd2 ceph journal, for /dev/sdd1
/dev/sdd1 ceph data, active, cluster psCluster01, osd.1, journal /dev/sdd2
/dev/sr0 other, unknown
/dev/sr1 other, unknown

#14

admin
2,930 Posts
September 27, 2017, 10:32 pm

The verbose output shows everything passing. However i suspect the following maybe be happening: in normal case ceph-disk creates a data partition on your first disk partition and initial assigns it a gpt partition type of 89c57f98-2fe5-4dc0-89c1-f3ad0ceff2be meaning it is an osd in progress and then when done preparing the osd it renames the partition type to 4fbd7e29-9d25-41b8-afd0-062c0ceff05d. This is done so in case of failure the partition is not left with a valid osd partition type.

Maybe upon completion, ceph-disk tried to rename the partition type to 4fbd7e29-9d25-41b8-afd0-062c0ceff05d and got a success return but it did not actually happen...maybe a timing issue with the disk in renaming the partition twice in succession too quickly for the disk.

To see if this is correct, get the partition type of the failed disk ( specify /dev/sdc1 partition )
blkid -o udev -p /dev/sdc1 | grep ID_PART_ENTRY_TYPE

if this shows 4fbd7e29-9d25-41b8-afd0-062c0ceff05d then my guess is wrong.

if this shows 89c57f98-2fe5-4dc0-89c1-f3ad0ceff2be then this is indeed the problem and could probably be fixed as follows:

Manually change the type for first partition ( specify /dev/sdc disk )
/sbin/sgdisk --typecode=1:4fbd7e29-9d25-41b8-afd0-062c0ceff05d -- /dev/sdc

Update kenrnel partition table ( specify /dev/sdc disk )
partprobe /dev/sdc

Now try to activate the osd
ceph-disk activate /dev/sdc1

Last edited on September 27, 2017, 10:36 pm by admin · #15

rickbharper
11 Posts
September 27, 2017, 11:20 pm

The filtered blkid command came back blank...

The entire output is one line as follows:

root@psNode03:~# blkid -o udev -p /dev/sdc1
ID_FS_AMBIVALENT=filesystem:xfs raid:zfs_member:5000

I don't know if helps at all, but I believe all 4 of the disks that failed to attach were previously part of zfs pool on FreeBSD - I cleared the gpt and mbr records using gdisk and zap so they should have been 'clean' but maybe there's something there?

Last edited on September 27, 2017, 11:23 pm by rickbharper · #16

admin
2,930 Posts
September 28, 2017, 12:00 am

looks like the same issue of old zfs metadata

http://tracker.ceph.com/issues/19248

 

 

Last edited on September 28, 2017, 12:01 am by admin · #17

rickbharper
11 Posts
September 28, 2017, 2:08 am

Ok.. finally some progress!

After using gdisk - zap-all to wipe the disk I realized that the output of ceph-disk list was listing the disk as a zfs disk...

I tried using dd to wipe out of the first 100 sectors of the disk (probably overkill but I wanted to make sure)

dd if=/dev/zero of=/dev/sdc bs=512 count=100

 At this point ceph-disk list still showed the disk as a zfs disk - time for some more reading - turns out that zfs writes metadata to both the start and end sectors of a disk...

I used fdisk to find the sector count:

root@psNode03:~# fdisk -l /dev/sdc
Disk /dev/sdc: 465.8 GiB, 500107862016 bytes, 976773168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 98AE0AD3-352A-4341-A706-897D7AF063C8

Then I used dd to zero out the end of the disk - again I went with the last 100 sectors

dd if=/dev/zero of=/dev/sdc bs=512 count=100 seek=976773068   [the total number of sectors minus the 100 I'm writing to]

At this point ceph-disk list no longer showed the zfs tag and the disk added with no issues

 

Thanks for all of the help!

#18
Pages: 1 2