Forums

Home / Forums

You need to log in to create posts and topics. Login · Register

NVMe drives in Dell R640 not an OSD option


hak
23 Posts
July 4, 2023, 4:19 pm

These are a new setup.

  • r640
    • 10x 2.5" drive chassis
    • boot from Dell BOSS card (dual m2 on pci card)
    • 2x 2.5" NVMe drives per host to be used for OSD, 3 dwpd 6.4TB each
    • 2x 1G, 2x 10G (SFP+), 2x 40G (QSFP)

the NVMe drives show up/are listed and recognized in BIOS, but when I get to the first node's storage/role setup page, they are not there.  I've tried rebooting/rebuilding > 1x, different drive slots up font, no difference.

Is there a driver needed?   anyone else running r740 or r640 NVME with the Dell factory PCIe NVMe expander card?

 

#1

admin
2,967 Posts
July 4, 2023, 6:34 pm

do the drives show up in

ls /sys/block/

#2

hak
23 Posts
July 5, 2023, 2:13 am

no.  only sda shows up (which is the BOSS card's raid1 m2's).

#3

admin
2,967 Posts
July 5, 2023, 8:36 am

This means they are not detected by the kernel. Could be a hardware or configuration issue. Can you try installing (or booting live CD) of other OSs like Windows or Ubuntu and see if they are detected.

#4

hak
23 Posts
July 5, 2023, 3:18 pm

Using ubuntu live v22.04.2:

testuser@test-for-nvme-drives:~$ ls -l /dev/block/
total 0
lrwxrwxrwx 1 root root 7 Jul 5 15:04 253:0 -> ../dm-0
lrwxrwxrwx 1 root root 8 Jul 5 15:04 7:0 -> ../loop0
lrwxrwxrwx 1 root root 8 Jul 5 15:04 7:1 -> ../loop1
lrwxrwxrwx 1 root root 8 Jul 5 15:04 7:2 -> ../loop2
lrwxrwxrwx 1 root root 8 Jul 5 15:06 7:3 -> ../loop3
lrwxrwxrwx 1 root root 8 Jul 5 15:06 7:4 -> ../loop4
lrwxrwxrwx 1 root root 8 Jul 5 15:04 7:5 -> ../loop5
lrwxrwxrwx 1 root root 8 Jul 5 15:04 7:6 -> ../loop6
lrwxrwxrwx 1 root root 8 Jul 5 15:04 7:7 -> ../loop7
lrwxrwxrwx 1 root root 6 Jul 5 15:04 8:0 -> ../sda
lrwxrwxrwx 1 root root 7 Jul 5 15:04 8:1 -> ../sda1
lrwxrwxrwx 1 root root 7 Jul 5 15:04 8:2 -> ../sda2
lrwxrwxrwx 1 root root 7 Jul 5 15:04 8:3 -> ../sda3
testuser@test-for-nvme-drives:~$

 

testuser@test-for-nvme-drives:~$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
loop0 7:0 0 63.3M 1 loop /snap/core20/1822
loop1 7:1 0 111.9M 1 loop /snap/lxd/24322
loop2 7:2 0 49.8M 1 loop /snap/snapd/18357
loop3 7:3 0 73.9M 1 loop /snap/core22/766
loop4 7:4 0 5M 1 loop /snap/stress-ng/7548
sda 8:0 0 119.2G 0 disk
├─sda1 8:1 0 1M 0 part
├─sda2 8:2 0 2G 0 part /boot
└─sda3 8:3 0 117.2G 0 part
└─ubuntu--vg-ubuntu--lv 253:0 0 58.6G 0 lvm /
testuser@test-for-nvme-drives:~$

 

i'll try vsphere next.

Last edited on July 5, 2023, 3:21 pm by hak · #5

hak
23 Posts
July 5, 2023, 7:03 pm

in vsphere they are found and ID'd properly, but i can't seem to use them... (ie: can't add them as a device)

https://drive.google.com/file/d/18ZJ-YbO9IxtojFW8g-R-KAHO9icrodqr/view?usp=sharing

These were new-old-stock 0-hour drives... I'm wondering if they're encrypted....  would they fail to load in /sys/block if they were encrypted?

#6

hak
23 Posts
July 6, 2023, 2:15 am

in windows, they're in purgatory as well, 'seen' but not fully by the OS and i cannot initialize either of them

https://drive.google.com/file/d/1SmQb950RNcOCbjtCDR7tBMtu3UHgdHQ9/view?usp=sharing

I have 5 other nvme drives arriving tomorrow, one for each host. I will try again with these new (Micron Max 6.4TB) and report back.

#7

hak
23 Posts
July 6, 2023, 4:26 pm

it's the drives.  with new Micron MAX it showed up just fine:

https://drive.google.com/file/d/12W_cJsQPS1G5ivN1EfIJzeVxQJKOhw77/view?usp=sharing

working with other vendor to correct/swap the original 10 drives.

#8

admin
2,967 Posts
July 6, 2023, 7:07 pm

Great 🙂 thanks for the feedback

#9