Nodes shutting down

khopkins
96 Posts

July 11, 2018, 3:56 pm

My mistake, it is disabled

modinfo bnx2
filename: /lib/modules/4.4.92-09-petasan/kernel/drivers/net/ethernet/broadcom/bnx2.ko
firmware: bnx2/bnx2-rv2p-09ax-6.0.17.fw
firmware: bnx2/bnx2-rv2p-09-6.0.17.fw
firmware: bnx2/bnx2-mips-09-6.2.1b.fw
firmware: bnx2/bnx2-rv2p-06-6.0.15.fw
firmware: bnx2/bnx2-mips-06-6.2.3.fw
version: 2.2.6
license: GPL
description: QLogic BCM5706/5708/5709/5716 Driver
author: Michael Chan <mchan@broadcom.com>
srcversion: DAEFDB682746C4E3AE27475
alias: pci:v000014E4d0000163Csv*sd*bc*sc*i*
alias: pci:v000014E4d0000163Bsv*sd*bc*sc*i*
alias: pci:v000014E4d0000163Asv*sd*bc*sc*i*
alias: pci:v000014E4d00001639sv*sd*bc*sc*i*
alias: pci:v000014E4d000016ACsv*sd*bc*sc*i*
alias: pci:v000014E4d000016AAsv*sd*bc*sc*i*
alias: pci:v000014E4d000016AAsv0000103Csd00003102bc*sc*i*
alias: pci:v000014E4d0000164Csv*sd*bc*sc*i*
alias: pci:v000014E4d0000164Asv*sd*bc*sc*i*
alias: pci:v000014E4d0000164Asv0000103Csd00003106bc*sc*i*
alias: pci:v000014E4d0000164Asv0000103Csd00003101bc*sc*i*
depends:
intree: Y
vermagic: 4.4.92-09-petasan SMP mod_unload modversions
parm: disable_msi:Disable Message Signaled Interrupt (MSI) (int)
root@PS-NODE-1:/sys/module/bnx2/parameters#

admin
2,930 Posts

July 11, 2018, 4:42 pm

I am not sure it is disabled from your last post. does "cat /sys/module/bnx2/parameters/disable_msi" output 1 ? Also the earlier post with dmesg showing "[ 28.244185] bnx2 0000:01:00.0 eth0: using MSIX" looks suspicious. you can also try putting the param in modprobe.conf:

echo "options bnx2 disable_msi=1" >> /etc/modprobe.conf

khopkins
96 Posts

July 11, 2018, 5:51 pm

The param was already there in the modprobe.conf.

cat modprobe.conf
options bnx2 disable_msi=1

It was the output of the modinfo that was concerning.

cat /sys/module/bnx2/parameters/disable_msi 0

So can't tell whether its disabled or not.

khopkins
96 Posts

July 11, 2018, 6:08 pm

Ran

update-initramfs -u

If a module is loaded at boot time (before the root filesystem is opened), you need to update the initial ramdisk. 
This is the first "filesystem" that is loaded, it will load drivers, discover and mount partitions, etc.


Then rebooted, output of /sys/module/bnx2/parameters/disable_msi is now 1.

All 3 nodes are done so we'll let it run for a while.  Thanks again.

admin
2,930 Posts

July 11, 2018, 7:25 pm

Excellent..that was a little tricky 🙂 hope things go well from now..

khopkins
96 Posts

July 11, 2018, 7:56 pm

We'll see how it goes, it looks promising now. I think this is a great product you have put together and your help has been outstanding, appreciate all your help.

khopkins
96 Posts

July 16, 2018, 12:58 pm

Here's an update. System crashed this weekend with two nodes turning off at the same time. Same thing with the ethernet NIC's going down. System was rebuilt and option on the NIC was verified, and everything normalized. Unable to find drivers for Broadcom nics for ubuntu, don't understand why they didn't update a most common piece of hardware. Where to go from here?

admin
2,930 Posts

July 17, 2018, 2:53 pm

This is too bad 🙁

The options now are limited:

Make 100% sure the issue is not external from a network switch
Use different Ethernet hardware
Wait for our next kernel version based on SUSE SLE 15 released yesterday, it is based on the 4.12 branch + new firmware. It might (just might) be better.

If you find anything else that might help, let us know.

khopkins
96 Posts

July 17, 2018, 9:52 pm

Can't really change out the NIC's or add a card on the R410's. Everything external has been checked or replaced. I'll wait for the SUSE edition and try that out, glad you went to SUSE.

When do you expect to be able to download it?

Again, thanks for the help.

admin
2,930 Posts

July 23, 2018, 11:48 am

download new kernel and firmware from:

https://drive.google.com/drive/folders/1kZYfW3MAz2fJKBIy57R4dF9h74SCMoNt?usp=sharing

install:

dpkg -i petasan-firmware_20180416.deb
dpkg -i linux-image-4.12.14-02-petasan_amd64.deb