Error re-assigning Link names in PetaSAN 3.0.x (Ubuntu 20.04)
podilarius
44 Posts
December 30, 2021, 2:20 pmQuote from podilarius on December 30, 2021, 2:20 pmProblem Description:
After upgrading or installing PetaSAN version 3.0.x based on Ubuntu 20.04, the custom NIC assignments stopped working.
Meaning that if I re-assign eth0 with eth2, this doesn't stick.
The error message is:
eth0: Failed to rename network interface 2 from 'eth0' to 'eth2': File exists
Indicating that it is at least trying to rename the interfaces from /etc/udev/rules.d/70-persistent-net.rules.
I tried a couple of methods found, but using systemd subsystem didn't work out.
This was working in 2.8.1 and lower.
Resolution:
In /boot/grub/grub.conf there is a kernel option on boot called net.ifnames.
This is set to 0.
This is telling the kernel to disable consistent interface naming.
Meaning it stop using ens172, ens192, etc and use eth0, eth1, etc.
Since these files already exist, the rename file fails.
To fix, change the value to 1 to tell the kernel to use consistent interface name.
Reboot after making the change.
This allowed the udev system to rename the interfaces based on MAC address.
Log Entries Before:
Dec 29 02:42:35 cph-svr4 systemd-udevd[607]: eth1: Failed to rename network interface 3 from 'eth1' to 'eth0': File exists
Dec 29 02:42:35 cph-svr4 systemd-udevd[616]: eth3: Failed to rename network interface 5 from 'eth3' to 'eth1': File exists
Dec 29 02:42:35 cph-svr4 systemd-udevd[591]: eth2: Failed to rename network interface 4 from 'eth2' to 'eth3': File exists
Dec 29 02:42:35 cph-svr4 systemd-udevd[596]: eth0: Failed to rename network interface 2 from 'eth0' to 'eth2': File exists
Log Entries After:
Dec 29 11:48:11 cph-svr4 kernel: [ 6.546551] igb 0000:85:00.1 eth1: renamed from enp133s0f1
Dec 29 11:48:11 cph-svr4 kernel: [ 6.592263] igb 0000:85:00.0 eth0: renamed from enp133s0f0
Dec 29 11:48:11 cph-svr4 kernel: [ 6.638968] bnx2x 0000:06:00.1 eth3: renamed from enp6s0f1
Dec 29 11:48:11 cph-svr4 kernel: [ 6.696162] bnx2x 0000:06:00.0 eth2: renamed from enp6s0f0
If there is a better way to fix this, please let me know.
Problem Description:
After upgrading or installing PetaSAN version 3.0.x based on Ubuntu 20.04, the custom NIC assignments stopped working.
Meaning that if I re-assign eth0 with eth2, this doesn't stick.
The error message is:
eth0: Failed to rename network interface 2 from 'eth0' to 'eth2': File exists
Indicating that it is at least trying to rename the interfaces from /etc/udev/rules.d/70-persistent-net.rules.
I tried a couple of methods found, but using systemd subsystem didn't work out.
This was working in 2.8.1 and lower.
Resolution:
In /boot/grub/grub.conf there is a kernel option on boot called net.ifnames.
This is set to 0.
This is telling the kernel to disable consistent interface naming.
Meaning it stop using ens172, ens192, etc and use eth0, eth1, etc.
Since these files already exist, the rename file fails.
To fix, change the value to 1 to tell the kernel to use consistent interface name.
Reboot after making the change.
This allowed the udev system to rename the interfaces based on MAC address.
Log Entries Before:
Dec 29 02:42:35 cph-svr4 systemd-udevd[607]: eth1: Failed to rename network interface 3 from 'eth1' to 'eth0': File exists
Dec 29 02:42:35 cph-svr4 systemd-udevd[616]: eth3: Failed to rename network interface 5 from 'eth3' to 'eth1': File exists
Dec 29 02:42:35 cph-svr4 systemd-udevd[591]: eth2: Failed to rename network interface 4 from 'eth2' to 'eth3': File exists
Dec 29 02:42:35 cph-svr4 systemd-udevd[596]: eth0: Failed to rename network interface 2 from 'eth0' to 'eth2': File exists
Log Entries After:
Dec 29 11:48:11 cph-svr4 kernel: [ 6.546551] igb 0000:85:00.1 eth1: renamed from enp133s0f1
Dec 29 11:48:11 cph-svr4 kernel: [ 6.592263] igb 0000:85:00.0 eth0: renamed from enp133s0f0
Dec 29 11:48:11 cph-svr4 kernel: [ 6.638968] bnx2x 0000:06:00.1 eth3: renamed from enp6s0f1
Dec 29 11:48:11 cph-svr4 kernel: [ 6.696162] bnx2x 0000:06:00.0 eth2: renamed from enp6s0f0
If there is a better way to fix this, please let me know.
Last edited on December 30, 2021, 2:30 pm by podilarius · #1
admin
2,930 Posts
December 31, 2021, 12:08 pmQuote from admin on December 31, 2021, 12:08 pmbefore you made the changes, had you previously manually edited /etc/udev/rules.d/70-persistent-net.rules ? if not can you post the file which was giving you issues.
before you made the changes, had you previously manually edited /etc/udev/rules.d/70-persistent-net.rules ? if not can you post the file which was giving you issues.
podilarius
44 Posts
December 31, 2021, 6:53 pmQuote from podilarius on December 31, 2021, 6:53 pmI followed the standard method through the console blue screen.
I looked at the file after the change, and it was as it should be.
Meaning that the correct MAC addresses were named where I wanted them to be.
As you can see in the first error message, Ubuntu did try to rename them, but the names were taken and rename failed.
So I didn't change 70-persistent-net.rules, I only had to change the kernel boot options in the grub.conf file for it work properly.
I did try a few things before, but I would always start with a vanilla install before trying the next method.
I know that is nuclear option, but I didn't want one set of edits to interfere with another attempt.
This method was the simplest resolution that worked consistently.
I followed the standard method through the console blue screen.
I looked at the file after the change, and it was as it should be.
Meaning that the correct MAC addresses were named where I wanted them to be.
As you can see in the first error message, Ubuntu did try to rename them, but the names were taken and rename failed.
So I didn't change 70-persistent-net.rules, I only had to change the kernel boot options in the grub.conf file for it work properly.
I did try a few things before, but I would always start with a vanilla install before trying the next method.
I know that is nuclear option, but I didn't want one set of edits to interfere with another attempt.
This method was the simplest resolution that worked consistently.
admin
2,930 Posts
December 31, 2021, 8:59 pmQuote from admin on December 31, 2021, 8:59 pmif you have a chance can can reproduce this: can please post the udev file causing the issue.
if you have a chance can can reproduce this: can please post the udev file causing the issue.
podilarius
44 Posts
January 3, 2022, 3:47 amQuote from podilarius on January 3, 2022, 3:47 amI can reproduce this tomorrow.
Please let me know what all you would like to get to troubleshoot.
I can reproduce this tomorrow.
Please let me know what all you would like to get to troubleshoot.
podilarius
44 Posts
January 4, 2022, 12:44 amQuote from podilarius on January 4, 2022, 12:44 amI had no trouble re-creating this issue.
Just changed the boot option (net.ifnames) to 0 and interfaces didn't load.
Changed it back to 1 and booted normally.
Edit:
It didn't just boot, I had to change the net.iframes to 1 and also remove the KERNEL=="eth*" from the persistent net udev rules.
After that it booted normally.
Here is the working udev file.
cat /etc/udev/rules.d/70-persistent-net.rules
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:25:90:88:f2:48", ATTR{type}=="1", NAME="eth2"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:25:90:88:c1:00", ATTR{type}=="1", NAME="eth0"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:25:90:88:f2:4a", ATTR{type}=="1", NAME="eth3"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:25:90:88:c1:01", ATTR{type}=="1", NAME="eth1"
I had no trouble re-creating this issue.
Just changed the boot option (net.ifnames) to 0 and interfaces didn't load.
Changed it back to 1 and booted normally.
Edit:
It didn't just boot, I had to change the net.iframes to 1 and also remove the KERNEL=="eth*" from the persistent net udev rules.
After that it booted normally.
Here is the working udev file.
cat /etc/udev/rules.d/70-persistent-net.rules
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:25:90:88:f2:48", ATTR{type}=="1", NAME="eth2"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:25:90:88:c1:00", ATTR{type}=="1", NAME="eth0"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:25:90:88:f2:4a", ATTR{type}=="1", NAME="eth3"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:25:90:88:c1:01", ATTR{type}=="1", NAME="eth1"
Last edited on January 4, 2022, 1:35 am by podilarius · #6
podilarius
44 Posts
January 10, 2022, 9:26 pmQuote from podilarius on January 10, 2022, 9:26 pmLet me know if you need any further information.
Let me know if you need any further information.
admin
2,930 Posts
February 6, 2022, 2:01 pmQuote from admin on February 6, 2022, 2:01 pmwe will update the interface naming as per your info in the next bug fix release, should be soon.
we will update the interface naming as per your info in the next bug fix release, should be soon.
Brandon
2 Posts
March 1, 2022, 1:01 amQuote from Brandon on March 1, 2022, 1:01 amJust did the upgrade to 3.0 on our cluster and also ran into this issue (when I rebooted). I couldn't try running the new update script at this point as the node was unreachable on the network.
After rebuilding the node and rejoining the cluster I ran the 2.8.1 to 3 update script, then apt update, and lastly ran the new updater which brings it up to 3.0.2. This worked perfect and all nodes retain their network settings on reboot.
Admin if you see this, I'd recommend throwing something in the Online Upgrade Guide in the 2.8.x to 3.0.0 section so others are aware when running it, in order to have it fully update to 3.0.2.
Otherwise the update was smooth and I like the new method for upgrading!
Just did the upgrade to 3.0 on our cluster and also ran into this issue (when I rebooted). I couldn't try running the new update script at this point as the node was unreachable on the network.
After rebuilding the node and rejoining the cluster I ran the 2.8.1 to 3 update script, then apt update, and lastly ran the new updater which brings it up to 3.0.2. This worked perfect and all nodes retain their network settings on reboot.
Admin if you see this, I'd recommend throwing something in the Online Upgrade Guide in the 2.8.x to 3.0.0 section so others are aware when running it, in order to have it fully update to 3.0.2.
Otherwise the update was smooth and I like the new method for upgrading!
Error re-assigning Link names in PetaSAN 3.0.x (Ubuntu 20.04)
podilarius
44 Posts
Quote from podilarius on December 30, 2021, 2:20 pmProblem Description:
After upgrading or installing PetaSAN version 3.0.x based on Ubuntu 20.04, the custom NIC assignments stopped working.
Meaning that if I re-assign eth0 with eth2, this doesn't stick.
The error message is:
eth0: Failed to rename network interface 2 from 'eth0' to 'eth2': File exists
Indicating that it is at least trying to rename the interfaces from /etc/udev/rules.d/70-persistent-net.rules.I tried a couple of methods found, but using systemd subsystem didn't work out.
This was working in 2.8.1 and lower.
Resolution:
In /boot/grub/grub.conf there is a kernel option on boot called net.ifnames.
This is set to 0.
This is telling the kernel to disable consistent interface naming.
Meaning it stop using ens172, ens192, etc and use eth0, eth1, etc.
Since these files already exist, the rename file fails.
To fix, change the value to 1 to tell the kernel to use consistent interface name.
Reboot after making the change.
This allowed the udev system to rename the interfaces based on MAC address.Log Entries Before:
Dec 29 02:42:35 cph-svr4 systemd-udevd[607]: eth1: Failed to rename network interface 3 from 'eth1' to 'eth0': File exists
Dec 29 02:42:35 cph-svr4 systemd-udevd[616]: eth3: Failed to rename network interface 5 from 'eth3' to 'eth1': File exists
Dec 29 02:42:35 cph-svr4 systemd-udevd[591]: eth2: Failed to rename network interface 4 from 'eth2' to 'eth3': File exists
Dec 29 02:42:35 cph-svr4 systemd-udevd[596]: eth0: Failed to rename network interface 2 from 'eth0' to 'eth2': File existsLog Entries After:
Dec 29 11:48:11 cph-svr4 kernel: [ 6.546551] igb 0000:85:00.1 eth1: renamed from enp133s0f1
Dec 29 11:48:11 cph-svr4 kernel: [ 6.592263] igb 0000:85:00.0 eth0: renamed from enp133s0f0
Dec 29 11:48:11 cph-svr4 kernel: [ 6.638968] bnx2x 0000:06:00.1 eth3: renamed from enp6s0f1
Dec 29 11:48:11 cph-svr4 kernel: [ 6.696162] bnx2x 0000:06:00.0 eth2: renamed from enp6s0f0If there is a better way to fix this, please let me know.
Problem Description:
After upgrading or installing PetaSAN version 3.0.x based on Ubuntu 20.04, the custom NIC assignments stopped working.
Meaning that if I re-assign eth0 with eth2, this doesn't stick.
The error message is:
eth0: Failed to rename network interface 2 from 'eth0' to 'eth2': File exists
Indicating that it is at least trying to rename the interfaces from /etc/udev/rules.d/70-persistent-net.rules.
I tried a couple of methods found, but using systemd subsystem didn't work out.
This was working in 2.8.1 and lower.
Resolution:
In /boot/grub/grub.conf there is a kernel option on boot called net.ifnames.
This is set to 0.
This is telling the kernel to disable consistent interface naming.
Meaning it stop using ens172, ens192, etc and use eth0, eth1, etc.
Since these files already exist, the rename file fails.
To fix, change the value to 1 to tell the kernel to use consistent interface name.
Reboot after making the change.
This allowed the udev system to rename the interfaces based on MAC address.
Log Entries Before:
Dec 29 02:42:35 cph-svr4 systemd-udevd[607]: eth1: Failed to rename network interface 3 from 'eth1' to 'eth0': File exists
Dec 29 02:42:35 cph-svr4 systemd-udevd[616]: eth3: Failed to rename network interface 5 from 'eth3' to 'eth1': File exists
Dec 29 02:42:35 cph-svr4 systemd-udevd[591]: eth2: Failed to rename network interface 4 from 'eth2' to 'eth3': File exists
Dec 29 02:42:35 cph-svr4 systemd-udevd[596]: eth0: Failed to rename network interface 2 from 'eth0' to 'eth2': File exists
Log Entries After:
Dec 29 11:48:11 cph-svr4 kernel: [ 6.546551] igb 0000:85:00.1 eth1: renamed from enp133s0f1
Dec 29 11:48:11 cph-svr4 kernel: [ 6.592263] igb 0000:85:00.0 eth0: renamed from enp133s0f0
Dec 29 11:48:11 cph-svr4 kernel: [ 6.638968] bnx2x 0000:06:00.1 eth3: renamed from enp6s0f1
Dec 29 11:48:11 cph-svr4 kernel: [ 6.696162] bnx2x 0000:06:00.0 eth2: renamed from enp6s0f0
If there is a better way to fix this, please let me know.
admin
2,930 Posts
Quote from admin on December 31, 2021, 12:08 pmbefore you made the changes, had you previously manually edited /etc/udev/rules.d/70-persistent-net.rules ? if not can you post the file which was giving you issues.
before you made the changes, had you previously manually edited /etc/udev/rules.d/70-persistent-net.rules ? if not can you post the file which was giving you issues.
podilarius
44 Posts
Quote from podilarius on December 31, 2021, 6:53 pmI followed the standard method through the console blue screen.
I looked at the file after the change, and it was as it should be.
Meaning that the correct MAC addresses were named where I wanted them to be.
As you can see in the first error message, Ubuntu did try to rename them, but the names were taken and rename failed.
So I didn't change 70-persistent-net.rules, I only had to change the kernel boot options in the grub.conf file for it work properly.I did try a few things before, but I would always start with a vanilla install before trying the next method.
I know that is nuclear option, but I didn't want one set of edits to interfere with another attempt.
This method was the simplest resolution that worked consistently.
I followed the standard method through the console blue screen.
I looked at the file after the change, and it was as it should be.
Meaning that the correct MAC addresses were named where I wanted them to be.
As you can see in the first error message, Ubuntu did try to rename them, but the names were taken and rename failed.
So I didn't change 70-persistent-net.rules, I only had to change the kernel boot options in the grub.conf file for it work properly.
I did try a few things before, but I would always start with a vanilla install before trying the next method.
I know that is nuclear option, but I didn't want one set of edits to interfere with another attempt.
This method was the simplest resolution that worked consistently.
admin
2,930 Posts
Quote from admin on December 31, 2021, 8:59 pmif you have a chance can can reproduce this: can please post the udev file causing the issue.
if you have a chance can can reproduce this: can please post the udev file causing the issue.
podilarius
44 Posts
Quote from podilarius on January 3, 2022, 3:47 amI can reproduce this tomorrow.
Please let me know what all you would like to get to troubleshoot.
I can reproduce this tomorrow.
Please let me know what all you would like to get to troubleshoot.
podilarius
44 Posts
Quote from podilarius on January 4, 2022, 12:44 amI had no trouble re-creating this issue.
Just changed the boot option (net.ifnames) to 0 and interfaces didn't load.
Changed it back to 1 and booted normally.
Edit:
It didn't just boot, I had to change the net.iframes to 1 and also remove the KERNEL=="eth*" from the persistent net udev rules.
After that it booted normally.Here is the working udev file.
cat /etc/udev/rules.d/70-persistent-net.rules
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:25:90:88:f2:48", ATTR{type}=="1", NAME="eth2"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:25:90:88:c1:00", ATTR{type}=="1", NAME="eth0"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:25:90:88:f2:4a", ATTR{type}=="1", NAME="eth3"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:25:90:88:c1:01", ATTR{type}=="1", NAME="eth1"
I had no trouble re-creating this issue.
Just changed the boot option (net.ifnames) to 0 and interfaces didn't load.
Changed it back to 1 and booted normally.
Edit:
It didn't just boot, I had to change the net.iframes to 1 and also remove the KERNEL=="eth*" from the persistent net udev rules.
After that it booted normally.
Here is the working udev file.
cat /etc/udev/rules.d/70-persistent-net.rules
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:25:90:88:f2:48", ATTR{type}=="1", NAME="eth2"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:25:90:88:c1:00", ATTR{type}=="1", NAME="eth0"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:25:90:88:f2:4a", ATTR{type}=="1", NAME="eth3"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:25:90:88:c1:01", ATTR{type}=="1", NAME="eth1"
podilarius
44 Posts
Quote from podilarius on January 10, 2022, 9:26 pmLet me know if you need any further information.
Let me know if you need any further information.
admin
2,930 Posts
Quote from admin on February 6, 2022, 2:01 pmwe will update the interface naming as per your info in the next bug fix release, should be soon.
we will update the interface naming as per your info in the next bug fix release, should be soon.
Brandon
2 Posts
Quote from Brandon on March 1, 2022, 1:01 amJust did the upgrade to 3.0 on our cluster and also ran into this issue (when I rebooted). I couldn't try running the new update script at this point as the node was unreachable on the network.
After rebuilding the node and rejoining the cluster I ran the 2.8.1 to 3 update script, then apt update, and lastly ran the new updater which brings it up to 3.0.2. This worked perfect and all nodes retain their network settings on reboot.
Admin if you see this, I'd recommend throwing something in the Online Upgrade Guide in the 2.8.x to 3.0.0 section so others are aware when running it, in order to have it fully update to 3.0.2.
Otherwise the update was smooth and I like the new method for upgrading!
Just did the upgrade to 3.0 on our cluster and also ran into this issue (when I rebooted). I couldn't try running the new update script at this point as the node was unreachable on the network.
After rebuilding the node and rejoining the cluster I ran the 2.8.1 to 3 update script, then apt update, and lastly ran the new updater which brings it up to 3.0.2. This worked perfect and all nodes retain their network settings on reboot.
Admin if you see this, I'd recommend throwing something in the Online Upgrade Guide in the 2.8.x to 3.0.0 section so others are aware when running it, in order to have it fully update to 3.0.2.
Otherwise the update was smooth and I like the new method for upgrading!