Add mon with different count of network adapters
yudorogov
31 Posts
January 17, 2018, 3:12 pmQuote from yudorogov on January 17, 2018, 3:12 pmHi, i am build petasan cluster with:
8 osd nodes, 36x6TB sata hdd per node, ssd on petasan, 2xSSD nvram for journals, 4x10GBE ethernet(2+2 in bonding for cluster and production networks)
3 mon nodes, ssd for petasan, 2x10GBE ethernet for cluster and production network
2 iscsi target node same configuration of mon nodes
PS: all nodes have dedicated network for manage is 1GBps
Cluster with osd nodes build successfully, after i try to add mon or iscsi nodes, and get error with different count of network adapters, how i can add this nodes to petasan cluster, and
Hi, i am build petasan cluster with:
8 osd nodes, 36x6TB sata hdd per node, ssd on petasan, 2xSSD nvram for journals, 4x10GBE ethernet(2+2 in bonding for cluster and production networks)
3 mon nodes, ssd for petasan, 2x10GBE ethernet for cluster and production network
2 iscsi target node same configuration of mon nodes
PS: all nodes have dedicated network for manage is 1GBps
Cluster with osd nodes build successfully, after i try to add mon or iscsi nodes, and get error with different count of network adapters, how i can add this nodes to petasan cluster, and
Last edited on January 17, 2018, 3:26 pm by yudorogov · #1
yudorogov
31 Posts
January 17, 2018, 3:13 pmQuote from yudorogov on January 17, 2018, 3:13 pmMigrate management role from osd to mon node
Migrate management role from osd to mon node
admin
2,930 Posts
January 17, 2018, 3:57 pmQuote from admin on January 17, 2018, 3:57 pmAll PetaSAN nodes need to have the same number of NICs and the mappings of NICs to subnets is the same, this is defined at cluster creation time. Having the same nics and subnet mapping across all nodes makes it easier to deploy as well as to operate/troubleshoot the cluster. We also allow to change the node roles at run time so we need to have all subnets available.
The 3 mons need to be on the PetaSAN Management nodes, the osd storage nodes can be on any node
All PetaSAN nodes need to have the same number of NICs and the mappings of NICs to subnets is the same, this is defined at cluster creation time. Having the same nics and subnet mapping across all nodes makes it easier to deploy as well as to operate/troubleshoot the cluster. We also allow to change the node roles at run time so we need to have all subnets available.
The 3 mons need to be on the PetaSAN Management nodes, the osd storage nodes can be on any node
Last edited on January 17, 2018, 4:04 pm by admin · #3
yudorogov
31 Posts
January 18, 2018, 12:09 amQuote from yudorogov on January 18, 2018, 12:09 amMaybe have method to add custom network configuration for adding management nodes? Edit config in management nodes? Or How disable fetching network setting from management nodes in process of join to cluster, and set it manually at time of join to cluster. All trouble is mgmt node not enough bonding interfaces qty. I am build cluster with bare metal.
osd nodes have 2 bondings(eth2+eth4 is backend1, eth3+eth5 is backend2, eth0 is mgmt) on mon nodes i have eth3 connected to backend1 and eth4 to backend2, eth0 is mgmt
Maybe have method to add custom network configuration for adding management nodes? Edit config in management nodes? Or How disable fetching network setting from management nodes in process of join to cluster, and set it manually at time of join to cluster. All trouble is mgmt node not enough bonding interfaces qty. I am build cluster with bare metal.
osd nodes have 2 bondings(eth2+eth4 is backend1, eth3+eth5 is backend2, eth0 is mgmt) on mon nodes i have eth3 connected to backend1 and eth4 to backend2, eth0 is mgmt
Last edited on January 18, 2018, 12:18 am by yudorogov · #4
admin
2,930 Posts
January 18, 2018, 9:05 amQuote from admin on January 18, 2018, 9:05 amIt is much simpler that all cluster nodes have the same nic count and subnet to nic mappings which in PetaSAN is defined at cluster creation. it makes deployment and troubleshooting simpler. So in your case either have 4 nics or 2 nics per node.
If you really need to have different configuration, the following may guide you, but it is not supported and it will break if you decide to add roles to a node. Assume you need to re-assign eth1 to eth 3, eth 2 to eth 4 and add dummy eth 1 and eth 2.
After installation using the bash shell from console menu (blue screen) to add temp password
use ssh to login
To view existing intergaces
ls /sys/class/net/
To add dummy interfaces
ip link add eth3 type dummy
ip link add eth4 type dummy
ip link del dummy0
To re-name eth1 -> eth3, eth2->4 use the node Interface Naming console menu (blue screen)
To re-assign the management interface, rename it in
/etc/network/interfaces
After reboot the real interfaces will be renamed
Before deploy re-add some dummy interfaces so interface count check will pass during join
ip link add eth1 type dummy
ip link add eth2 type dummy
ip link del dummy0
Perform deploy
It is much simpler that all cluster nodes have the same nic count and subnet to nic mappings which in PetaSAN is defined at cluster creation. it makes deployment and troubleshooting simpler. So in your case either have 4 nics or 2 nics per node.
If you really need to have different configuration, the following may guide you, but it is not supported and it will break if you decide to add roles to a node. Assume you need to re-assign eth1 to eth 3, eth 2 to eth 4 and add dummy eth 1 and eth 2.
After installation using the bash shell from console menu (blue screen) to add temp password
use ssh to login
To view existing intergaces
ls /sys/class/net/
To add dummy interfaces
ip link add eth3 type dummy
ip link add eth4 type dummy
ip link del dummy0
To re-name eth1 -> eth3, eth2->4 use the node Interface Naming console menu (blue screen)
To re-assign the management interface, rename it in
/etc/network/interfaces
After reboot the real interfaces will be renamed
Before deploy re-add some dummy interfaces so interface count check will pass during join
ip link add eth1 type dummy
ip link add eth2 type dummy
ip link del dummy0
Perform deploy
Add mon with different count of network adapters
yudorogov
31 Posts
Quote from yudorogov on January 17, 2018, 3:12 pmHi, i am build petasan cluster with:
8 osd nodes, 36x6TB sata hdd per node, ssd on petasan, 2xSSD nvram for journals, 4x10GBE ethernet(2+2 in bonding for cluster and production networks)
3 mon nodes, ssd for petasan, 2x10GBE ethernet for cluster and production network
2 iscsi target node same configuration of mon nodes
PS: all nodes have dedicated network for manage is 1GBps
Cluster with osd nodes build successfully, after i try to add mon or iscsi nodes, and get error with different count of network adapters, how i can add this nodes to petasan cluster, and
Hi, i am build petasan cluster with:
8 osd nodes, 36x6TB sata hdd per node, ssd on petasan, 2xSSD nvram for journals, 4x10GBE ethernet(2+2 in bonding for cluster and production networks)
3 mon nodes, ssd for petasan, 2x10GBE ethernet for cluster and production network
2 iscsi target node same configuration of mon nodes
PS: all nodes have dedicated network for manage is 1GBps
Cluster with osd nodes build successfully, after i try to add mon or iscsi nodes, and get error with different count of network adapters, how i can add this nodes to petasan cluster, and
yudorogov
31 Posts
Quote from yudorogov on January 17, 2018, 3:13 pmMigrate management role from osd to mon node
Migrate management role from osd to mon node
admin
2,930 Posts
Quote from admin on January 17, 2018, 3:57 pmAll PetaSAN nodes need to have the same number of NICs and the mappings of NICs to subnets is the same, this is defined at cluster creation time. Having the same nics and subnet mapping across all nodes makes it easier to deploy as well as to operate/troubleshoot the cluster. We also allow to change the node roles at run time so we need to have all subnets available.
The 3 mons need to be on the PetaSAN Management nodes, the osd storage nodes can be on any node
All PetaSAN nodes need to have the same number of NICs and the mappings of NICs to subnets is the same, this is defined at cluster creation time. Having the same nics and subnet mapping across all nodes makes it easier to deploy as well as to operate/troubleshoot the cluster. We also allow to change the node roles at run time so we need to have all subnets available.
The 3 mons need to be on the PetaSAN Management nodes, the osd storage nodes can be on any node
yudorogov
31 Posts
Quote from yudorogov on January 18, 2018, 12:09 amMaybe have method to add custom network configuration for adding management nodes? Edit config in management nodes? Or How disable fetching network setting from management nodes in process of join to cluster, and set it manually at time of join to cluster. All trouble is mgmt node not enough bonding interfaces qty. I am build cluster with bare metal.
osd nodes have 2 bondings(eth2+eth4 is backend1, eth3+eth5 is backend2, eth0 is mgmt) on mon nodes i have eth3 connected to backend1 and eth4 to backend2, eth0 is mgmt
Maybe have method to add custom network configuration for adding management nodes? Edit config in management nodes? Or How disable fetching network setting from management nodes in process of join to cluster, and set it manually at time of join to cluster. All trouble is mgmt node not enough bonding interfaces qty. I am build cluster with bare metal.
osd nodes have 2 bondings(eth2+eth4 is backend1, eth3+eth5 is backend2, eth0 is mgmt) on mon nodes i have eth3 connected to backend1 and eth4 to backend2, eth0 is mgmt
admin
2,930 Posts
Quote from admin on January 18, 2018, 9:05 amIt is much simpler that all cluster nodes have the same nic count and subnet to nic mappings which in PetaSAN is defined at cluster creation. it makes deployment and troubleshooting simpler. So in your case either have 4 nics or 2 nics per node.
If you really need to have different configuration, the following may guide you, but it is not supported and it will break if you decide to add roles to a node. Assume you need to re-assign eth1 to eth 3, eth 2 to eth 4 and add dummy eth 1 and eth 2.
After installation using the bash shell from console menu (blue screen) to add temp password
use ssh to login
To view existing intergaces
ls /sys/class/net/
To add dummy interfaces
ip link add eth3 type dummy
ip link add eth4 type dummy
ip link del dummy0
To re-name eth1 -> eth3, eth2->4 use the node Interface Naming console menu (blue screen)
To re-assign the management interface, rename it in
/etc/network/interfaces
After reboot the real interfaces will be renamed
Before deploy re-add some dummy interfaces so interface count check will pass during join
ip link add eth1 type dummy
ip link add eth2 type dummy
ip link del dummy0
Perform deploy
It is much simpler that all cluster nodes have the same nic count and subnet to nic mappings which in PetaSAN is defined at cluster creation. it makes deployment and troubleshooting simpler. So in your case either have 4 nics or 2 nics per node.
If you really need to have different configuration, the following may guide you, but it is not supported and it will break if you decide to add roles to a node. Assume you need to re-assign eth1 to eth 3, eth 2 to eth 4 and add dummy eth 1 and eth 2.
After installation using the bash shell from console menu (blue screen) to add temp password
use ssh to login
To view existing intergaces
ls /sys/class/net/
To add dummy interfaces
ip link add eth3 type dummy
ip link add eth4 type dummy
ip link del dummy0
To re-name eth1 -> eth3, eth2->4 use the node Interface Naming console menu (blue screen)
To re-assign the management interface, rename it in
/etc/network/interfaces
After reboot the real interfaces will be renamed
Before deploy re-add some dummy interfaces so interface count check will pass during join
ip link add eth1 type dummy
ip link add eth2 type dummy
ip link del dummy0
Perform deploy