Forums

Home / Forums

You need to log in to create posts and topics. Login · Register

Network Upgrade: Updating cluster_info.json


wailer
75 Posts
July 30, 2019, 11:08 pm

Hi,

We have upgraded network hardware. We previously had  6 x 1Gbit network interfaces: 2 bonds (2 +2) for backend and 2 interfaces for iSCSI targets. Now all nodes have 4 x 1 gbit and 4x10Gbit interfaces. Eth0(1gbit) is used for management. Eth 4 and 5 are bonded for backend traffic , and eth6 and 7 are going to be used for iSCSI traffic.

We are doing LACP against 2 Cisco Nexus switches with vPC technology, but it doesn't seem to work.

We have updated cluster_info.json file accordingly as follows:

{
"backend_1_base_ip": "10.74.0.0",
"backend_1_eth_name": "backend",
"backend_1_mask": "255.255.255.0",
"backend_1_vlan_id": "74",
"backend_2_base_ip": "10.75.0.0",
"backend_2_eth_name": "backend",
"backend_2_mask": "255.255.255.0",
"backend_2_vlan_id": "75",
"bonds": [
{
"interfaces": "eth4,eth5",
"is_jumbo_frames": true,
"mode": "802.3ad",
"name": "backend",
"primary_interface": ""
}
],
"eth_count": 8,
"iscsi_1_eth_name": "eth6",
"iscsi_2_eth_name": "eth7",
"jumbo_frames": [
"eth6",
"eth7"
],
"management_eth_name": "eth0",
"management_nodes": [
{
"backend_1_ip": "10.74.0.211",
"backend_2_ip": "10.75.0.211",
"is_iscsi": true,
"is_management": true,
"is_storage": true,
"management_ip": "192.168.77.211",
"name": "ceph-11-osd"
},
{
"backend_1_ip": "10.74.0.212",
"backend_2_ip": "10.75.0.212",
"is_iscsi": true,
"is_management": true,
"is_storage": true,
"management_ip": "192.168.77.212",
"name": "ceph12-osd"
},
{
"backend_1_ip": "10.74.0.213",

"backend_2_ip": "10.75.0.213",
"is_iscsi": true,
"is_management": true,
"is_storage": true,
"management_ip": "192.168.77.213",
"name": "ceph-13-osd"
}
],
"name": "cluster1"
}

EDIT: Looks like network config is being applied correctly , at least the bonding part.

Ifconfig -a shows the following info:

backend Link encap:Ethernet HWaddr 9c:69:b4:60:9b:80
inet6 addr: fe80::9e69:b4ff:fe60:9b80/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:9000 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:817 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:38316 (38.3 KB)

backend.74 Link encap:Ethernet HWaddr 9c:69:b4:60:9b:80
inet addr:10.74.0.211 Bcast:0.0.0.0 Mask:255.255.255.0
inet6 addr: fe80::9e69:b4ff:fe60:9b80/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:760 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:32352 (32.3 KB)

backend.75 Link encap:Ethernet HWaddr 9c:69:b4:60:9b:80
inet addr:10.75.0.211 Bcast:0.0.0.0 Mask:255.255.255.0
inet6 addr: fe80::9e69:b4ff:fe60:9b80/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:12 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:936 (936.0 B)

eth0 Link encap:Ethernet HWaddr ac:16:2d:6e:0d:5c
inet addr:192.168.77.211 Bcast:0.0.0.0 Mask:255.255.255.0
inet6 addr: fe80::ae16:2dff:fe6e:d5c/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1857 errors:0 dropped:0 overruns:0 frame:0
TX packets:792 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:150616 (150.6 KB) TX bytes:101446 (101.4 KB)
Interrupt:27

eth1 Link encap:Ethernet HWaddr ac:16:2d:6e:0d:5d
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
Interrupt:41

eth2 Link encap:Ethernet HWaddr ac:16:2d:6e:0d:5e
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0

TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
Interrupt:27

eth4 Link encap:Ethernet HWaddr 9c:69:b4:60:9b:80
UP BROADCAST RUNNING SLAVE MULTICAST MTU:9000 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:801 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:36332 (36.3 KB)

eth5 Link encap:Ethernet HWaddr 9c:69:b4:60:9b:80
UP BROADCAST RUNNING SLAVE MULTICAST MTU:9000 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:16 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:1984 (1.9 KB)

eth6 Link encap:Ethernet HWaddr 9c:69:b4:60:9b:68
inet6 addr: fe80::9e69:b4ff:fe60:9b68/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:12 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:936 (936.0 B)

eth7 Link encap:Ethernet HWaddr 9c:69:b4:60:9b:69
inet6 addr: fe80::9e69:b4ff:fe60:9b69/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:12 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:936 (936.0 B)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:15241 errors:0 dropped:0 overruns:0 frame:0
TX packets:15241 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1372834 (1.3 MB) TX bytes:1372834 (1.3 MB)

rename5 Link encap:Ethernet HWaddr ac:16:2d:6e:0d:5f
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0

TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
Interrupt:41

 

 

EDIT: Everything looks correct, right?

After some more investigation I am not sure how the bonding creation works in petasan , as I see no config at /etc/network/interfaces. I would like to make some tuning to the bonds.. like checking BONDING_OPTS which are being set/used currently. Where could I find these? I have also noticed partner mac address is being set to null:

cat /proc/net/bonding/backend

Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: slow
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: 9c:69:b4:60:9b:80
Active Aggregator Info:
Aggregator ID: 1
Number of ports: 1
Actor Key: 15
Partner Key: 1
Partner Mac Address: 00:00:00:00:00:00

Slave Interface: eth4
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 9c:69:b4:60:9b:80
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: churned
Actor Churned Count: 0
Partner Churned Count: 1
details actor lacp pdu:
system priority: 65535
system mac address: 9c:69:b4:60:9b:80
port key: 15
port priority: 255
port number: 1
port state: 77
details partner lacp pdu:
system priority: 65535
system mac address: 00:00:00:00:00:00
oper key: 1
port priority: 255

port number: 1
port state: 1

Slave Interface: eth5
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 9c:69:b4:60:9b:81
Slave queue ID: 0
Aggregator ID: 2
Actor Churn State: churned
Partner Churn State: churned
Actor Churned Count: 1
Partner Churned Count: 1
details actor lacp pdu:
system priority: 65535
system mac address: 9c:69:b4:60:9b:80
port key: 15
port priority: 255
port number: 2
port state: 69
details partner lacp pdu:
system priority: 65535
system mac address: 00:00:00:00:00:00
oper key: 1
port priority: 255
port number: 1
port state: 1

 

Thanks!

 

 

 

Last edited on July 31, 2019, 2:23 am by wailer · #1

admin
2,930 Posts
July 31, 2019, 6:21 am

Hi

There are 2 config files in /opt/petasan/config:  cluser_info.json and node_info.json , the first defines cluster wide configuration, the second is node specific. cluster_info is the same across all nodes and includes the node_info of the first 3 nodes so nodes can connect to the cluster, node_info is required on all nodes including the first 3 nodes.

If you do not want to edit cluster_info by hand, you can create a dummy vm on your desktop (eg using virtualbox) and create a cluster on this single node that matches your cluster settings, then copy the config file.

If you want to tweak the bonding step, the startup script is in   /opt/petasan/scripts/node_start_ips.py

Last edited on July 31, 2019, 6:24 am by admin · #2

wailer
75 Posts
July 31, 2019, 4:05 pm

Hi, thanks for your tips.

Finally we found the issue to be /etc/udev/rules.d persistents rules which were misbehaving and changing nich order... we removed the file from all nodes and restarted them and now backend connection looks fine.

We only see 1 issue now in ceph. 1 mon is down with this message:

 

root@ceph12-osd:/opt/petasan/log# ceph -c /etc/ceph/cluster1.conf
ceph> health detail
HEALTH_WARN 1/3 mons down, quorum ceph-11-osd,ceph-13-osd
MON_DOWN 1/3 mons down, quorum ceph-11-osd,ceph-13-osd
mon.ceph12-osd (rank 1) addr 10.74.0.212:6789/0 is down (out of quorum)

We have tried to restart this node, but the service never seems to start...backend connectivity looks ok , as we can ping 10.74.0.212 from the other nodes without problem.

Any idea?

Here are the logs for the mon node not working.

31/07/2019 17:31:21 WARNING , retrying in 2 seconds...

31/07/2019 17:31:17 WARNING , retrying in 2 seconds...

31/07/2019 17:31:17 WARNING , retrying in 2 seconds...

31/07/2019 17:31:17 WARNING , retrying in 2 seconds...

31/07/2019 17:31:17 WARNING , retrying in 2 seconds...

31/07/2019 17:31:13 WARNING , retrying in 1 seconds...

31/07/2019 17:31:09 WARNING , retrying in 1 seconds...

31/07/2019 17:31:09 WARNING , retrying in 1 seconds...

31/07/2019 17:31:09 WARNING , retrying in 1 seconds...

31/07/2019 17:31:08 WARNING , retrying in 1 seconds...

31/07/2019 17:31:02 INFO Cleaning unused ips.

31/07/2019 17:31:02 INFO Cleaning unused rbd images.

31/07/2019 17:31:01 INFO Cleaning all mapped disks

31/07/2019 17:31:01 INFO Cleaning unused configurations.

31/07/2019 17:31:01 INFO Service is starting.

31/07/2019 17:31:01 INFO Starting OSDs

31/07/2019 17:31:01 INFO Starting Node Stats Service

31/07/2019 17:31:01 INFO Starting Cluster Management application

31/07/2019 17:31:01 INFO Starting iSCSI Service

31/07/2019 17:31:01 INFO Starting cluster file sync service

31/07/2019 17:30:59 INFO str_start_command: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>consul agent -config-dir /opt/petasan/config/etc/consul.d/server -bind 10.74.0.212 -retry-join 10.74.0.211 -retry-join 10.74.0.213

31/07/2019 17:30:48 INFO GlusterFS mount attempt

31/07/2019 17:30:46 INFO Start settings IPs

Last edited on July 31, 2019, 4:05 pm by wailer · #3

admin
2,930 Posts
July 31, 2019, 6:18 pm

Hi

Has the ip address charged ?

What messages do you see in

systemctl status ceph-mon@HOSTNAME

what messages if you start it im command line

ceph-mon -f --cluster ceph  --id HOSTNAME  --setuser ceph --setgroup ceph

Also the logs could be useful:

cat /var/log/ceph/ceph-mon.HOSTNAME.log

if needed you can increase the log level in conf file

debug_mon = 10/10

or

debug_mon = 20/20

 

 

#4

wailer
75 Posts
July 31, 2019, 7:23 pm

Hi there,

The IP has not changed, all backend IP's are the same.

root@ceph12-osd:~# systemctl status ceph-mon@ceph12-osd
● ceph-mon@ceph12-osd.service - Ceph cluster monitor daemon
Loaded: loaded (/lib/systemd/system/ceph-mon@.service; enabled; vendor pre
: enabled)
Drop-In: /etc/systemd/system/ceph-mon@.service.d
└─override.conf
Active: inactive (dead)

Trying to manually start the service show this output:

 

root@ceph12-osd:~# ceph-mon -f --cluster ceph --id HOSTNAME --setuser ceph -- setgroup ceph
2019-07-31 21:18:30.778009 7f58069a1080 -1 did not load config file, using defa ult settings.
2019-07-31 21:18:30.780307 7f58069a1080 -1 Errors while parsing config file!
2019-07-31 21:18:30.780311 7f58069a1080 -1 parse_file: cannot open /etc/ceph/ce ph.conf: (2) No such file or directory
2019-07-31 21:18:30.780312 7f58069a1080 -1 parse_file: cannot open ~/.ceph/ceph .conf: (2) No such file or directory
2019-07-31 21:18:30.780313 7f58069a1080 -1 parse_file: cannot open ceph.conf: ( 2) No such file or directory
2019-07-31 21:18:30.782728 7f58069a1080 -1 Errors while parsing config file!
2019-07-31 21:18:30.782729 7f58069a1080 -1 parse_file: cannot open /etc/ceph/ce ph.conf: (2) No such file or directory
2019-07-31 21:18:30.782732 7f58069a1080 -1 parse_file: cannot open ~/.ceph/ceph .conf: (2) No such file or directory
2019-07-31 21:18:30.782733 7f58069a1080 -1 parse_file: cannot open ceph.conf: ( 2) No such file or directory
2019-07-31 21:18:30.783085 7f58069a1080 -1 monitor data directory at '/var/lib/ ceph/mon/ceph-HOSTNAME' does not exist: have you run 'mkfs'?

This is the contents of /var/lib/ceph/mon/ :

131074 4.0K drwxr-xr-x 3 ceph ceph 4.0K Feb 4 12:36 cluster1-ceph12-osd

And inside this folder we have:

131073 4.0K drwxr-xr-x 3 ceph ceph 4.0K Feb 4 12:36 ..
131084 0 -rw-r--r-- 1 ceph ceph 0 Feb 4 12:36 done
131083 4.0K -rw------- 1 ceph ceph 77 Feb 4 12:36 keyring
131075 4.0K -rw-r--r-- 1 ceph ceph 8 Feb 4 12:36 kv_backend
131076 4.0K drwxr-xr-x 2 ceph ceph 4.0K Jul 30 16:11 store.db
131085 0 -rw-r--r-- 1 ceph ceph 0 Feb 4 12:36 systemd

 

Thanks!

 

 

#5

admin
2,930 Posts
July 31, 2019, 7:40 pm

pardon me  you need to specify the cluster name

ceph-mon -f --cluster CLUSTER_NAME --id HOSTNAME --setuser ceph -- setgroup ceph

+ as before :

Also the logs could be useful:

cat /var/log/ceph/ceph-mon.HOSTNAME.log

if needed you can increase the log level in conf file

debug_mon = 10/10

or

debug_mon = 20/20

#6

wailer
75 Posts
July 31, 2019, 7:56 pm

Thank you , just found it! the mon folder had root:root owner , while doing checks from your previous tip I noticed it... changed it to ceph and now it's working correctly!

 

Last edited on July 31, 2019, 7:56 pm by wailer · #7

admin
2,930 Posts
July 31, 2019, 8:19 pm

Excellent 🙂

One thing to watch is if you did change the interface names for iSCSI subnets 1 and 2 ( such as from eth2 to bond3),  this data needs to be updated in the ceph image metadata of the iSCIS disks,  as the interface name is used to configure starting the disk as well as during path re-assignment and failover. One way is to stop / detach and re-attach the disks, this will clear the metadata and update it with new data including interface name. Note that doing this  will most probably assign new ip values. If you are using 2.3.0, there is command line utility to update the metadata manually. Again if you did not change the interface names, you do not need this.

#8

wailer
75 Posts
July 31, 2019, 10:56 pm

Yep,  we did change interface names and had to detach and reattach the disk in order to make it work. Luckily before doing this change we already searched in this forum and found a similar case from another used where you were mentioning that. So all is good right now.

Thanks again!

#9