Forums

Home / Forums

You need to log in to create posts and topics. Login · Register

Node interfaces do not match cluster interface settings

Pages: 1 2

afrima
17 Posts
February 19, 2018, 6:47 pm

we have a 3-node cluster working just fine. Using the same hardware (same machines with the same number of 10 GB and 1 GB NICs) we are going to add 3 more physical nodes for testing purposes.

While joining to the new cluster at first step it jumps to 6th step and it gives this error:

 Alert!

Node interfaces do not match cluster interface settings.

There is already another discussion with the same topic, but none of the solutions there, worked for me!

NICs are in the same order as the ones shown during the installation wizard.

Any help is appreciated.

 

afrima

#1

admin
2,930 Posts
February 19, 2018, 8:02 pm

Strange..Have you tried to join all 3 new nodes and they all failed or just one the first new one ?

I suggest you re-try installing from iso and re-joining for the new nodes. All try clearing your browser cache.

If this does not solve it, can you please post the cluster config file :

cat /opt/petasan/config/cluster_info.json

Last edited on February 19, 2018, 8:03 pm by admin · #2

afrima
17 Posts
February 19, 2018, 9:49 pm

i just tried one node, since it faced this issue i didn't proceed with other two.

I tried it with a new iso image as well. but still the same problem.

here is the cluster config file:

{
"backend_1_base_ip": "192.168.11.0",
"backend_1_eth_name": "eth3",
"backend_1_mask": "255.255.255.0",
"backend_2_base_ip": "192.168.12.0",
"backend_2_eth_name": "eth2",
"backend_2_mask": "255.255.255.0",
"bonds": [],
"eth_count": 4,
"iscsi_1_eth_name": "eth2",
"iscsi_2_eth_name": "eth2",
"jumbo_frames": [
"eth2"
],
"management_eth_name": "eth1",
"management_nodes": [
{
"backend_1_ip": "192.168.11.11",
"backend_2_ip": "192.168.12.11",
"is_iscsi": true,
"is_management": true,
"is_storage": true,
"management_ip": "172.20.0.44",
"name": "Node1"
},
{
"backend_1_ip": "192.168.11.12",
"backend_2_ip": "192.168.12.12",
"is_iscsi": true,
"is_management": true,
"is_storage": true,
"management_ip": "172.20.0.45",
"name": "Node2"
},
{
"backend_1_ip": "192.168.11.13",
"backend_2_ip": "192.168.12.13",
"is_iscsi": true,
"is_management": true,
"is_storage": true,
"management_ip": "172.20.0.46",
"name": "Node3"
}
],
"name": "Peta1"
}

#3

admin
2,930 Posts
February 19, 2018, 10:38 pm

The cluster info file show the cluster has 4 nics: is this correct ? do you have eth0 to eth 3 ?

#4

afrima
17 Posts
February 20, 2018, 1:34 pm

that's correct

4 NICs on all machines 2*10GB one for backend and the other one for iscsi an 2*1GB for management (which one is only used)

Last edited on February 20, 2018, 2:25 pm by afrima · #5

admin
2,930 Posts
February 20, 2018, 3:06 pm

Hi,

Since we need all cluster nodes to have the same network setup, in addition to the nic count, the nic for the management interface needs to be the same. The management nic is setup during installation from the installer, so make sure in the installer you pick eth1 for the management interface since this is what was used for the first 3 nodes.

Let me know if you still have an issue..

#6

afrima
17 Posts
February 20, 2018, 9:02 pm
Quote from admin on February 20, 2018, 3:06 pm

Hi,

Since we need all cluster nodes to have the same network setup, in addition to the nic count, the nic for the management interface needs to be the same. The management nic is setup during installation from the installer, so make sure in the installer you pick eth1 for the management interface since this is what was used for the first 3 nodes.

Let me know if you still have an issue..

Thanks for the help. it did resolve the issue with joining the management node, then it goes all the way taking the info for backend ip and Disk assignment (journal/OSD) but there is still a problem joining the cluster in last step. (Step 8)

it does give me an error log file

#7

admin
2,930 Posts
February 20, 2018, 10:25 pm

You can post the log

cat /opt/petasan/log/PetaSAN.log

also if you can test adding the other 2 new nodes and see if they work

#8

afrima
17 Posts
February 21, 2018, 3:27 pm
Quote from admin on February 20, 2018, 10:25 pm

You can post the log

cat /opt/petasan/log/PetaSAN.log

also if you can test adding the other 2 new nodes and see if they work

sure. Here is the log file (i just put a few pages of it since it is too long):

But my question is the cluster should be able to accept any nodes at any time (not necessarily all 3 new nodes together), isn't that so?

 

21/02/2018 04:13:50 ERROR No known Consul servers
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/file_sync_manager.py", line 75, in sync
index, data = base.watch(self.root_path, current_index)
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/base.py", line 49, in watch
return cons.kv.get(key, index=current_index, recurse=True)
File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 391, in get
callback, '/v1/kv/%s' % key, params=params)
File "/usr/local/lib/python2.7/dist-packages/retry/compat.py", line 16, in wrapper
return caller(f, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/retry/api.py", line 74, in retry_decorator
logger)
File "/usr/local/lib/python2.7/dist-packages/retry/api.py", line 33, in __retry_internal
return f()
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/ps_consul.py", line 72, in get
return callback(res)
File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 377, in callback
raise ConsulException(response.body)
ConsulException: No known Consul servers
21/02/2018 04:13:52 ERROR No known Consul servers
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/file_sync_manager.py", line 75, in sync
index, data = base.watch(self.root_path, current_index)
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/base.py", line 49, in watch
return cons.kv.get(key, index=current_index, recurse=True)
File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 391, in get
callback, '/v1/kv/%s' % key, params=params)
File "/usr/local/lib/python2.7/dist-packages/retry/compat.py", line 16, in wrapper
return caller(f, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/retry/api.py", line 74, in retry_decorator
logger)
File "/usr/local/lib/python2.7/dist-packages/retry/api.py", line 33, in __retry_internal
return f()
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/ps_consul.py", line 72, in get
return callback(res)
File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 377, in callback
raise ConsulException(response.body)
ConsulException: No known Consul servers
21/02/2018 04:13:54 ERROR No known Consul servers
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/file_sync_manager.py", line 75, in sync
index, data = base.watch(self.root_path, current_index)
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/base.py", line 49, in watch
return cons.kv.get(key, index=current_index, recurse=True)
File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 391, in get
callback, '/v1/kv/%s' % key, params=params)
File "/usr/local/lib/python2.7/dist-packages/retry/compat.py", line 16, in wrapper
return caller(f, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/retry/api.py", line 74, in retry_decorator
logger)
File "/usr/local/lib/python2.7/dist-packages/retry/api.py", line 33, in __retry_internal
return f()
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/ps_consul.py", line 72, in get
return callback(res)
File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 377, in callback
raise ConsulException(response.body)
ConsulException: No known Consul servers
21/02/2018 04:13:56 ERROR No known Consul servers
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/file_sync_manager.py", line 75, in sync
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/file_sync_manager.py", line 75, in sync
index, data = base.watch(self.root_path, current_index)
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/base.py", line 49, in watch
return cons.kv.get(key, index=current_index, recurse=True)
File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 391, in get
callback, '/v1/kv/%s' % key, params=params)
File "/usr/local/lib/python2.7/dist-packages/retry/compat.py", line 16, in wrapper
return caller(f, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/retry/api.py", line 74, in retry_decorator
logger)
File "/usr/local/lib/python2.7/dist-packages/retry/api.py", line 33, in __retry_internal
return f()
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/ps_consul.py", line 72, in get
return callback(res)
File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 377, in callback
raise ConsulException(response.body)
ConsulException: No known Consul servers
21/02/2018 04:13:58 ERROR No known Consul servers
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/file_sync_manager.py", line 75, in sync
index, data = base.watch(self.root_path, current_index)
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/base.py", line 49, in watch
return cons.kv.get(key, index=current_index, recurse=True)
File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 391, in get
callback, '/v1/kv/%s' % key, params=params)
File "/usr/local/lib/python2.7/dist-packages/retry/compat.py", line 16, in wrapper
return caller(f, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/retry/api.py", line 74, in retry_decorator
logger)
File "/usr/local/lib/python2.7/dist-packages/retry/api.py", line 33, in __retry_internal
return f()
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/ps_consul.py", line 72, in get
return callback(res)
File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 377, in callback
raise ConsulException(response.body)
ConsulException: No known Consul servers
21/02/2018 04:14:00 ERROR No known Consul servers
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/file_sync_manager.py", line 75, in sync
index, data = base.watch(self.root_path, current_index)
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/base.py", line 49, in watch
return cons.kv.get(key, index=current_index, recurse=True)
File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 391, in get
callback, '/v1/kv/%s' % key, params=params)
File "/usr/local/lib/python2.7/dist-packages/retry/compat.py", line 16, in wrapper
return caller(f, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/retry/api.py", line 74, in retry_decorator
logger)
File "/usr/local/lib/python2.7/dist-packages/retry/api.py", line 33, in __retry_internal
return f()
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/ps_consul.py", line 72, in get
return callback(res)
File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 377, in callback
raise ConsulException(response.body)
ConsulException: No known Consul servers
21/02/2018 04:14:02 ERROR No known Consul servers
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/file_sync_manager.py", line 75, in sync
index, data = base.watch(self.root_path, current_index)
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/base.py", line 49, in watch
return cons.kv.get(key, index=current_index, recurse=True)
File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 391, in get
callback, '/v1/kv/%s' % key, params=params)
File "/usr/local/lib/python2.7/dist-packages/retry/compat.py", line 16, in wrapper
return caller(f, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/retry/api.py", line 74, in retry_decorator
logger)
File "/usr/local/lib/python2.7/dist-packages/retry/api.py", line 33, in __retry_internal
return f()
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/ps_consul.py", line 72, in get
return callback(res)
File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 377, in callback
raise ConsulException(response.body)
ConsulException: No known Consul servers
21/02/2018 04:14:04 ERROR No known Consul servers
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/file_sync_manager.py", line 75, in sync
index, data = base.watch(self.root_path, current_index)
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/base.py", line 49, in watch
return cons.kv.get(key, index=current_index, recurse=True)
File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 391, in get
callback, '/v1/kv/%s' % key, params=params)
File "/usr/local/lib/python2.7/dist-packages/retry/compat.py", line 16, in wrapper
return caller(f, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/retry/api.py", line 74, in retry_decorator
logger)
File "/usr/local/lib/python2.7/dist-packages/retry/api.py", line 33, in __retry_internal
return f()
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/ps_consul.py", line 72, in get
return callback(res)
File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 377, in callback
raise ConsulException(response.body)
ConsulException: No known Consul servers
21/02/2018 04:14:06 ERROR No known Consul servers
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/file_sync_manager.py", line 75, in sync
index, data = base.watch(self.root_path, current_index)
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/base.py", line 49, in watch
return cons.kv.get(key, index=current_index, recurse=True)
File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 391, in get
callback, '/v1/kv/%s' % key, params=params)
File "/usr/local/lib/python2.7/dist-packages/retry/compat.py", line 16, in wrapper
return caller(f, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/retry/api.py", line 74, in retry_decorator
logger)
File "/usr/local/lib/python2.7/dist-packages/retry/api.py", line 33, in __retry_internal
return f()
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/ps_consul.py", line 72, in get
return callback(res)
File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 377, in callback
raise ConsulException(response.body)
ConsulException: No known Consul servers
21/02/2018 04:14:08 ERROR No known Consul servers
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/file_sync_manager.py", line 75, in sync
index, data = base.watch(self.root_path, current_index)
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/base.py", line 49, in watch
return cons.kv.get(key, index=current_index, recurse=True)
File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 391, in get
callback, '/v1/kv/%s' % key, params=params)
File "/usr/local/lib/python2.7/dist-packages/retry/compat.py", line 16, in wrapper
return caller(f, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/retry/api.py", line 74, in retry_decorator
logger)
File "/usr/local/lib/python2.7/dist-packages/retry/api.py", line 33, in __retry_internal
return f()
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/ps_consul.py", line 72, in get
return callback(res)
File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 377, in callback
raise ConsulException(response.body)
ConsulException: No known Consul servers
21/02/2018 04:14:10 ERROR No known Consul servers
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/file_sync_manager.py", line 75, in sync
index, data = base.watch(self.root_path, current_index)
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/base.py", line 49, in watch
return cons.kv.get(key, index=current_index, recurse=True)
File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 391, in get
callback, '/v1/kv/%s' % key, params=params)
File "/usr/local/lib/python2.7/dist-packages/retry/compat.py", line 16, in wrapper
return caller(f, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/retry/api.py", line 74, in retry_decorator
logger)
File "/usr/local/lib/python2.7/dist-packages/retry/api.py", line 33, in __retry_internal
return f()
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/ps_consul.py", line 72, in get
return callback(res)
File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 377, in callback
raise ConsulException(response.body)
ConsulException: No known Consul servers
21/02/2018 04:14:12 ERROR No known Consul servers
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/file_sync_manager.py", line 75, in sync
index, data = base.watch(self.root_path, current_index)
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/base.py", line 49, in watch
return cons.kv.get(key, index=current_index, recurse=True)
File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 391, in get
callback, '/v1/kv/%s' % key, params=params)
File "/usr/local/lib/python2.7/dist-packages/retry/compat.py", line 16, in wrapper
return caller(f, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/retry/api.py", line 74, in retry_decorator
logger)
File "/usr/local/lib/python2.7/dist-packages/retry/api.py", line 33, in __retry_internal
return f()
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/ps_consul.py", line 72, in get
return callback(res)
File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 377, in callback
raise ConsulException(response.body)
ConsulException: No known Consul servers
21/02/2018 04:14:14 ERROR No known Consul servers
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/file_sync_manager.py", line 75, in sync
index, data = base.watch(self.root_path, current_index)
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/consul/base.py", line 49, in watch
return cons.kv.get(key, index=current_index, recurse=True)
File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 391, in get
callback, '/v1/kv/%s' % key, params=params)
File "/usr/local/lib/python2.7/dist-packages/retry/compat.py", line 16, in wrapper
return caller(f, *args, **kwargs)

............

#9

admin
2,930 Posts
February 21, 2018, 4:15 pm

Yes of course it can accept any nodes at any time.

I just want you to try the other 2 nodes ( or any one of them) to rule out any configuration issue with current node.

Also have you changed the cluster config manually, if so do let me know what changed.

Cheers Hatem

#10
Pages: 1 2