Forums

Home / Forums

You need to log in to create posts and topics. Login · Register

Consul too many open files


roy.lei
2 Posts
December 15, 2020, 7:24 pm

Hi,

Consul on one of the management nodes was logging "too many open files" to syslog. Nodes reboot fixed the issue.

I am running PetaSAN version 2.5.0.

Is there any way to prevent that?

Thanks

Dec 16 05:07:34 ps-ak-management-02 files_sync.py[1656]: -------------------------------------
Dec 16 05:07:34 ps-ak-management-02 files_sync.py[1656]: ('updated current_index: ', '5243087')
Dec 16 05:07:34 ps-ak-management-02 files_sync.py[1656]: file_path
Dec 16 05:07:34 ps-ak-management-02 files_sync.py[1656]: /etc/hosts
Dec 16 05:07:34 ps-ak-management-02 files_sync.py[1656]: Gonna write file
Dec 16 05:07:34 ps-ak-management-02 files_sync.py[1656]: wrote file
Dec 16 05:07:34 ps-ak-management-02 consul[1237]: consul.rpc: failed to accept RPC conn: accept tcp 10.10.0.52:8300: accept4: too many open files
Dec 16 05:07:34 ps-ak-management-02 files_sync.py[1656]: ('index', '5243087')
Dec 16 05:07:34 ps-ak-management-02 consul[1237]: consul.rpc: failed to accept RPC conn: accept tcp 10.10.0.52:8300: accept4: too many open files
Dec 16 05:07:35 ps-ak-management-02 consul[1237]: message repeated 26299 times: [ consul.rpc: failed to accept RPC conn: accept tcp 10.10.0.52:8300: accept4: too many open files]
Dec 16 05:07:35 ps-ak-management-02 consul[1237]: memberlist: Error accepting TCP connection: accept tcp 10.10.0.52:8301: accept4: too many open files
Dec 16 05:07:35 ps-ak-management-02 consul[1237]: consul.rpc: failed to accept RPC conn: accept tcp 10.10.0.52:8300: accept4: too many open files
Dec 16 05:07:36 ps-ak-management-02 consul[1237]: message repeated 50804 times: [ consul.rpc: failed to accept RPC conn: accept tcp 10.10.0.52:8300: accept4: too many open files]
Dec 16 05:07:36 ps-ak-management-02 consul[1237]: memberlist: Error accepting TCP connection: accept tcp 10.10.0.52:8301: accept4: too many open files
Dec 16 05:07:36 ps-ak-management-02 consul[1237]: consul.rpc: failed to accept RPC conn: accept tcp 10.10.0.52:8300: accept4: too many open files
Dec 16 05:07:37 ps-ak-management-02 consul[1237]: message repeated 53399 times: [ consul.rpc: failed to accept RPC conn: accept tcp 10.10.0.52:8300: accept4: too many open files]
Dec 16 05:07:37 ps-ak-management-02 consul[1237]: memberlist: Error accepting TCP connection: accept tcp 10.10.0.52:8301: accept4: too many open files
PC conn: accept tcp 10.10.0.52:8300: accept4: too many open filesled to accept R--More--(17%)
Dec 16 05:07:38 ps-ak-management-02 consul[1237]: message repeated 45804 times: [ consul.rpc: failed to accept RPC conn: accept tcp 10.10.0.52:8300: accept4: too many open files]
Dec 16 05:07:38 ps-ak-management-02 consul[1237]: memberlist: Error accepting TCP connection: accept tcp 10.10.0.52:8301: accept4: too many open files
Dec 16 05:07:38 ps-ak-management-02 consul[1237]: consul.rpc: failed to accept RPC conn: accept tcp 10.10.0.52:8300: accept4: too many open files
Dec 16 05:07:39 ps-ak-management-02 consul[1237]: message repeated 45223 times: [ consul.rpc: failed to accept RPC conn: accept tcp 10.10.0.52:8300: accept4: too many open files]

Dec 16 05:07:39 ps-ak-management-02 consul[1237]: memberlist: Error accepting TCP connection: accept tcp 10.10.0.52:8301: accept4: too many open files
Dec 16 05:07:39 ps-ak-management-02 consul[1237]: consul.rpc: failed to accept RPC conn: accept tcp 10.10.0.52:8300: accept4: too many open files
Dec 16 05:07:40 ps-ak-management-02 consul[1237]: message repeated 53037 times: [ consul.rpc: failed to accept RPC conn: accept tcp 10.10.0.52:8300: accept4: too many open files]
Dec 16 05:07:40 ps-ak-management-02 consul[1237]: memberlist: Error accepting TCP connection: accept tcp 10.10.0.52:8301: accept4: too many open files
Dec 16 05:07:40 ps-ak-management-02 consul[1237]: consul.rpc: failed to accept RPC conn: accept tcp 10.10.0.52:8300: accept4: too many open files
[ consul.rpc: failed to accept RPC conn: accept tcp 10.10.0.52:8300: accept4: too many open files]
Dec 16 05:07:41 ps-ak-management-02 consul[1237]: memberlist: Error accepting TCP connection: accept tcp 10.10.0.52:8301: accept4: too many open files
Dec 16 05:07:41 ps-ak-management-02 consul[1237]: consul.rpc: failed to accept RPC conn: accept tcp 10.10.0.52:8300: accept4: too many open files
Dec 16 05:07:41 ps-ak-management-02 consul[1237]: message repeated 23893 times: [ consul.rpc: failed to accept RPC conn: accept tcp 10.10.0.52:8300: accept4: too many open files]
Dec 16 05:07:41 ps-ak-management-02 consul[1237]: memberlist: Push/Pull with ps-ak-iscsi-04 failed: dial tcp 10.10.0.204:8301: socket: too many open files
Dec 16 05:07:41 ps-ak-management-02 consul[1237]: consul.rpc: failed to accept RPC conn: accept tcp 10.10.0.52:8300: accept4: too many open files
Dec 16 05:07:42 ps-ak-management-02 consul[1237]: message repeated 20797 times: [ consul.rpc: failed to accept RPC conn: accept tcp 10.10.0.52:8300: accept4: too many open files]
Dec 16 05:07:42 ps-ak-management-02 consul[1237]: memberlist: Error accepting TCP connection: accept tcp 10.10.0.52:8301: accept4: too many open files
Dec 16 05:07:42 ps-ak-management-02 consul[1237]: consul.rpc: failed to accept RPC conn: accept tcp 10.10.0.52:8300: accept4: too many open files
Dec 16 05:07:43 ps-ak-management-02 consul[1237]: message repeated 57888 times: [ consul.rpc: failed to accept RPC conn: accept tcp 10.10.0.52:8300: accept4: too many open files]
Dec 16 05:07:43 ps-ak-management-02 consul[1237]: memberlist: Error accepting TCP connection: accept tcp 10.10.0.52:8301: accept4: too many open files
Dec 16 05:07:43 ps-ak-management-02 consul[1237]: consul.rpc: failed to accept RPC conn: accept tcp 10.10.0.52:8300: accept4: too many open files

#1

therm
121 Posts
December 16, 2020, 8:39 am

Hi,

we faced this error several times. The bad was that it often came in combination with fencing-feature shutting down the good nodes.

In the end we got rid of that error by moving ISCSI-Service to 2 dedicated ISCSI-Servers (only iscsi, no osd, no mon, and ISCSI disabled on all other nodes).

#2