Forums

Home / Forums

You need to log in to create posts and topics. Login · Register

graphite (carbon-cache) service failing to start


DividedByPi
32 Posts
November 27, 2020, 6:32 pm

Hey guys, bit of an odd one. I brought a Petasan cluster up and am not able to get the stats script to start, when I look at whats going on it seems as though carbon-cache is failing to start.

so if I run a systemctl status carbon-cache I am seeing the following error

 

An error has occurred: b"CannotListenError: Couldn't listen on 0.0.0.0:7002: [Errno 98] Address already in use."

I have tried stopping the service, then running the stats-stop script, followed by the stats-setup script and finally stats-start but am seeing the same process failing with the same error.

Has anyone seen this?

#1

DividedByPi
32 Posts
November 27, 2020, 6:56 pm

So I have found some more here..... This could be a pretty major issue.

This cluster has about 30+ OSD's and by default they grab ports in the 6000 range, but there is nothing stopping them from grabbing port 7002 which is one of the ports that graphite uses.

So the issue I am having is that OSD's are using the ports that graphite (carbon-cache) is trying to use.

When I stop all OSD's on the node, I can then restart the stats and carbon-cache starts back up without issue.

I wonder if I am the first person to see this. The way I see it being possible to fix is to start graphite before starting OSD's at boot.

Last edited on November 27, 2020, 7:26 pm by DividedByPi · #2

admin
2,930 Posts
November 27, 2020, 8:19 pm

1) Is this a freshly installed cluster ? did the issue appear right after installation ? are you see-ing some charts and not others or no charts at all ?

2) just to double check, carbon-cache/stats server should be run on just 1 of the management nodes at a time ( active/passive)
so on all 3 management nodes, make sure it is stopped:
/opt/petasan/scripts/stats-stop.sh

then to find the designated active stats server, run
/opt/petasan/scripts/util/get_cluster_leader.py

on that node start it with:
/opt/petasan/scripts/stats-setup.sh
/opt/petasan/scripts/stats-start.sh

3) if still carbon-cache cannot start due to port in use

first make sure there is no existing carbon process running
ps aux | grep carbon-cache

check what ports are used by ceph
netstat -nlp | grep ceph

check what processes are using port 2003, 7002
netstat -pln | grep 2003
netstat -pln | grep 7002
ps aux | grep PID

Last edited on November 27, 2020, 8:20 pm by admin · #3

DividedByPi
32 Posts
November 30, 2020, 1:55 pm

Hey there,

Sorry I should have been more specific. I did post a little more in my follow up but here goes:

It was a newly built cluster, where the metrics were not working from the beginning. I was able to track down the issue pretty quick after posting by using netstat and seeing that OSD's had grabbed the port that Graphite wanted so it could not start. I am aware it is only supposed to run on a single node, however.

So - what I did to repair this was to stop all OSD's on the node that is handling stats, and then restarted all stats services with the stats-setup then stats-start scripts and once Graphite was working as intended I started the OSDs back up.

This worked thankfully! However, it seems like a pretty scary issue that Graphite uses a port within the range of ports that OSD's are allowed to use.

The only remaining issue is that not all metrics have come back in the dashboard. This is because for some reason not all folders were created in the /opt/petasan/config/shared/graphite/whisper folder. There is many folders that are missing. I am going to try to create them manually, and give ownership to _graphite to see if the metrics will begin populating.

Thanks!

Last edited on November 30, 2020, 1:55 pm by DividedByPi · #4

admin
2,930 Posts
November 30, 2020, 2:34 pm

Ceph ports should be far away from 7002, can you list the output of:

netstat -nlp | grep ceph

 

 

#5

DividedByPi
32 Posts
December 1, 2020, 6:25 pm

Hey,

It looks as though OSD's will grab ports from the range of 6800-7100 as per Ceph documentation.

The strange thing is, this cluster has 70 OSD's total however there are many more processes spawned. I sanity checked this with a very small test Petasan cluster I have virtualized with only 3 OSD's but even that cluster returned 16 PID's for ceph-osd with differnt ports used for each.

Anyways, here is the output of netstat - nlp | grep ceph on the cluster that had an OSD steal port 7001 before Graphite could take it :

tcp 0 0 0.0.0.0:7081 0.0.0.0:* LISTEN 117892/ceph-osd
tcp 0 0 0.0.0.0:7049 0.0.0.0:* LISTEN 117829/ceph-osd
tcp 0 0 0.0.0.0:7017 0.0.0.0:* LISTEN 118091/ceph-osd
tcp 0 0 0.0.0.0:6985 0.0.0.0:* LISTEN 117984/ceph-osd
tcp 0 0 0.0.0.0:6953 0.0.0.0:* LISTEN 117657/ceph-osd
tcp 0 0 0.0.0.0:6921 0.0.0.0:* LISTEN 117699/ceph-osd
tcp 0 0 0.0.0.0:6889 0.0.0.0:* LISTEN 117577/ceph-osd
tcp 0 0 0.0.0.0:6857 0.0.0.0:* LISTEN 117483/ceph-osd
tcp 0 0 0.0.0.0:6825 0.0.0.0:* LISTEN 117419/ceph-osd
tcp 0 0 0.0.0.0:7082 0.0.0.0:* LISTEN 117892/ceph-osd
tcp 0 0 0.0.0.0:7050 0.0.0.0:* LISTEN 118116/ceph-osd
tcp 0 0 0.0.0.0:7018 0.0.0.0:* LISTEN 117787/ceph-osd
tcp 0 0 0.0.0.0:6986 0.0.0.0:* LISTEN 117766/ceph-osd
tcp 0 0 0.0.0.0:6954 0.0.0.0:* LISTEN 117957/ceph-osd
tcp 0 0 0.0.0.0:6922 0.0.0.0:* LISTEN 117678/ceph-osd
tcp 0 0 0.0.0.0:6890 0.0.0.0:* LISTEN 117596/ceph-osd
tcp 0 0 0.0.0.0:6858 0.0.0.0:* LISTEN 117539/ceph-osd
tcp 0 0 0.0.0.0:6826 0.0.0.0:* LISTEN 117426/ceph-osd
tcp 0 0 0.0.0.0:7051 0.0.0.0:* LISTEN 117829/ceph-osd
tcp 0 0 0.0.0.0:7019 0.0.0.0:* LISTEN 118063/ceph-osd
tcp 0 0 0.0.0.0:6987 0.0.0.0:* LISTEN 118035/ceph-osd
tcp 0 0 0.0.0.0:6955 0.0.0.0:* LISTEN 117913/ceph-osd
tcp 0 0 0.0.0.0:6923 0.0.0.0:* LISTEN 117699/ceph-osd
tcp 0 0 0.0.0.0:6891 0.0.0.0:* LISTEN 117596/ceph-osd
tcp 0 0 0.0.0.0:6859 0.0.0.0:* LISTEN 117539/ceph-osd
tcp 0 0 0.0.0.0:6827 0.0.0.0:* LISTEN 117426/ceph-osd
tcp 0 0 0.0.0.0:7052 0.0.0.0:* LISTEN 118116/ceph-osd
tcp 0 0 0.0.0.0:7020 0.0.0.0:* LISTEN 118091/ceph-osd
tcp 0 0 0.0.0.0:6988 0.0.0.0:* LISTEN 117766/ceph-osd
tcp 0 0 0.0.0.0:6956 0.0.0.0:* LISTEN 117657/ceph-osd
tcp 0 0 0.0.0.0:6924 0.0.0.0:* LISTEN 117678/ceph-osd
tcp 0 0 0.0.0.0:6892 0.0.0.0:* LISTEN 117596/ceph-osd
tcp 0 0 0.0.0.0:6860 0.0.0.0:* LISTEN 117539/ceph-osd
tcp 0 0 0.0.0.0:6828 0.0.0.0:* LISTEN 117426/ceph-osd
tcp 0 0 0.0.0.0:7053 0.0.0.0:* LISTEN 118116/ceph-osd
tcp 0 0 0.0.0.0:7021 0.0.0.0:* LISTEN 118005/ceph-osd
tcp 0 0 0.0.0.0:6989 0.0.0.0:* LISTEN 118035/ceph-osd
tcp 0 0 0.0.0.0:6957 0.0.0.0:* LISTEN 117957/ceph-osd
tcp 0 0 0.0.0.0:6925 0.0.0.0:* LISTEN 117934/ceph-osd
tcp 0 0 0.0.0.0:6893 0.0.0.0:* LISTEN 117596/ceph-osd
tcp 0 0 0.0.0.0:6861 0.0.0.0:* LISTEN 117539/ceph-osd
tcp 0 0 0.0.0.0:6829 0.0.0.0:* LISTEN 117426/ceph-osd
tcp 0 0 0.0.0.0:7054 0.0.0.0:* LISTEN 118116/ceph-osd
tcp 0 0 0.0.0.0:7022 0.0.0.0:* LISTEN 117808/ceph-osd
tcp 0 0 0.0.0.0:6990 0.0.0.0:* LISTEN 117766/ceph-osd
tcp 0 0 0.0.0.0:6958 0.0.0.0:* LISTEN 117913/ceph-osd
tcp 0 0 0.0.0.0:6926 0.0.0.0:* LISTEN 117699/ceph-osd
tcp 0 0 0.0.0.0:6894 0.0.0.0:* LISTEN 117596/ceph-osd
tcp 0 0 0.0.0.0:6862 0.0.0.0:* LISTEN 117539/ceph-osd
tcp 0 0 0.0.0.0:6830 0.0.0.0:* LISTEN 117426/ceph-osd
tcp 0 0 0.0.0.0:7055 0.0.0.0:* LISTEN 118116/ceph-osd
tcp 0 0 0.0.0.0:7023 0.0.0.0:* LISTEN 117787/ceph-osd
tcp 0 0 0.0.0.0:6991 0.0.0.0:* LISTEN 118035/ceph-osd
tcp 0 0 0.0.0.0:6959 0.0.0.0:* LISTEN 117657/ceph-osd
tcp 0 0 0.0.0.0:6927 0.0.0.0:* LISTEN 117934/ceph-osd
tcp 0 0 0.0.0.0:6895 0.0.0.0:* LISTEN 117596/ceph-osd
tcp 0 0 0.0.0.0:6863 0.0.0.0:* LISTEN 117539/ceph-osd
tcp 0 0 0.0.0.0:6831 0.0.0.0:* LISTEN 117426/ceph-osd
tcp 0 0 0.0.0.0:7056 0.0.0.0:* LISTEN 118116/ceph-osd
tcp 0 0 0.0.0.0:7024 0.0.0.0:* LISTEN 118005/ceph-osd
tcp 0 0 0.0.0.0:6992 0.0.0.0:* LISTEN 117766/ceph-osd
tcp 0 0 0.0.0.0:6960 0.0.0.0:* LISTEN 117957/ceph-osd
tcp 0 0 0.0.0.0:6928 0.0.0.0:* LISTEN 117699/ceph-osd
tcp 0 0 0.0.0.0:6896 0.0.0.0:* LISTEN 117596/ceph-osd
tcp 0 0 0.0.0.0:6864 0.0.0.0:* LISTEN 117539/ceph-osd
tcp 0 0 0.0.0.0:6832 0.0.0.0:* LISTEN 117426/ceph-osd
tcp 0 0 0.0.0.0:6800 0.0.0.0:* LISTEN 5673/ceph-mds
tcp 0 0 0.0.0.0:7057 0.0.0.0:* LISTEN 118116/ceph-osd
tcp 0 0 0.0.0.0:7025 0.0.0.0:* LISTEN 118063/ceph-osd
tcp 0 0 0.0.0.0:6993 0.0.0.0:* LISTEN 118035/ceph-osd
tcp 0 0 0.0.0.0:6961 0.0.0.0:* LISTEN 117913/ceph-osd
tcp 0 0 0.0.0.0:6929 0.0.0.0:* LISTEN 117934/ceph-osd
tcp 0 0 0.0.0.0:6897 0.0.0.0:* LISTEN 117596/ceph-osd
tcp 0 0 0.0.0.0:6865 0.0.0.0:* LISTEN 117539/ceph-osd
tcp 0 0 0.0.0.0:6833 0.0.0.0:* LISTEN 117426/ceph-osd
tcp 0 0 0.0.0.0:6801 0.0.0.0:* LISTEN 5673/ceph-mds
tcp 0 0 0.0.0.0:7058 0.0.0.0:* LISTEN 118116/ceph-osd
tcp 0 0 0.0.0.0:7026 0.0.0.0:* LISTEN 118091/ceph-osd
tcp 0 0 0.0.0.0:6994 0.0.0.0:* LISTEN 117766/ceph-osd
tcp 0 0 0.0.0.0:6962 0.0.0.0:* LISTEN 117743/ceph-osd
tcp 0 0 0.0.0.0:6930 0.0.0.0:* LISTEN 117699/ceph-osd
tcp 0 0 0.0.0.0:6898 0.0.0.0:* LISTEN 117615/ceph-osd
tcp 0 0 0.0.0.0:6866 0.0.0.0:* LISTEN 117522/ceph-osd
tcp 0 0 0.0.0.0:6834 0.0.0.0:* LISTEN 117464/ceph-osd
tcp 0 0 0.0.0.0:6802 0.0.0.0:* LISTEN 115337/ceph-osd
tcp 0 0 0.0.0.0:7059 0.0.0.0:* LISTEN 117871/ceph-osd
tcp 0 0 0.0.0.0:7027 0.0.0.0:* LISTEN 117808/ceph-osd
tcp 0 0 0.0.0.0:6995 0.0.0.0:* LISTEN 118035/ceph-osd
tcp 0 0 0.0.0.0:6963 0.0.0.0:* LISTEN 117743/ceph-osd
tcp 0 0 0.0.0.0:6931 0.0.0.0:* LISTEN 117934/ceph-osd
tcp 0 0 0.0.0.0:6899 0.0.0.0:* LISTEN 117615/ceph-osd
tcp 0 0 0.0.0.0:6867 0.0.0.0:* LISTEN 117522/ceph-osd
tcp 0 0 0.0.0.0:6835 0.0.0.0:* LISTEN 117464/ceph-osd
tcp 0 0 0.0.0.0:6803 0.0.0.0:* LISTEN 115337/ceph-osd
tcp 0 0 0.0.0.0:7060 0.0.0.0:* LISTEN 117871/ceph-osd
tcp 0 0 0.0.0.0:7028 0.0.0.0:* LISTEN 117829/ceph-osd
tcp 0 0 0.0.0.0:6996 0.0.0.0:* LISTEN 117766/ceph-osd
tcp 0 0 0.0.0.0:6964 0.0.0.0:* LISTEN 117720/ceph-osd
tcp 0 0 0.0.0.0:6932 0.0.0.0:* LISTEN 117699/ceph-osd
tcp 0 0 0.0.0.0:6900 0.0.0.0:* LISTEN 117615/ceph-osd
tcp 0 0 0.0.0.0:6868 0.0.0.0:* LISTEN 117522/ceph-osd
tcp 0 0 0.0.0.0:6836 0.0.0.0:* LISTEN 117464/ceph-osd
tcp 0 0 0.0.0.0:6804 0.0.0.0:* LISTEN 115337/ceph-osd
tcp 0 0 0.0.0.0:7061 0.0.0.0:* LISTEN 117871/ceph-osd
tcp 0 0 0.0.0.0:7029 0.0.0.0:* LISTEN 118063/ceph-osd
tcp 0 0 0.0.0.0:6997 0.0.0.0:* LISTEN 118035/ceph-osd
tcp 0 0 0.0.0.0:6965 0.0.0.0:* LISTEN 117743/ceph-osd
tcp 0 0 0.0.0.0:6933 0.0.0.0:* LISTEN 117934/ceph-osd
tcp 0 0 0.0.0.0:6901 0.0.0.0:* LISTEN 117615/ceph-osd
tcp 0 0 0.0.0.0:6869 0.0.0.0:* LISTEN 117522/ceph-osd
tcp 0 0 0.0.0.0:6837 0.0.0.0:* LISTEN 117464/ceph-osd
tcp 0 0 0.0.0.0:6805 0.0.0.0:* LISTEN 115337/ceph-osd
tcp 0 0 0.0.0.0:7062 0.0.0.0:* LISTEN 117850/ceph-osd
tcp 0 0 0.0.0.0:7030 0.0.0.0:* LISTEN 118005/ceph-osd
tcp 0 0 0.0.0.0:6998 0.0.0.0:* LISTEN 117766/ceph-osd
tcp 0 0 0.0.0.0:6966 0.0.0.0:* LISTEN 117720/ceph-osd
tcp 0 0 0.0.0.0:6934 0.0.0.0:* LISTEN 117699/ceph-osd
tcp 0 0 0.0.0.0:6902 0.0.0.0:* LISTEN 117615/ceph-osd
tcp 0 0 0.0.0.0:6870 0.0.0.0:* LISTEN 117522/ceph-osd
tcp 0 0 0.0.0.0:6838 0.0.0.0:* LISTEN 117464/ceph-osd
tcp 0 0 0.0.0.0:6806 0.0.0.0:* LISTEN 115337/ceph-osd
tcp 0 0 0.0.0.0:7063 0.0.0.0:* LISTEN 117871/ceph-osd
tcp 0 0 0.0.0.0:7031 0.0.0.0:* LISTEN 118091/ceph-osd
tcp 0 0 0.0.0.0:6999 0.0.0.0:* LISTEN 118035/ceph-osd
tcp 0 0 0.0.0.0:6967 0.0.0.0:* LISTEN 117743/ceph-osd
tcp 0 0 0.0.0.0:6935 0.0.0.0:* LISTEN 117934/ceph-osd
tcp 0 0 0.0.0.0:6903 0.0.0.0:* LISTEN 117615/ceph-osd
tcp 0 0 0.0.0.0:6871 0.0.0.0:* LISTEN 117522/ceph-osd
tcp 0 0 0.0.0.0:6839 0.0.0.0:* LISTEN 117464/ceph-osd
tcp 0 0 0.0.0.0:6807 0.0.0.0:* LISTEN 115337/ceph-osd
tcp 0 0 0.0.0.0:7064 0.0.0.0:* LISTEN 117850/ceph-osd
tcp 0 0 0.0.0.0:7032 0.0.0.0:* LISTEN 117808/ceph-osd
tcp 0 0 0.0.0.0:7000 0.0.0.0:* LISTEN 117766/ceph-osd
tcp 0 0 0.0.0.0:6968 0.0.0.0:* LISTEN 117720/ceph-osd
tcp 0 0 0.0.0.0:6936 0.0.0.0:* LISTEN 117934/ceph-osd
tcp 0 0 0.0.0.0:6904 0.0.0.0:* LISTEN 117615/ceph-osd
tcp 0 0 0.0.0.0:6872 0.0.0.0:* LISTEN 117522/ceph-osd
tcp 0 0 0.0.0.0:6840 0.0.0.0:* LISTEN 117464/ceph-osd
tcp 0 0 0.0.0.0:6808 0.0.0.0:* LISTEN 115337/ceph-osd
tcp 0 0 0.0.0.0:7065 0.0.0.0:* LISTEN 117871/ceph-osd
tcp 0 0 0.0.0.0:7033 0.0.0.0:* LISTEN 117829/ceph-osd
tcp 0 0 0.0.0.0:7001 0.0.0.0:* LISTEN 118035/ceph-osd
tcp 0 0 0.0.0.0:6969 0.0.0.0:* LISTEN 117720/ceph-osd
tcp 0 0 0.0.0.0:6937 0.0.0.0:* LISTEN 117934/ceph-osd
tcp 0 0 0.0.0.0:6905 0.0.0.0:* LISTEN 117615/ceph-osd
tcp 0 0 0.0.0.0:6873 0.0.0.0:* LISTEN 117522/ceph-osd
tcp 0 0 0.0.0.0:6841 0.0.0.0:* LISTEN 117464/ceph-osd
tcp 0 0 0.0.0.0:6809 0.0.0.0:* LISTEN 115337/ceph-osd
tcp 0 0 0.0.0.0:7066 0.0.0.0:* LISTEN 117850/ceph-osd
tcp 0 0 0.0.0.0:7034 0.0.0.0:* LISTEN 118063/ceph-osd
tcp 0 0 0.0.0.0:6970 0.0.0.0:* LISTEN 117743/ceph-osd
tcp 0 0 0.0.0.0:6938 0.0.0.0:* LISTEN 117657/ceph-osd
tcp 0 0 0.0.0.0:6906 0.0.0.0:* LISTEN 117636/ceph-osd
tcp 0 0 0.0.0.0:6874 0.0.0.0:* LISTEN 117558/ceph-osd
tcp 0 0 0.0.0.0:6842 0.0.0.0:* LISTEN 117503/ceph-osd
tcp 0 0 0.0.0.0:6810 0.0.0.0:* LISTEN 117445/ceph-osd
tcp 0 0 0.0.0.0:7067 0.0.0.0:* LISTEN 117871/ceph-osd
tcp 0 0 0.0.0.0:7035 0.0.0.0:* LISTEN 118005/ceph-osd
tcp 0 0 0.0.0.0:7003 0.0.0.0:* LISTEN 117787/ceph-osd
tcp 0 0 0.0.0.0:6971 0.0.0.0:* LISTEN 117984/ceph-osd
tcp 0 0 0.0.0.0:6939 0.0.0.0:* LISTEN 117957/ceph-osd
tcp 0 0 0.0.0.0:6907 0.0.0.0:* LISTEN 117636/ceph-osd
tcp 0 0 0.0.0.0:6875 0.0.0.0:* LISTEN 117558/ceph-osd
tcp 0 0 0.0.0.0:6843 0.0.0.0:* LISTEN 117503/ceph-osd
tcp 0 0 0.0.0.0:6811 0.0.0.0:* LISTEN 117445/ceph-osd
tcp 0 0 0.0.0.0:7068 0.0.0.0:* LISTEN 117850/ceph-osd
tcp 0 0 0.0.0.0:7036 0.0.0.0:* LISTEN 118091/ceph-osd
tcp 0 0 0.0.0.0:7004 0.0.0.0:* LISTEN 117787/ceph-osd
tcp 0 0 0.0.0.0:6972 0.0.0.0:* LISTEN 117720/ceph-osd
tcp 0 0 0.0.0.0:6940 0.0.0.0:* LISTEN 117913/ceph-osd
tcp 0 0 0.0.0.0:6908 0.0.0.0:* LISTEN 117636/ceph-osd
tcp 0 0 0.0.0.0:6876 0.0.0.0:* LISTEN 117558/ceph-osd
tcp 0 0 0.0.0.0:6844 0.0.0.0:* LISTEN 117503/ceph-osd
tcp 0 0 0.0.0.0:6812 0.0.0.0:* LISTEN 117445/ceph-osd
tcp 0 0 0.0.0.0:7069 0.0.0.0:* LISTEN 117871/ceph-osd
tcp 0 0 0.0.0.0:7037 0.0.0.0:* LISTEN 117808/ceph-osd
tcp 0 0 0.0.0.0:7005 0.0.0.0:* LISTEN 117787/ceph-osd
tcp 0 0 0.0.0.0:6973 0.0.0.0:* LISTEN 117743/ceph-osd
tcp 0 0 0.0.0.0:6941 0.0.0.0:* LISTEN 117957/ceph-osd
tcp 0 0 0.0.0.0:6909 0.0.0.0:* LISTEN 117636/ceph-osd
tcp 0 0 0.0.0.0:6877 0.0.0.0:* LISTEN 117558/ceph-osd
tcp 0 0 0.0.0.0:6845 0.0.0.0:* LISTEN 117503/ceph-osd
tcp 0 0 0.0.0.0:6813 0.0.0.0:* LISTEN 117445/ceph-osd
tcp 0 0 0.0.0.0:7070 0.0.0.0:* LISTEN 117850/ceph-osd
tcp 0 0 0.0.0.0:7038 0.0.0.0:* LISTEN 117829/ceph-osd
tcp 0 0 0.0.0.0:7006 0.0.0.0:* LISTEN 118091/ceph-osd
tcp 0 0 0.0.0.0:6974 0.0.0.0:* LISTEN 117984/ceph-osd
tcp 0 0 0.0.0.0:6942 0.0.0.0:* LISTEN 117657/ceph-osd
tcp 0 0 0.0.0.0:6910 0.0.0.0:* LISTEN 117636/ceph-osd
tcp 0 0 0.0.0.0:6878 0.0.0.0:* LISTEN 117558/ceph-osd
tcp 0 0 0.0.0.0:6846 0.0.0.0:* LISTEN 117503/ceph-osd
tcp 0 0 0.0.0.0:6814 0.0.0.0:* LISTEN 117445/ceph-osd
tcp 0 0 0.0.0.0:7071 0.0.0.0:* LISTEN 117892/ceph-osd
tcp 0 0 0.0.0.0:7039 0.0.0.0:* LISTEN 118063/ceph-osd
tcp 0 0 0.0.0.0:7007 0.0.0.0:* LISTEN 117787/ceph-osd
tcp 0 0 0.0.0.0:6975 0.0.0.0:* LISTEN 117720/ceph-osd
tcp 0 0 0.0.0.0:6943 0.0.0.0:* LISTEN 117913/ceph-osd
tcp 0 0 0.0.0.0:6911 0.0.0.0:* LISTEN 117636/ceph-osd
tcp 0 0 0.0.0.0:6879 0.0.0.0:* LISTEN 117558/ceph-osd
tcp 0 0 0.0.0.0:6847 0.0.0.0:* LISTEN 117503/ceph-osd
tcp 0 0 0.0.0.0:6815 0.0.0.0:* LISTEN 117445/ceph-osd
tcp 0 0 0.0.0.0:7072 0.0.0.0:* LISTEN 117871/ceph-osd
tcp 0 0 0.0.0.0:7040 0.0.0.0:* LISTEN 118005/ceph-osd
tcp 0 0 0.0.0.0:7008 0.0.0.0:* LISTEN 118091/ceph-osd
tcp 0 0 0.0.0.0:6976 0.0.0.0:* LISTEN 117743/ceph-osd
tcp 0 0 0.0.0.0:6944 0.0.0.0:* LISTEN 117657/ceph-osd
tcp 0 0 0.0.0.0:6912 0.0.0.0:* LISTEN 117636/ceph-osd
tcp 0 0 0.0.0.0:6880 0.0.0.0:* LISTEN 117558/ceph-osd
tcp 0 0 0.0.0.0:6848 0.0.0.0:* LISTEN 117503/ceph-osd
tcp 0 0 0.0.0.0:6816 0.0.0.0:* LISTEN 117445/ceph-osd
tcp 0 0 0.0.0.0:7073 0.0.0.0:* LISTEN 117850/ceph-osd
tcp 0 0 0.0.0.0:7041 0.0.0.0:* LISTEN 117808/ceph-osd
tcp 0 0 0.0.0.0:7009 0.0.0.0:* LISTEN 117787/ceph-osd
tcp 0 0 0.0.0.0:6977 0.0.0.0:* LISTEN 117984/ceph-osd
tcp 0 0 0.0.0.0:6945 0.0.0.0:* LISTEN 117957/ceph-osd
tcp 0 0 0.0.0.0:6913 0.0.0.0:* LISTEN 117636/ceph-osd
tcp 0 0 0.0.0.0:6881 0.0.0.0:* LISTEN 117558/ceph-osd
tcp 0 0 0.0.0.0:6849 0.0.0.0:* LISTEN 117503/ceph-osd
tcp 0 0 0.0.0.0:6817 0.0.0.0:* LISTEN 117445/ceph-osd
tcp 0 0 0.0.0.0:7074 0.0.0.0:* LISTEN 117892/ceph-osd
tcp 0 0 0.0.0.0:7042 0.0.0.0:* LISTEN 117829/ceph-osd
tcp 0 0 0.0.0.0:7010 0.0.0.0:* LISTEN 117808/ceph-osd
tcp 0 0 0.0.0.0:6978 0.0.0.0:* LISTEN 117720/ceph-osd
tcp 0 0 0.0.0.0:6946 0.0.0.0:* LISTEN 117913/ceph-osd
tcp 0 0 0.0.0.0:6914 0.0.0.0:* LISTEN 117678/ceph-osd
tcp 0 0 0.0.0.0:6882 0.0.0.0:* LISTEN 117577/ceph-osd
tcp 0 0 0.0.0.0:6850 0.0.0.0:* LISTEN 117483/ceph-osd
tcp 0 0 0.0.0.0:6818 0.0.0.0:* LISTEN 117419/ceph-osd
tcp 0 0 0.0.0.0:7075 0.0.0.0:* LISTEN 117850/ceph-osd
tcp 0 0 0.0.0.0:7043 0.0.0.0:* LISTEN 118063/ceph-osd
tcp 0 0 0.0.0.0:7011 0.0.0.0:* LISTEN 118091/ceph-osd
tcp 0 0 0.0.0.0:6979 0.0.0.0:* LISTEN 117743/ceph-osd
tcp 0 0 0.0.0.0:6947 0.0.0.0:* LISTEN 117657/ceph-osd
tcp 0 0 0.0.0.0:6915 0.0.0.0:* LISTEN 117678/ceph-osd
tcp 0 0 0.0.0.0:6883 0.0.0.0:* LISTEN 117577/ceph-osd
tcp 0 0 0.0.0.0:6851 0.0.0.0:* LISTEN 117483/ceph-osd
tcp 0 0 0.0.0.0:6819 0.0.0.0:* LISTEN 117419/ceph-osd
tcp 0 0 0.0.0.0:7076 0.0.0.0:* LISTEN 117892/ceph-osd
tcp 0 0 0.0.0.0:7044 0.0.0.0:* LISTEN 118005/ceph-osd
tcp 0 0 0.0.0.0:7012 0.0.0.0:* LISTEN 118005/ceph-osd
tcp 0 0 0.0.0.0:6980 0.0.0.0:* LISTEN 117984/ceph-osd
tcp 0 0 0.0.0.0:6948 0.0.0.0:* LISTEN 117957/ceph-osd
tcp 0 0 0.0.0.0:6916 0.0.0.0:* LISTEN 117678/ceph-osd
tcp 0 0 0.0.0.0:6884 0.0.0.0:* LISTEN 117577/ceph-osd
tcp 0 0 0.0.0.0:6852 0.0.0.0:* LISTEN 117483/ceph-osd
tcp 0 0 0.0.0.0:6820 0.0.0.0:* LISTEN 117419/ceph-osd
tcp 0 0 10.35.70.10:3300 0.0.0.0:* LISTEN 5674/ceph-mon
tcp 0 0 0.0.0.0:7077 0.0.0.0:* LISTEN 117850/ceph-osd
tcp 0 0 0.0.0.0:7045 0.0.0.0:* LISTEN 117808/ceph-osd
tcp 0 0 0.0.0.0:7013 0.0.0.0:* LISTEN 117787/ceph-osd
tcp 0 0 0.0.0.0:6981 0.0.0.0:* LISTEN 117720/ceph-osd
tcp 0 0 0.0.0.0:6949 0.0.0.0:* LISTEN 117913/ceph-osd
tcp 0 0 0.0.0.0:6917 0.0.0.0:* LISTEN 117678/ceph-osd
tcp 0 0 0.0.0.0:6885 0.0.0.0:* LISTEN 117577/ceph-osd
tcp 0 0 0.0.0.0:6853 0.0.0.0:* LISTEN 117483/ceph-osd
tcp 0 0 0.0.0.0:6821 0.0.0.0:* LISTEN 117419/ceph-osd
tcp 0 0 10.35.70.10:6789 0.0.0.0:* LISTEN 5674/ceph-mon
tcp 0 0 0.0.0.0:7078 0.0.0.0:* LISTEN 117892/ceph-osd
tcp 0 0 0.0.0.0:7046 0.0.0.0:* LISTEN 117829/ceph-osd
tcp 0 0 0.0.0.0:7014 0.0.0.0:* LISTEN 118063/ceph-osd
tcp 0 0 0.0.0.0:6982 0.0.0.0:* LISTEN 117984/ceph-osd
tcp 0 0 0.0.0.0:6950 0.0.0.0:* LISTEN 117657/ceph-osd
tcp 0 0 0.0.0.0:6918 0.0.0.0:* LISTEN 117678/ceph-osd
tcp 0 0 0.0.0.0:6886 0.0.0.0:* LISTEN 117577/ceph-osd
tcp 0 0 0.0.0.0:6854 0.0.0.0:* LISTEN 117483/ceph-osd
tcp 0 0 0.0.0.0:6822 0.0.0.0:* LISTEN 117419/ceph-osd
tcp 0 0 0.0.0.0:7079 0.0.0.0:* LISTEN 117892/ceph-osd
tcp 0 0 0.0.0.0:7047 0.0.0.0:* LISTEN 118063/ceph-osd
tcp 0 0 0.0.0.0:7015 0.0.0.0:* LISTEN 117808/ceph-osd
tcp 0 0 0.0.0.0:6983 0.0.0.0:* LISTEN 117984/ceph-osd
tcp 0 0 0.0.0.0:6951 0.0.0.0:* LISTEN 117957/ceph-osd
tcp 0 0 0.0.0.0:6919 0.0.0.0:* LISTEN 117678/ceph-osd
tcp 0 0 0.0.0.0:6887 0.0.0.0:* LISTEN 117577/ceph-osd
tcp 0 0 0.0.0.0:6855 0.0.0.0:* LISTEN 117483/ceph-osd
tcp 0 0 0.0.0.0:6823 0.0.0.0:* LISTEN 117419/ceph-osd
tcp 0 0 0.0.0.0:7080 0.0.0.0:* LISTEN 117892/ceph-osd
tcp 0 0 0.0.0.0:7048 0.0.0.0:* LISTEN 117829/ceph-osd
tcp 0 0 0.0.0.0:7016 0.0.0.0:* LISTEN 118005/ceph-osd
tcp 0 0 0.0.0.0:6984 0.0.0.0:* LISTEN 117984/ceph-osd
tcp 0 0 0.0.0.0:6952 0.0.0.0:* LISTEN 117913/ceph-osd
tcp 0 0 0.0.0.0:6920 0.0.0.0:* LISTEN 117699/ceph-osd
tcp 0 0 0.0.0.0:6888 0.0.0.0:* LISTEN 117577/ceph-osd
tcp 0 0 0.0.0.0:6856 0.0.0.0:* LISTEN 117483/ceph-osd
tcp 0 0 0.0.0.0:6824 0.0.0.0:* LISTEN 117419/ceph-osd

Last edited on December 1, 2020, 6:26 pm by DividedByPi · #6

admin
2,930 Posts
December 1, 2020, 9:46 pm

Thanks for output. Yes there is a chance of port conflict if you have a large number of OSDs (above 20+) on each of the Management nodes (first 3 nodes) which also run the carbon-cache service. Each OSD process can have 10 ports for async messagener, starting at port 6800, some of the config settings:

"ms_bind_port_max": "7300",
"ms_bind_port_min": "6800",
"ms_async_max_op_threads": "5",
"ms_async_op_threads": "3",

The ports assigned start at 6800 and increment rather than being random. Can you verify the netstat output you listed has 20+ OSDs ?

If you have a large number of OSD on the first three ndoes, you can re-configure the carbon-cache port from 7002 to 17002

on first 3 nodes

nano /opt/petasan/config/stats/carbon/carbon.conf
change line 115
CACHE_QUERY_PORT = 17002

nano /opt/petasan/config/stats/graphite/local_settings.py
add at top
CARBONLINK_HOSTS = ['127.0.0.1:17002']

then to find the designated active stats server, run

/opt/petasan/scripts/util/get_cluster_leader.py

on that node:

/opt/petasan/scripts/stats-stop.sh
/opt/petasan/scripts/stats-setup.sh
/opt/petasan/scripts/stats-start.sh

#7