Can Cluster Servers connect on https?

Asked by Nick

I have 3 graphite servers all running on https, however when they try to connect to each other they fail (I do not see any errors in the logs though) - I know they fail because the data from the other nodes are not in the one graphite node I'm currently looking at.

I tried:

CLUSTER_SERVERS = ["ip_address:443", "ip_address:443"]

and tried:

CLUSTER_SERVERS = ["https://ip_address", "https://ip_address"]

-- When I migrated apache on all of the graphite nodes to use 80 only, the usual config worked:

CLUSTER_SERVERS = ["ip_address", "ip_address"]

How can I make sure Graphite can connect to another node on 443?

Question information

Language:
English Edit question
Status:
Answered
For:
Graphite Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Nicholas Leskiw (nleskiw) said :
#1

I've never used clustering, but I'm pretty sure that cluster servers do not talk HTTP. It uses it's own protocols defined in /opt/graphite/lib/carbon/protocols.py

(Someone who's used or is using clustering please verify...)

Revision history for this message
chrismd (chrismd) said :
#2

Cluster communication happens through various channels. The carbon daemons do not speak http to one another but the webapps do. The CLUSTER_SERVERS setting lists other webapps in the cluster but only http is supported for webapp-to-webapp communication. There is a lot of overhead to establishing https connections and the connections are not persistent (there is no pooling) so it would not scale at all. If you really need to have communications between webapps encrypted then I would suggest setting up a lower-level secure transport like a vpn or an ssh tunnel.

In general, Graphite is designed to be run in a trusted environment by trusted users. It is very easy to denial-of-service Graphite if that is one's goal.

Revision history for this message
Cody Stevens (cody-stevens) said :
#3

I was having this problem as well and solved it by changing my apache configuration. I have posted it below, hope it helps you.

NameVirtualHost 10.49.0.45:443
NameVirtualHost 10.49.0.45:80

# You may need to manually edit this file to fit your needs.
# This configuration assumes the default installation prefix
# of /opt/graphite/, if you installed graphite somewhere else
# you will need to change all the occurances of /opt/graphite/
# in this file to your chosen install location.

LoadModule wsgi_module modules/mod_wsgi.so
WSGISocketPrefix /var/run

WSGIDaemonProcess graphite processes=5 threads=5 display-name='%{GROUP}' inactivity-timeout=120 python-path=/usr/local/lib/
WSGIProcessGroup graphite

<VirtualHost 10.49.0.45:80>
        ServerName graphch01.int.domain.com
        DocumentRoot "/usr/local/webapp"

        # I've found that an equal number of processes & threads tends
        # to show the best performance for Graphite (ymmv).

        # You will need to create this file! There is a graphite.wsgi.example
        # file in this directory that you can safely use, just copy it to graphite.wgsi
        WSGIScriptAlias / /usr/local/conf/graphite.wsgi

        Alias /content/ /usr/local/webapp/content/
        <Location "/content/">
                SetHandler None
        </Location>

        # NOTE: In order for the django admin site media to work you
        # must change @DJANGO_ROOT@ to be the path to your django
        # installation, which is probably something like:
        # /usr/lib/python2.6/site-packages/django
        Alias /media/ "/usr/local/lib/python2.6/site-packages/django/contrib/admin/media/"
        <Location "/media/">
                SetHandler None
        </Location>

        # The graphite.wsgi file has to be accessible by apache. It won't
        # be visible to clients because of the DocumentRoot though.
        <Directory /usr/local/conf/>
                Order deny,allow
                Allow from all
        </Directory>

</VirtualHost>

<VirtualHost 10.49.0.45:443>
        SSLEngine On
        SSLCertificateFile /etc/pki/tls/certs/wildcard.crt
        SSLCertificateKeyFile /etc/pki/tls/private/wildcard.key
        ServerName graphite.domain.com
        DocumentRoot "/usr/local/webapp"

        # I've found that an equal number of processes & threads tends
        # to show the best performance for Graphite (ymmv).
        #WSGIDaemonProcess graphite processes=5 threads=5 display-name='%{GROUP}' inactivity-timeout=120 python-path=/usr/local/lib/
        #WSGIProcessGroup graphite

        # You will need to create this file! There is a graphite.wsgi.example
        # file in this directory that you can safely use, just copy it to graphite.wgsi
        WSGIScriptAlias / /usr/local/conf/graphite.wsgi

        Alias /content/ /usr/local/webapp/content/
        <Location "/content/">
                SetHandler None
        </Location>

        # NOTE: In order for the django admin site media to work you
        # must change @DJANGO_ROOT@ to be the path to your django
        # installation, which is probably something like:
        # /usr/lib/python2.6/site-packages/django
        Alias /media/ "/usr/local/lib/python2.6/site-packages/django/contrib/admin/media/"
        <Location "/media/">
                SetHandler None
        </Location>

        # The graphite.wsgi file has to be accessible by apache. It won't
        # be visible to clients because of the DocumentRoot though.
        <Directory /usr/local/conf/>
                Order deny,allow
                Allow from all
        </Directory>

</VirtualHost>

Of course, my company doesn't have anything installed in the default paths so you may not be able to just copy/paste. The important things are to make the WSGIProcessDaemon and group available to both http and https so put them outside the Virtualhosts Also, don't forget to remove the "SSLEngine On" and other SSL stuff from the virtualhost listening on port 80. In our configuration the hosts communicate with an internal name so make an alias in your hosts file if you need so you have a ServerName that matches for port 80 and one for 443.

Can you help with this problem?

Provide an answer of your own, or ask Nick for more information if necessary.

To post a message you must log in.