Graphite

error 404 when accessing /metrics/

Asked by Ran Levkovich

Hi,

So, I've managed to install Graphite and can access the console.

Sensu is running on the same server as well and pushing metrics into carbon-cache using the metrics-vmstat.rb plugin

I can see the metrics are created:

[root@sendbox1 sendbox1-a]# pwd
/opt/graphite/storage/whisper/carbon/agents/sendbox1-a
[root@sendbox1 sendbox1-a]# ll | grep wsp
-rw-rw-rw- 1 root root 1555228 Apr 12 16:25 activeConnections.wsp
-rw-rw-rw- 1 root root 1555228 Apr 19 19:11 avgUpdateTime.wsp
-rw-rw-rw- 1 root root 1555228 Apr 19 19:11 blacklistMatches.wsp
-rw-rw-rw- 1 root root 1555228 Apr 19 19:11 committedPoints.wsp
-rw-rw-rw- 1 root root 1555228 Apr 19 19:11 cpuUsage.wsp
-rw-rw-rw- 1 root root 1555228 Apr 19 19:11 creates.wsp
-rw-rw-rw- 1 root root 1555228 Apr 12 16:25 droppedCreates.wsp
-rw-rw-rw- 1 root root 1555228 Apr 19 19:11 errors.wsp
-rw-rw-rw- 1 root root 1555228 Apr 19 19:11 memUsage.wsp
-rw-rw-rw- 1 root root 1555228 Apr 19 19:11 metricsReceived.wsp
-rw-rw-rw- 1 root root 1555228 Apr 19 19:11 pointsPerUpdate.wsp
-rw-rw-rw- 1 root root 1555228 Apr 19 19:11 updateOperations.wsp
-rw-rw-rw- 1 root root 1555228 Apr 19 19:11 whitelistRejects.wsp
[root@sendbox1 sendbox1-a]#

I wish to see the metrics on the Graphite console, but nothing appears under the "Metrics" tree root, when I try to expend it.

Looking at the HTTPD logs I can see the following 404 error in the access log (nothing else in the rest of the HTTPD logs):

192.168.174.102 - - [19/Apr/2016:18:22:05 +0300] "GET /metrics/find/?_dc=1461078899567&query=*&format=treejson&contexts=1&path=&node=GraphiteTree HTTP/1.1" 400 34

When I try to access this URL directly from my browser (e.g the URL I’m hitting is http://sendbox1/metrics/find/?_dc=1461078899567&query=*&format=treejson&contexts=1&path=&node=GraphiteTree)
I get the following response on my browser:

"Missing required parameter 'query'"

Any ideas how to deal with this one?

I’d be happy to provide any additional information if required

Thanks for the help

Ran

Question information

Language:
English Edit question
Status:
Expired
For:
Graphite Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Ran Levkovich (ran-levkovich) said : 		#1

Hi

Did anyone had the chance to look into this one?

I'm still stuck with it :-|

Cheers

Ran

Revision history for this message
Denis Zhdanov (deniszhdanov) said : 		#2

Hi Ran,

Something is really wrong with your web setup. Could you please describe your setup in bit more details?
Which webserver are you using, what's config etc?

Revision history for this message
Ran Levkovich (ran-levkovich) said : 		#3

Hi Denis,

Thank you for your reply

I’m using the following httpd version:

[root@sendbox1 httpd]# /usr/sbin/httpd -version
Server version: Apache/2.4.6 (CentOS)
Server built: Nov 19 2015 21:43:13

Graphite is running above Django 1.9.5 with postgres

I’m pasting below the contents of: (didn’t see a way to attach)

1. Graphite's local_settings.py
2. graphite.wsgi
3. The httpd graphite.conf file

Is there any other info which could be helpful?

Thanks again

Ran

[root@sendbox1 conf]# cat graphite.wsgi
import os
import sys
sys.path.append('/opt/graphite/webapp')

try:
    from importlib import import_module
except ImportError:
    from django.utils.importlib import import_module

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'graphite.settings') # noqa

from django.conf import settings
from django.core.wsgi import get_wsgi_application
from graphite.logger import log

application = get_wsgi_application()

try:
    import whitenoise
except ImportError:
    whitenoise = False
else:
    # WhiteNoise < 2.0.1 does not support Python 2.6
    if sys.version_info[:2] < (2, 7):
        whitenoise_version = tuple(map(
                int, getattr(whitenoise, '__version__', '0').split('.')))
        if whitenoise_version < (2, 0, 1):
            whitenoise = False

if whitenoise:
    from whitenoise.django import DjangoWhiteNoise
    application = DjangoWhiteNoise(application)
    prefix = "/".join((settings.URL_PREFIX.strip('/'), 'static'))
    for directory in settings.STATICFILES_DIRS:
        application.add_files(directory, prefix=prefix)
    for app_path in settings.INSTALLED_APPS:
        module = import_module(app_path)
        directory = os.path.join(os.path.dirname(module.__file__), 'static')
        if os.path.isdir(directory):
            application.add_files(directory, prefix=prefix)

# Initializing the search index can be very expensive. The import below
# ensures the index is preloaded before any requests are handed to the
# process.
#log.info("graphite.wsgi - pid %d - reloading search index" % os.getpid())
#import graphite.metrics.search # noqa
[root@sendbox1 conf]#

===========================================================================

[root@sendbox1 graphite]# pwd
/opt/graphite/webapp/graphite
[root@sendbox1 graphite]# cat local_settings.py
## Graphite local_settings.py
# Edit this file to customize the default Graphite webapp settings
#
# Additional customizations to Django settings can be added to this file as well

#####################################
# General Configuration #
#####################################
# Set this to a long, random unique string to use as a secret key for this
# install. This key is used for salting of hashes used in auth tokens,
# CRSF middleware, cookie storage, etc. This should be set identically among
# instances if used behind a load balancer.
#SECRET_KEY = 'UNSAFE_DEFAULT'
SECRET_KEY = 'yalatahavod'

# In Django 1.5+ set this to the list of hosts your graphite instances is
# accessible as. See:
# https://docs.djangoproject.com/en/dev/ref/settings/#std:setting-ALLOWED_HOSTS
#ALLOWED_HOSTS = [ '*' ]

# Set your local timezone (Django's default is America/Chicago)
# If your graphs appear to be offset by a couple hours then this probably
# needs to be explicitly set to your local timezone.
#TIME_ZONE = 'America/Los_Angeles'
TIME_ZONE = 'Asia/Jerusalem'

# Override this to provide documentation specific to your Graphite deployment
#DOCUMENTATION_URL = "http://graphite.readthedocs.org/"

# Metric data and graphs are cached for one minute by default
#DEFAULT_CACHE_DURATION = 60
# Logging
#LOG_ROTATION = True
#LOG_ROTATION_COUNT = 1
#LOG_RENDERING_PERFORMANCE = True
#LOG_CACHE_PERFORMANCE = True
#LOG_METRIC_ACCESS = True

# Enable full debug page display on exceptions (Internal Server Error pages)
DEBUG = True

# If using RRD files and rrdcached, set to the address or socket of the daemon
#FLUSHRRDCACHED = 'unix:/var/run/rrdcached.sock'

# This lists the memcached servers that will be used by this webapp.
# If you have a cluster of webapps you should ensure all of them
# have the *exact* same value for this setting. That will maximize cache
# efficiency. Setting MEMCACHE_HOSTS to be empty will turn off use of
# memcached entirely.
#
# You should not use the loopback address (127.0.0.1) here if using clustering
# as every webapp in the cluster should use the exact same values to prevent
# unneeded cache misses. Set to [] to disable caching of images and fetched data
#MEMCACHE_HOSTS = ['10.10.10.10:11211', '10.10.10.11:11211', '10.10.10.12:11211']
# Metric data and graphs are cached for one minute by default. If defined,
# DEFAULT_CACHE_POLICY is a list of tuples of minimum query time ranges mapped
# to the cache duration for the results. This allows for larger queries to be
# cached for longer periods of times. All times are in seconds. If the policy is
# empty or undefined, all results will be cached for DEFAULT_CACHE_DURATION.
#DEFAULT_CACHE_DURATION = 60 # Cache images and data for 1 minute
#DEFAULT_CACHE_POLICY = [(0, 60), # default is 60 seconds
# (7200, 120), # >= 2 hour queries are cached 2 minutes
# (21600, 180)] # >= 6 hour queries are cached 3 minutes
#MEMCACHE_KEY_PREFIX = 'graphite'

# Set URL_PREFIX when deploying graphite-web to a non-root location
URL_PREFIX = '/graphite'

#####################################
# Filesystem Paths #
#####################################
# Change only GRAPHITE_ROOT if your install is merely shifted from /opt/graphite
# to somewhere else
#GRAPHITE_ROOT = '/opt/graphite'
STATIC_ROOT = '/opt/graphite/static'
STATIC_URL = '/static/'
STATICFILES_DIRS='/opt/graphite/webapp/content'

# Most installs done outside of a separate tree such as /opt/graphite will only
# need to change these three settings. Note that the default settings for each
# of these is relative to GRAPHITE_ROOT
#CONF_DIR = '/opt/graphite/conf'
#STORAGE_DIR = '/opt/graphite/storage'
#CONTENT_DIR = '/opt/graphite/webapp/content'

# To further or fully customize the paths, modify the following. Note that the
# default settings for each of these are relative to CONF_DIR and STORAGE_DIR
#
## Webapp config files
#DASHBOARD_CONF = '/opt/graphite/conf/dashboard.conf'
#GRAPHTEMPLATES_CONF = '/opt/graphite/conf/graphTemplates.conf'

## Data directories
# NOTE: If any directory is unreadable in DATA_DIRS it will break metric browsing
#WHISPER_DIR = '/opt/graphite/storage/whisper'
#RRD_DIR = '/opt/graphite/storage/rrd'
#DATA_DIRS = [WHISPER_DIR, RRD_DIR] # Default: set from the above variables
#LOG_DIR = '/opt/graphite/storage/log/webapp'
#INDEX_FILE = '/opt/graphite/storage/index' # Search index file

#####################################
# Email Configuration #
#####################################
# This is used for emailing rendered Graphs
# Default backend is SMTP
#EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'
#EMAIL_HOST = 'localhost'
#EMAIL_PORT = 25
#EMAIL_HOST_USER = ''
#EMAIL_HOST_PASSWORD = ''
#EMAIL_USE_TLS = False
# To drop emails on the floor, enable the Dummy backend:
#EMAIL_BACKEND = 'django.core.mail.backends.dummy.EmailBackend'

#####################################
# Authentication Configuration #
#####################################
## LDAP / ActiveDirectory authentication setup
#USE_LDAP_AUTH = True
#LDAP_SERVER = "ldap.mycompany.com"
#LDAP_PORT = 389
# OR
#LDAP_URI = "ldaps://ldap.mycompany.com:636"
#LDAP_SEARCH_BASE = "OU=users,DC=mycompany,DC=com"
#LDAP_BASE_USER = "CN=some_readonly_account,DC=mycompany,DC=com"
#LDAP_BASE_PASS = "readonly_account_password"
#LDAP_USER_QUERY = "(username=%s)" #For Active Directory use "(sAMAccountName=%s)"
#
# User DN template to use for binding (and authentication) against
# the LDAP server. %(username) is replaced with the username supplied at
# graphite login.
#LDAP_USER_DN_TEMPLATE = "CN=%(username)s,OU=users,DC=mycompany,DC=com"
# If you want to further customize the ldap connection options you should
# directly use ldap.set_option to set the ldap module's global options.
# For example:
#
#import ldap
#ldap.set_option(ldap.OPT_X_TLS_REQUIRE_CERT, ldap.OPT_X_TLS_ALLOW)
#ldap.set_option(ldap.OPT_X_TLS_CACERTDIR, "/etc/ssl/ca")
#ldap.set_option(ldap.OPT_X_TLS_CERTFILE, "/etc/ssl/mycert.pem")
#ldap.set_option(ldap.OPT_X_TLS_KEYFILE, "/etc/ssl/mykey.pem")
# See http://www.python-ldap.org/ for further details on these options.

## REMOTE_USER authentication. See: https://docs.djangoproject.com/en/dev/howto/auth-remote-user/
#USE_REMOTE_USER_AUTHENTICATION = True

# Override the URL for the login link (e.g. for django_openid_auth)
#LOGIN_URL = '/account/login'
###############################
# Authorization for Dashboard #
###############################
# By default, there is no security on dashboards - any user can add, change or delete them.
# This section provides 3 different authorization models, of varying strictness.

# If set to True, users must be logged in to save or delete dashboards. Defaults to False
#DASHBOARD_REQUIRE_AUTHENTICATION = True

# If set to the name of a user group, dashboards can be saved and deleted by any user in this
# group. Groups can be set in the Django Admin app, or in LDAP. Defaults to None.
# NOTE: Ignored if DASHBOARD_REQUIRE_AUTHENTICATION is not set
#DASHBOARD_REQUIRE_EDIT_GROUP = 'dashboard-editors-group'

# If set to True, dashboards can be saved or deleted by any user having the appropriate
# (change or delete) permission (as set in the Django Admin app). Defaults to False
# NOTE: Ignored if DASHBOARD_REQUIRE_AUTHENTICATION is not set
#DASHBOARD_REQUIRE_PERMISSIONS = True

##########################
# Database Configuration #
##########################
# By default sqlite is used. If you cluster multiple webapps you will need
# to setup an external database (such as MySQL) and configure all of the webapp
# instances to use the same database. Note that this database is only used to store
# Django models such as saved graphs, dashboards, user preferences, etc.
# Metric data is not stored here.
#
# DO NOT FORGET TO RUN 'manage.py syncdb' AFTER SETTING UP A NEW DATABASE
#
# The following built-in database engines are available:
# django.db.backends.postgresql # Removed in Django 1.4
# django.db.backends.postgresql_psycopg2
# django.db.backends.mysql
# django.db.backends.sqlite3
# django.db.backends.oracle
#
# The default is 'django.db.backends.sqlite3' with file 'graphite.db'
# located in STORAGE_DIR
#
#DATABASES = {
# 'default': {
# 'NAME': '/opt/graphite/storage/graphite.db',
# 'ENGINE': 'django.db.backends.sqlite3',
# 'USER': '',
# 'PASSWORD': '',
# 'HOST': '127.0.0.1',
# 'PORT': ''
# }
#}

DATABASES = {
    'default': {
        'NAME': 'graphite',
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'USER': 'graphite',
        'PASSWORD': 'graphite123',
        'HOST': '127.0.0.1',
        'PORT': ''
    }
}

#########################
# Cluster Configuration #
#########################
# (To avoid excessive DNS lookups you want to stick to using IP addresses only in this entire section)
#
# This should list the IP address (and optionally port) of the webapp on each
# remote server in the cluster. These servers must each have local access to
# metric data. Note that the first server to return a match for a query will be
# used.
#CLUSTER_SERVERS = ["10.0.2.2:80", "10.0.2.3:80"]

## These are timeout values (in seconds) for requests to remote webapps
#REMOTE_STORE_FETCH_TIMEOUT = 6 # Timeout to fetch series data
#REMOTE_STORE_FIND_TIMEOUT = 2.5 # Timeout for metric find requests
#REMOTE_STORE_RETRY_DELAY = 60 # Time before retrying a failed remote webapp
#REMOTE_STORE_USE_POST = False # Use POST instead of GET for remote requests
#REMOTE_FIND_CACHE_DURATION = 300 # Time to cache remote metric find results

## Prefetch cache
# set to True to fetch all metrics using a single http request per remote server
# instead of one http request per target, per remote server.
# Especially useful when generating graphs with more than 4-5 targets or if
# there's significant latency between this server and the backends. (>20ms)
#REMOTE_PREFETCH_DATA = False

# During a rebalance of a consistent hash cluster, after a partition event on a replication > 1 cluster,
# or in other cases we might receive multiple TimeSeries data for a metric key. Merge them together rather
# that choosing the "most complete" one (pre-0.9.14 behaviour).
#REMOTE_STORE_MERGE_RESULTS = True

## Remote rendering settings
# Set to True to enable rendering of Graphs on a remote webapp
#REMOTE_RENDERING = True
# List of IP (and optionally port) of the webapp on each remote server that
# will be used for rendering. Note that each rendering host should have local
# access to metric data or should have CLUSTER_SERVERS configured
#RENDERING_HOSTS = []
#REMOTE_RENDER_CONNECT_TIMEOUT = 1.0

# If you are running multiple carbon-caches on this machine (typically behind a relay using
# consistent hashing), you'll need to list the ip address, cache query port, and instance name of each carbon-cache
# instance on the local machine (NOT every carbon-cache in the entire cluster). The default cache query port is 7002
# and a common scheme is to use 7102 for instance b, 7202 for instance c, etc.
#
# You *should* use 127.0.0.1 here in most cases
#CARBONLINK_HOSTS = ["127.0.0.1:7002:a", "127.0.0.1:7102:b", "127.0.0.1:7202:c"]
#CARBONLINK_TIMEOUT = 1.0
# Using 'query-bulk' queries for carbon
# It's more effective, but python-carbon 0.9.13 (or latest from 0.9.x branch) is required
# See https://github.com/graphite-project/carbon/pull/132 for details
#CARBONLINK_QUERY_BULK = False

#####################################
# Additional Django Settings #
#####################################
# Uncomment the following line for direct access to Django settings such as
# MIDDLEWARE_CLASSES or APPS
#from graphite.app_settings import *

[root@sendbox1 graphite]#

=======================================================================================

[root@sendbox1 conf.d]# pwd
/etc/httpd/conf.d
[root@sendbox1 conf.d]# cat graphite.conf
# This needs to be in your server's config somewhere, probably
# the main httpd.conf
# NameVirtualHost *:80

# This line also needs to be in your server's config.
# LoadModule wsgi_module modules/mod_wsgi.so

# You need to manually edit this file to fit your needs.
# This configuration assumes the default installation prefix
# of /opt/graphite/, if you installed graphite somewhere else
# you will need to change all the occurances of /opt/graphite/
# in this file to your chosen install location.

<IfModule !wsgi_module.c>
    LoadModule wsgi_module modules/mod_wsgi.so
</IfModule>

# XXX You need to set this up!
# Read http://code.google.com/p/modwsgi/wiki/ConfigurationDirectives#WSGISocketPrefix
#WSGISocketPrefix run/wsgi
WSGISocketPrefix /var/run/wsgi

<VirtualHost *:80>
        ServerName graphite
        DocumentRoot "/opt/graphite/webapp"
        ErrorLog /opt/graphite/storage/log/webapp/error.log
        CustomLog /opt/graphite/storage/log/webapp/access.log common

        # I've found that an equal number of processes & threads tends
        # to show the best performance for Graphite (ymmv).
        WSGIDaemonProcess graphite processes=5 threads=5 display-name='%{GROUP}' inactivity-timeout=120
        WSGIProcessGroup graphite
        WSGIApplicationGroup %{GLOBAL}
        WSGIImportScript /opt/graphite/conf/graphite.wsgi process-group=graphite application-group=%{GLOBAL}

        # XXX You will need to create this file! There is a graphite.wsgi.example
        # file in this directory that you can safely use, just copy it to graphite.wgsi
        WSGIScriptAlias / /opt/graphite/conf/graphite.wsgi

        # XXX To serve static files, either:
        # django-admin.py collectstatic --noinput --settings=graphite.settings
        # * Install the whitenoise Python package (pip install whitenoise)
        # or
        # * Collect static files in a directory by running:
        # django-admin.py collectstatic --noinput --settings=graphite.settings
        # And set an alias to serve static files with Apache:
        Alias /content/ /opt/graphite/webapp/content/
        <Location "/content/">
                SetHandler None
        </Location>

        # XXX In order for the django admin site media to work you
        # must change @DJANGO_ROOT@ to be the path to your django
        # installation, which is probably something like:
        # /usr/lib/python2.6/site-packages/django
        Alias /media/ "/usr/lib/python2.7/site-packages/django/contrib/media/"
        <Location "/media/">
                SetHandler None
        </Location>

        # The graphite.wsgi file has to be accessible by apache. It won't
        # be visible to clients because of the DocumentRoot though.
        <Directory /opt/graphite/conf>
                #Order deny,allow
                #Allow from all
                Options All
                AllowOverride All
                Require all granted
        </Directory>
        <Directory /opt/graphite/webapp/content>
                #Order deny,allow
                #Allow from all
                Options All
                AllowOverride All
                Require all granted
        </Directory>
</VirtualHost>
[root@sendbox1 conf.d]#

Revision history for this message
Ran Levkovich (ran-levkovich) said : 		#4

Hi Denis,

Did you manage to take a look?

Thanks for the help !

Ran

Revision history for this message
Ran Levkovich (ran-levkovich) said : 		#5

Hi,

Does anyone else have any idea?

Cheers

Ran

Revision history for this message
Launchpad Janitor (janitor) said : 		#6

This question was expired because it remained in the 'Open' state without activity for the last 15 days.

Revision history for this message
Ran Levkovich (ran-levkovich) said : 		#7

Hi

My issue is still opened

I'd really like to use Graphite but I really need the help here to set it up ..

Is there any chance to get help here? :-)

cheers

Ran

Revision history for this message
Denis Zhdanov (deniszhdanov) said : 		#8

Sorry, Ran, I was on vacation. Could you please try this as graphite.wsgi - https://gist.github.com/deniszh/133befd85d2e5ada2de4 ?
No need to rename file, just put its content to graphite.wsgi

Revision history for this message
Ran Levkovich (ran-levkovich) said : 		#9

thanks Denis (Hope you enjoyed your holidays !)

I've deployed the above graphite.wsgi and restarted http:

=======================================================================================
[root@sendbox1 ~]# cat /opt/graphite/conf/graphite.wsgi
import os
import sys
sys.path.append('/opt/graphite/webapp')

try:
    from importlib import import_module
except ImportError:
    from django.utils.importlib import import_module

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'graphite.settings') # noqa

from django.conf import settings
from django.core.wsgi import get_wsgi_application
from graphite.logger import log

application = get_wsgi_application()

# Initializing the search index can be very expensive. The import below
# ensures the index is preloaded before any requests are handed to the
# process.
log.info("graphite.wsgi - pid %d - reloading search index" % os.getpid())
import graphite.metrics.search # noqa[root@sendbox1 ~]#
=====================================================================================

but seem to be getting the same error )-:

==> access.log <==
192.168.174.102 - - [08/May/2016:16:37:31 +0300] "GET /metrics/find/?_dc=1462714651293&query=*&format=treejson&contexts=1&path=&node=GraphiteTree HTTP/1.1" 400 34

Anything else I could try?

Thanks

Ran

Revision history for this message
Launchpad Janitor (janitor) said : 		#10

This question was expired because it remained in the 'Open' state without activity for the last 15 days.