Getting internal error for graphite

Asked by rocky on 2017-09-26

Hi,

After installation of graphite getting Internal server error (error code : 500) while hitting from the browser. Could you please check and provide your inputs.

Below log path : /opt/graphite/storage/log/webapp

file : error.log

[Tue Sep 26 01:28:41 2017] [error] [client 10.2.200.22] mod_wsgi (pid=21224): Target WSGI script '/opt/graphite/conf/graphite.wsgi' cannot be loaded as Python module.
[Tue Sep 26 01:28:41 2017] [error] [client 10.2.200.22] mod_wsgi (pid=21224): Exception occurred processing WSGI script '/opt/graphite/conf/graphite.wsgi'.
[Tue Sep 26 01:28:41 2017] [error] [client 10.2.200.22] Traceback (most recent call last):
[Tue Sep 26 01:28:41 2017] [error] [client 10.2.200.22] File "/opt/graphite/conf/graphite.wsgi", line 4, in <module>
[Tue Sep 26 01:28:41 2017] [error] [client 10.2.200.22] from graphite.wsgi import application
[Tue Sep 26 01:28:41 2017] [error] [client 10.2.200.22] File "/opt/graphite/webapp/graphite/wsgi.py", line 7, in <module>
[Tue Sep 26 01:28:41 2017] [error] [client 10.2.200.22] from django.utils.importlib import import_module
[Tue Sep 26 01:28:41 2017] [error] [client 10.2.200.22] ImportError: No module named django.utils.importlib

Thank you in advance.

Question information

Language:
English Edit question
Status:
Solved
For:
Graphite Edit question
Assignee:
No assignee Edit question
Solved by:
rocky
Solved:
2017-10-09
Last query:
2017-10-09
Last reply:
2017-09-28
Denis Zhdanov (deniszhdanov) said : #1

What version of Graphite you trying to install? and what's Django version do you have?

Denis Zhdanov (deniszhdanov) said : #2

PS: For Graphite 1.0.x wsgi can be very simple - https://github.com/graphite-project/graphite-web/blob/master/conf/graphite.wsgi.example, .e.g.

import sys
sys.path.append('/opt/graphite/webapp')
from graphite.wsgi import application

rocky (rocky.munich1) said : #3

Hi Denis

Please find the details.

[root@XX ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.9 (Santiago)
[root@XX ~]#

[root@XX ~]# python -V
Python 2.7.3
[root@XX ~]#

>>> import django
>>> django.VERSION
(1, 9, 9, 'final', 0)
>>>

graphite_web-1.1.0

rocky (rocky.munich1) said : #4

Hi

please find the more information.

[root@XX conf.d]# cat /opt/graphite/conf/graphite.wsgi
import sys
sys.path.append('/opt/graphite/webapp')

from graphite.wsgi import application
[root@XX conf.d]#

[root@XX conf.d]# cat /opt/graphite/webapp/graphite/wsgi.py
import os
import sys

try:
    from importlib import import_module
except ImportError:
    from django.utils.importlib import import_module

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'graphite.settings') # noqa

from django.conf import settings
from django.core.wsgi import get_wsgi_application

application = get_wsgi_application()

try:
    import whitenoise
except ImportError:
    whitenoise = False
else:
    # WhiteNoise < 2.0.1 does not support Python 2.6
    if sys.version_info[:2] < (2, 7):
        whitenoise_version = tuple(map(
                int, getattr(whitenoise, '__version__', '0').split('.')))
        if whitenoise_version < (2, 0, 1):
            whitenoise = False

if whitenoise:
    from whitenoise.django import DjangoWhiteNoise
    application = DjangoWhiteNoise(application)
    prefix = "/".join((settings.URL_PREFIX.strip('/'), 'static'))
    for directory in settings.STATICFILES_DIRS:
        application.add_files(directory, prefix=prefix)
    for app_path in settings.INSTALLED_APPS:
        module = import_module(app_path)
        directory = os.path.join(os.path.dirname(module.__file__), 'static')
        if os.path.isdir(directory):
            application.add_files(directory, prefix=prefix)
[root@XX conf.d]#

[root@XX conf.d]# cat /etc/httpd/conf.d/graphite.conf
# Enable virtualhosts, perhaps by adding this to your server's config somewhere,
# probably the main httpd.conf
# NameVirtualHost *:80

# This line also needs to be in your server's config.
# LoadModule wsgi_module modules/mod_wsgi.so

# You need to manually edit this file to fit your needs.
# This configuration assumes the default installation prefix
# of /opt/graphite/, if you installed graphite somewhere else
# you will need to change all the occurrences of /opt/graphite/
# in this file to your chosen install location.

<IfModule !wsgi_module.c>
    LoadModule wsgi_module modules/mod_wsgi.so
</IfModule>

# XXX You need to set this up!
# Read http://code.google.com/p/modwsgi/wiki/ConfigurationDirectives#WSGISocketPrefix
# For example, create a directory /var/run/wsgi and use that.
WSGISocketPrefix run/wsgi

<VirtualHost *:8080>
        ServerName XX.XX.XX.XX
        DocumentRoot "/opt/graphite/webapp"
        ErrorLog /opt/graphite/storage/log/webapp/error.log
        CustomLog /opt/graphite/storage/log/webapp/access.log common
        Header set Access-Control-Allow-Origin "*"

        # I've found that an equal number of processes & threads tends
        # to show the best performance for Graphite (ymmv).
        WSGIDaemonProcess graphite processes=5 threads=5 display-name='%{GROUP}' inactivity-timeout=120
        WSGIProcessGroup graphite
        WSGIApplicationGroup %{GLOBAL}
        WSGIImportScript /opt/graphite/conf/graphite.wsgi process-group=graphite application-group=%{GLOBAL}

        # XXX You will need to create this file! There is a graphite.wsgi.example
        # file in this directory that you can safely use, just copy it to graphite.wgsi
        WSGIScriptAlias / /opt/graphite/conf/graphite.wsgi

        # XXX To serve static files, either:
        # * Install the whitenoise Python package (pip install whitenoise)
        # * Collect static files in a directory by running:
        # django-admin.py collectstatic --noinput --settings=graphite.settings
        # And set an alias to serve static files with Apache:
        Alias /static/ /opt/graphite/static/
        <Directory /opt/graphite/static/>
                        Order deny,allow
                        Allow from all
        </Directory>

        ########################
        # URL-prefixed install #
        ########################
        # If using URL_PREFIX in local_settings for URL-prefixed install (that is not located at "/"))
        # your WSGIScriptAlias line should look like the following (e.g. URL_PREFX="/graphite"

        # WSGIScriptAlias /graphite /srv/graphite-web/conf/graphite.wsgi/graphite
        # Alias /graphite/static /opt/graphite/webapp/content
        # <Location "/graphite/static/">
        # SetHandler None
        # </Location>

        # XXX In order for the django admin site media to work you
        # must change @DJANGO_ROOT@ to be the path to your django
        # installation, which is probably something like:
        # /usr/lib/python2.6/site-packages/django
        Alias /media/ "@DJANGO_ROOT@/contrib/admin/media/"

        # The graphite.wsgi file has to be accessible by apache. It won't
        # be visible to clients because of the DocumentRoot though.
        <Directory /opt/graphite/conf/>
                <IfVersion < 2.4>
                        Order deny,allow
                        Allow from all
                </IfVersion>
                <IfVersion >= 2.4>
                        Require all granted
                </IfVersion>
        </Directory>

</VirtualHost>
[root@XX conf.d]#

Denis Zhdanov (deniszhdanov) said : #5

Just use simple wsgi file which I posted above.

rocky (rocky.munich1) said : #6

Thank you for your response.

We are already using same same.

Denis Zhdanov (deniszhdanov) said : #7

Ah, true.
django.utils.importlib was deprecated in django 1.9, but looks like your code still using old version oj django. try to remove all versions of django except 1.9

rocky (rocky.munich1) said : #8

Thank you for response.

I can see only one Django==1.9.9 in the system.

Could you please help me to find out other version in the system.

rocky (rocky.munich1) said : #9

Could you please some one help on this.. really appreciated..

Denis Zhdanov (deniszhdanov) said : #10

Remove old fine (/opt/graphite/webapp/graphite/wsgi.py) and try again.

rocky (rocky.munich1) said : #12

Hi Deniz

After removing the file same repose.

[root@XX webapp]# rm /opt/graphite/webapp/graphite/wsgi.py
rm: remove regular file `/opt/graphite/webapp/graphite/wsgi.py'? y
[root@icingamonitoringdr webapp]#

[root@XX webapp]# ls -l "/opt/graphite/webapp/graphite/wsgi.py"
ls: cannot access /opt/graphite/webapp/graphite/wsgi.py: No such file or directory
[root@XX webapp]#

Still getting error

:58:55 2017] [error] [client 10.2.201.115] mod_wsgi (pid=11944): Exception occurred processing WSGI script '/opt/graphite/conf/graphite.wsgi'.
[Thu Sep 28 02:58:55 2017] [error] [client 10.2.201.115] Traceback (most recent call last):
[Thu Sep 28 02:58:55 2017] [error] [client 10.2.201.115] File "/opt/graphite/conf/graphite.wsgi", line 4, in <module>
[Thu Sep 28 02:58:55 2017] [error] [client 10.2.201.115] from graphite.wsgi import application
[Thu Sep 28 02:58:55 2017] [error] [client 10.2.201.115] File "/opt/graphite/webapp/graphite/wsgi.py", line 7, in <module>
[Thu Sep 28 02:58:55 2017] [error] [client 10.2.201.115] ImportError: No module named django.utils.importlib

Denis Zhdanov (deniszhdanov) said : #13

How you can get error in line 7 of non-existing File "/opt/graphite/webapp/graphite/wsgi.py" ?

rocky (rocky.munich1) said : #14

Seems to be something else having the issue.

rocky (rocky.munich1) said : #15

after rebooting the server.. it is saying site can't be reached.

rocky (rocky.munich1) said : #16

Hi Denis,

Could you please cross check the file "/etc/httpd/conf.d/graphite.conf" mentioned above.

Denis Zhdanov (deniszhdanov) said : #17

Looks sane. You need to remove 'Alias /media/ "@DJANGO_ROOT@/contrib/admin/media/"' line, but it should not bring any troubles.

rocky (rocky.munich1) said : #18

We removed the file and tried..
still getting same issue in the error log

[Tue Sep 26 01:28:41 2017] [error] [client 10.2.200.22] File "/opt/graphite/conf/graphite.wsgi", line 4, in <module>
[Tue Sep 26 01:28:41 2017] [error] [client 10.2.200.22] from graphite.wsgi import application
[Tue Sep 26 01:28:41 2017] [error] [client 10.2.200.22] File "/opt/graphite/webapp/graphite/wsgi.py", line 7, in <module>
[Tue Sep 26 01:28:41 2017] [error] [client 10.2.200.22] from django.utils.importlib import import_module
[Tue Sep 26 01:28:41 2017] [error] [client 10.2.200.22] ImportError: No module named django.utils.importlib

Denis Zhdanov (deniszhdanov) said : #19

Sorry, rocky, but this lines are old - from September 26. I'm really doubt that non-existing file can give you this error, especially after server reboot. Please check fresh errors.

rocky (rocky.munich1) said : #20

ok sure...
 Same error.. that is the reason copied for the above line..
Let me share the fresh error log.

[Fri Sep 29 00:11:06 2017] [error] [client 10.2.200.191] mod_wsgi (pid=17662): Target WSGI script '/opt/graphite/conf/graphite.wsgi' cannot be loaded as Python module.
[Fri Sep 29 00:11:06 2017] [error] [client 10.2.200.191] mod_wsgi (pid=17662): Exception occurred processing WSGI script '/opt/graphite/conf/graphite.wsgi'.
[Fri Sep 29 00:11:06 2017] [error] [client 10.2.200.191] Traceback (most recent call last):
[Fri Sep 29 00:11:06 2017] [error] [client 10.2.200.191] File "/opt/graphite/conf/graphite.wsgi", line 4, in <module>
[Fri Sep 29 00:11:06 2017] [error] [client 10.2.200.191] from graphite.wsgi import application
[Fri Sep 29 00:11:06 2017] [error] [client 10.2.200.191] File "/opt/graphite/webapp/graphite/wsgi.py", line 7, in <module>
[Fri Sep 29 00:11:06 2017] [error] [client 10.2.200.191] from django.utils.importlib import import_module
[Fri Sep 29 00:11:06 2017] [error] [client 10.2.200.191] ImportError: No module named django.utils.importlib

Denis Zhdanov (deniszhdanov) said : #21

Sorry, rocky, I ran out of ideas. You should remove file /opt/graphite/webapp/graphite/wsgi.py and restart webserver - error should gone. If it's not - probably you're checking wrong server, or something else is seriously wrong.

rocky (rocky.munich1) said : #22

Thank you for response.

Found something .. let me share with you..

[root@XX ~]# ldd /usr/lib64/httpd/modules/mod_wsgi.so
        linux-vdso.so.1 => (0x00007ffd08dcb000)
        libpython2.6.so.1.0 => /usr/lib64/libpython2.6.so.1.0 (0x00007febc266b000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007febc244e000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007febc2249000)
        libutil.so.1 => /lib64/libutil.so.1 (0x00007febc2046000)
        libm.so.6 => /lib64/libm.so.6 (0x00007febc1dc2000)
        libc.so.6 => /lib64/libc.so.6 (0x00007febc1a2d000)
        /lib64/ld-linux-x86-64.so.2 (0x0000003adba00000)
[root@XX ~]#

[root@XX ~]# find / -name libpython2.7*
/usr/local/lib/python2.7/config/libpython2.7.a
/usr/local/lib/libpython2.7.a
/usr/local/bin/Python-2.7.3/libpython2.7.a
[root@XX ~]#

How to load the module with python version 2.7 ?

Denis Zhdanov (deniszhdanov) said : #23

You should install proper mod_wsgi module, which build against python 2.7
It depends where do you get python 2.7
If you installed that from 3rd party repo - it's better to pick up mod_wsgi package also there.
if you install python from sources - then you need to build mod_wsgi also from sources, as described for example here - https://blog.webhostpython.com/how-to-install-python-2-7-with-mod_wsgi-on-a-centos-6-vps-or-server-with-cpanel/

rocky (rocky.munich1) said : #24

Thank you for the update..

Issue resolved after updating the module mod_wsgi.so. Please find the reference links and commands

https://serverfault.com/questions/453977/configuring-mod-wsgi-to-use-python-2-7
yum install httpd-devel

https://stackoverflow.com/questions/20842732/libpython2-7-so-1-0-cannot-open-shared-object-file-no-such-file-or-directory

/etc/ld.so.conf --> Need to update with location
locate libpython2.7.so.1.0
ldconfig

cd /usr/local/bin/Python-2.7.3
./configure –-enable-shared CFLAGS=-fPIC
Make
Make install

Cd /root/mod_wsgi-3.4
./configure –-with-python=/usr/local/bin/Python-2.7.3/python
Make
Make install

Service httpd restart

Ldd /usr/lib64/httpd/modules/mod_wsgi.so

Netstat –ltnp | grep 80