Data not getting through for display

Asked by Aseem

Hi.

Some background on how I am currently using graphite trunk/main.

I am using the bzr version of the code. I don't run any apps as root so I had to munge with carbon-agent to make it to accept command line parameters for various things like log dir and pid dir. I also had to munge with init-script.sh to insert variables to be replaced during build time. None of the changes modified any core code in any of the scripts.

I also had trouble with mod_python not agreeing with virtualpython about python-paths and despite help from our local pythonista, uttering a few magic incantations, I could not get it to work. So I hooked it up to mod_wsgi using Django's wsgi entry point as outlined in the docs (I have not used Django before so there's a possiblity that I messed up - but it does not look like it). Finally I switched from distutils to setuptools as its a bit more saner during build time. I also created the schema file per the docs and simply replaced the ^server\. with ^system\. to match the example script.

I finally got everything to work from a self contained folder. The graphite main page shows up. I then fired up the example-client with 10 sec delay and it goes its merry way. Django settings have DEBUG set to True. carbon-cache logs show data coming in fine. The whisper files exist for the metric. However when I search for the name of the metric or try autocomplete it fails with a 500 with nothing in the logs. CLI simply shows a black box with No Data in red. I am not quite sure what is wrong and before I dug into the actual app code itself, I was wondering if there are any obvious things I should try to figure out what went wrong. Log snippets are included below.

If you would like to take a look I would be more than glad to dump the tarball somewhere for download. Its ~31M with all the packages (Django, pyparse...) and 13M without.

Thanks.
AM

Log snippets:
carbon-agent.log
Thu Jul 17 22:16:36 2008: carbon-agent started (pid=6477)
Thu Jul 17 22:16:36 2008: created cache pipe, fds=['4', '5']
Thu Jul 17 22:16:36 2008: created persister pipe, fds=['6', '7']
Thu Jul 17 22:16:36 2008: carbon-cache started with pid 6478
Thu Jul 17 22:16:36 2008: carbon-persister started with pid 6480
Thu Jul 17 22:16:36 2008: listener socket ready
Thu Jul 17 22:16:57 2008: accepted connection from ('127.0.0.1', 38171)
Thu Jul 17 22:17:01 2008: lost connection from ('127.0.0.1', 38171)
Thu Jul 17 22:17:03 2008: accepted connection from ('127.0.0.1', 38172)
Thu Jul 17 22:17:57 2008: lost connection from ('127.0.0.1', 38172)
Thu Jul 17 22:18:19 2008: accepted connection from ('127.0.0.1', 38194)
Thu Jul 17 22:19:11 2008: Received SIGTERM, killing children
Thu Jul 17 22:19:11 2008: Sent SIGTERM to carbon-cache

cache.log
Thu Jul 17 22:38:00 2008: update_many: /work/mw/dev/graphite/_data/carbon-agent/whisper/carbon/agents/narada/cache/size.wsp 1 data points took 0.000163 seconds
Thu Jul 17 22:38:00 2008: update_many: /work/mw/dev/graphite/_data/carbon-agent/whisper/carbon/agents/narada/cache/queues.wsp 1 data points took 0.000156 seconds
Thu Jul 17 22:38:01 2008: doCheckpoint() now=1216359421 stats={'creates': 0, 'updateTimes': 0.0011448860168457031, 'errors': 0, 'committedMessages': 6, 'updates': 6}
Thu Jul 17 22:38:01 2008: update_many: /work/mw/dev/graphite/_data/carbon-agent/whisper/carbon/agents/narada/errors.wsp 1 data points took 0.000211 seconds
<snip>....</snip>
Thu Jul 17 22:38:08 2008: Resorting 3 cache queues by size...
Thu Jul 17 22:38:08 2008: Sorted 3 cache queues in 0.000 seconds
Thu Jul 17 22:38:08 2008: update_many: /work/mw/dev/graphite/_data/carbon-agent/whisper/system/loadavg_5min.wsp 1 data points took 0.000215 seconds
Thu Jul 17 22:38:08 2008: update_many: /work/mw/dev/graphite/_data/carbon-agent/whisper/system/loadavg_1min.wsp 1 data points took 0.000155 seconds
Thu Jul 17 22:38:08 2008: update_many: /work/mw/dev/graphite/_data/carbon-agent/whisper/system/loadavg_15min.wsp 1 data points took 0.000152 seconds

(I am guessing these are Django logs)
exception.log
Nothing

info.log
Fri Jul 18 00:32:56 2008 :: path= pattern=*
Fri Jul 18 00:38:16 2008 :: path= pattern=*

Question information

Language:
English Edit question
Status:
Solved
For:
Graphite Edit question
Assignee:
No assignee Edit question
Solved by:
Aseem
Solved:
Last query:
Last reply:
Revision history for this message
chrismd (chrismd) said :
#1

Hi Aseem, can you tell me what revision of trunk you are using by running "bzr version-info"? I made several changes to trunk right before the 0.9.4 release so that non-root installations work and another change that may possibly fix your problem. I would be able to help troubleshoot your problem better if your setup was more familar. Can you bzr pull the latest trunk revision (89) and re-install using "install.py --install-root"? That should allow you to install as a non-root user. It would be easier to troubleshoot if you are on the latest version of code.

Also, it looks like from your carbon logs that the carbon-persister process is not running because you do not have a persister.log, that is probably caused by the graphite package not being importable, so something is missing from your PYTHONPATH environment variable, or the graphite package is not installed in site-packages. I also see that your data is not the standard location (which is $GRAPHITE_ROOT/storage/whisper/), did you set the STORAGE_DIR in your local_settings.py so that the webapp can find the data?

Revision history for this message
Aseem (amohanty) said :
#2

>> Hi Aseem, can you tell me what revision of trunk you are using by running "bzr version-info"?

$ bzr version-info
revision-id: <email address hidden>
date: 2008-07-10 11:40:52 -0500
build-date: 2008-07-21 16:39:00 -0700
revno: 79
branch-nick: graphite-main

>> Also, it looks like from your carbon logs that the carbon-persister process is not running because you do not have a persister.log,

Hmm. from the ps I see that the persister is running.

$ ps aux | grep carbon
aseem 8742 0.0 0.2 7772 3788 ? S 16:37 0:00 /work/mw/dev/graphite/_install/bin/python /work/mw/dev/graphite/_install/bin/carbon-agent.py -l /work/mw/dev/graphite/_logs -x /work/mw/dev/graphite/_data/carbon-agent -d /work/mw/dev/graphite/_data/carbon-agent
aseem 8743 0.0 0.2 7220 4620 ? S 16:37 0:00 /work/mw/dev/graphite/_install/bin/python /work/mw/dev/graphite/_install/bin/carbon-cache.py 4 7 /work/mw/dev/graphite/_logs
aseem 8745 0.0 0.2 7152 4588 ? S 16:37 0:00 /work/mw/dev/graphite/_install/bin/python /work/mw/dev/graphite/_install/bin/carbon-persister.py 6 /work/mw/dev/graphite/_logs /work/mw/dev/graphite/_data/carbon-agent

>> I also see that your data is not the standard location (which is $GRAPHITE_ROOT/storage/whisper/), did you set the STORAGE_DIR in your local_settings.py so that the webapp can find the data?

I basically copied over all of settings.py and setup some replaceable string in there to customize paths to be able to run it under virtualenv.

I will try fresh checkout and see if I can get that working.

Thanks.
AM

Revision history for this message
Aseem (amohanty) said :
#3

Well I had tried the regular install previously of v0.9.2 and it had worked. The latest trunk seems to work too. So everything seems to be fine. I just need to figure out why the virtualenv install does not work.

I will post back on my findings.

Thanks.
AM

Revision history for this message
Aseem (amohanty) said :
#4

Hi.

So I was able to setup a completely self contained project that can check out the bzr trunk and get a virtualenv/mod_wsgi/apache or virtualenv/django-wsgi based server on custom port up and running in two steps. Thanks for the help.

If anybody is interested in the scripts and stuff, feel free to contact me.

Thanks again.
AM