Graphite

No such file or directory: /opt/graphite/storage/whisper/carbon/agents/...

Asked by François Beausoleil

The gist of it: https://gist.github.com/1202515

Relevant part of the Traceback:

File "/opt/graphite/lib/carbon/writer.py", line 108, in writeCachedDataPoints
  whisper.create(dbFilePath, archiveConfig)
File "/usr/local/lib/python2.7/dist-packages/whisper.py", line 209, in create
  fh = open(path,'wb')
exceptions.IOError: [Errno 2] No such file or directory: '/opt/graphite/storage/whisper/carbon/agents/master01/creates.wsp'

I installed and configured Graphite 0.9.8 according to the instructions on http://readthedocs.org/docs/graphite/en/latest/install.html. Whisper was manually installed rather than using the binary package. Google is unhelpful searching for "graphite carbon no such file or directory".

I have plenty of free diskspace. Carbon cache runs as the graphite user, and graphite owns /opt/graphite/storage and descendants:

# ls -ld /opt/graphite/storage/
drwxr-xr-x 6 graphite graphite 4096 2011-09-08 01:34 /opt/graphite/storage/
# ls -ld /opt/graphite/storage/whisper/
drwxr-xr-x 2 graphite graphite 4096 2011-09-08 01:34 /opt/graphite/storage/whisper/

storage-schemas.conf says:

# cat /opt/graphite/conf/storage-schemas.conf
[everything]
priority = 100
pattern = .*
retentions = 10s:1d,1m:7d,15m:30d

I hope this means "keep 10s precision data for 1 day, then minutely precision for 7 days, then 15 minute precision data for 30 days". By the way, does this mean I'll have 30 + 7 + 1 days of data in the archive, or 30 days total?

I'm starting carbon-cache.py under daemontools control, using the following:

PATH=/usr/local/bin:/usr/bin
cd /opt/graphite/
exec /usr/local/bin/setuidgid graphite /usr/bin/python bin/carbon-cache.py --debug start

I'm using --debug to keep carbon-cache in the foreground, because it makes carbon-cache much easier to manage under daemontools.

Any help greatly appreciated in starting making my carbon-cache writing data to disk.

Thanks!
François

Question information

Language:
English Edit question
Status:
Solved
For:
Graphite Edit question
Assignee:
No assignee Edit question
Solved by:
François Beausoleil
Solved:
Last query:
Last reply:

This question was reopened

Revision history for this message
Aman Gupta (tmm1) said : 		#1

The retention syntax you're using only works on trunk I think. Try installing from trunk, it is generally better to use than the 0.9.8 release. Run `python setup.py install --force` on all three setup.py files to upgrade.

Revision history for this message
François Beausoleil (fbeausoleil) said : 		#2

I'm inexperienced, and on a fresh VM I get an error running syncdb:

# python manage.py syncdb
Error: No module named tagging

This seems to be related to INSTALLED_APPS in settings.py, which was last modified by chrismd in r541: http://bazaar.launchpad.net/~graphite-dev/graphite/main/revision/541

Is there a dependency I'm missing?

Revision history for this message
François Beausoleil (fbeausoleil) said : 		#3

Yup, I was missing django-tagging, which isn't checked in check-dependencies.py. How can I add that check?

Revision history for this message
François Beausoleil (fbeausoleil) said : 		#4

Using trunk, I still have errors creating files. Same Traceback, different metric. I chown -R graphite:graphite /opt/graphite, thus everything belongs to the correct user now, rather than only storage/. I'll sleep on it and see if some Web Fairy helped me before I wake up :)

Revision history for this message
François Beausoleil (fbeausoleil) said : 		#5

I'm looking at whisper.py, line 289, which is the line mentioned in the Traceback (from trunk). I'm looking at the code http://bazaar.launchpad.net/~graphite-dev/graphite/main/view/head:/whisper/whisper.py#L289 and I don't understand why I'd get a No such file or directory when line 286 says "if os.path.exists(): raise" ? Also, Carbon is apparently creating the directory as well according to this http://bazaar.launchpad.net/~graphite-dev/graphite/main/view/head:/carbon/lib/carbon/writer.py#L112

Even though I get those errors all the time in my log files, I can use the Graphite Composer and get graphs using the names that are referenced in the error message.

What am I missing?
François

Revision history for this message
chrismd (chrismd) said : 		#6

Whisper doesn't create the directories, only the wsp file. Carbon is supposed to create the directories but it sounds like that is not happening for some reason. Do you have anything under /opt/graphite/storage/whisper/ or is it completely empty?

Try running carbon-cache.py --debug in a terminal as the graphite user and if you get the same stacktrace try to touch the file it failed to create, ie: touch /opt/graphite/storage/whisper/foo/bar/baz.wsp. Also try doing, mkdir -p -m 755 /opt/graphite/storage/whisper/foo/bar/. Post the output of those commands and I'm sure we'll figure out what is going wrong.

Revision history for this message
François Beausoleil (fbeausoleil) said : 		#7

/opt/graphite/storage/whisper wasn't empty: the named directories were present and accounted for. I was even able to browse the metrics from the Graphite UI. I've setup a new instance to try to reproduce the problem. I don't see the issue in production and on the new instance. It may have been a permission problem, as I've changed my instance setup script quite a few times.

Must have been something on my end. Maybe a note on the installation instructions to make /opt/graphite owned by the user under which carbon / graphite run as?