DB Configuration confusion

Asked by Mark Seger

I'm a little confused between the documentation and how the software is installed. The documentation talks about schema files in /usr/local/graphite/storage/schema but I have no such directory as graphite has installed in /opt/graphite. I do have a storage directory but no schema files.

Earlier the installation talks about /opt/graphite/conf/storage-schemas.conf and it's not clear to me if THIS is the schema and you simply append stanzas to it OR if you create separate schema files. Further, since the instructions tell you to generate some data before talking about schema files, the implication to me is graphite knows what to do with the data w/o the schema files.

As for the schema.conf file, since I want to send data every 10 seconds and retain it for a week, should it look like this:

[everything_10sec_1week]
priority = 100
pattern = .*
retentions = 10:60480

The reason I ask is when I tried sending data to graphite via collectl it happily created a bunch of files in /opt/graphite/storage/whisper that are named for my metrics.

Finally, how does one verify the data is actually being stored somewhere?

-mark

Question information

Language:
English Edit question
Status:
Solved
For:
Graphite Edit question
Assignee:
No assignee Edit question
Solved by:
Mark Seger
Solved:
Last query:
Last reply:
Revision history for this message
Michael Leinartas (mleinartas) said :
#1

You'll want to use the newer docs linked from the wiki at http://readthedocs.org/docs/graphite/en/latest/install.html for installation as the ones on the Wiki are quite out of date - the storage schemas are in /opt/graphite/conf/storage-schemas.conf and yes, you append new stanzas to it. The first pattern that matches is the one that's applied, so be sure to replace the default or put it at the end so it doesnt take precedence over your customizations.

This should give you more detail on storage-schemas: http://readthedocs.org/docs/graphite/en/latest/config-carbon.html#storage-schemas-conf

If carbon generated a bunch of *.wsp files in your storage directory, you're already storing some data. You can spot check a few of the files to ensure data is in them with the whisper-fetch.py command. By default it will show you the last 24 hours of data stored in the file. If the files were created before you added your storage-schema you'll need to either remove them and allow them to be recreated (easier) or run whisper-resize.py to resize them to your desired retention.

Revision history for this message
Mark Seger (mjseger) said :
#2

very cool. I'll have to give the newer docs a read. would sure be nice if even if there isn't time to redo the grapite website to at least say to look elsewhere. ;(

so I looked at my data and realized the interval was set to 1 minute which is far too high, so I dropped it to 10 seconds, killed carbon, removed the old files and restarted it. Data still getting stored in 1 minute buckets. clearly there is more to getting things synchronized.

so to be clear, if I don't update anything in the conf file all files will be configured the same way? that's exactly what I want.

now if I can only get the webapp working. ;(

fwiw - I do seem to have collectl working correctly and logging data to graphite. if anyone is interested, collectl can grabs tons more stuff than most other monitoring tools and do it all from one place in a very light manner. here's a quick overview of the types of data it can collect:

The following subsystems can be specified in any combinations with -s or
--subsys in both record and playbackmode. [default=bcdfijlmnstx]

These generate summary, which is the total of ALL data for a particular type
  b - buddy info (memory fragmentation)
  c - cpu
  d - disk
  f - nfs
  i - inodes
  j - interrupts by CPU
  l - lustre
  m - memory
  n - network
  s - sockets
  t - tcp
  x - interconnect (currently supported: Infiniband and Quadrics)
  y - slabs

These generate detail data, typically but not limited to the device level

  C - individual CPUs, including interrupts if -sj or -sJ
  D - individual Disks
  E - environmental (fan, power, temp) [requires ipmitool]
  F - nsf data
  J - interrupts by CPU by interrupt number
  L - lustre
  N - individual Networks
  T - tcp details (lots of data!)
  X - interconnect ports/rails (Infiniband/Quadrics)
  Y - slabs/slubs
  Z - processes

I'd love to have someone take it for a spin with my test code and once we're all happy I can make it part of the standard package. To learn more about collectl see collectl.sourceforge.net

-mark

Revision history for this message
Mark Seger (mjseger) said :
#3

ok, so I'm back to being confused and maybe because I didn't see any description of how to change the schema. I tried again to get carbon to log at 10 second intervals so I killed it, again removed /opt/graphite/storage/whisper/*. I then restarted carbon again. when I log data I can see any data files being created nor is anything being written the the create or update logs.

what did I do wrong? do I need to restart something? reboot?

-mark

Revision history for this message
Mark Seger (mjseger) said :
#4

I'm good to go. Just took paying more careful attention to what I was doing
-mark