Sidestepping Carbon?

1.) In what format are the data stored on your statistics server.
2.) What version of Graphite are you using?
I did a
$bzr log -v >> log.txt
$ grep finders log.txt
$ grep readers log.txt

didn't return any hits.
This means I can't find any reference to those two files in any revision from 1 to 559 of the Bazaar codebase. They should at least show up once in the verbose log when they were first added or the initial commit in revision 1.

Is it possible those files were written by someone at your company and that's why it looks like they've been removed (they were never part of the codebase to begin with?)

Chris has been working on this way longer than I have, so it's possible those are just really old files that I've never heard of and were never added to Bazaar. Chris care to comment on sidestepping Carbon? I know we can read RRD files, so it's at least possible to read other kinds of data than Whisper / Ceres...

1)

I can't really specify much about our server's storage, but the stats server listens for requests and serves back a JSON format in a response.

2)

Okay, so after some investigation the version we were using before looking at bazaar, was graphite 1.1. And that does have webapp/graphite/finders.py and webapp/graphite/readers.py. We certainly didn't have to add those ourselves, but we did extend this to allow requests to our stats server.

I noticed that webapp/graphite/storage.py in the trunk version has some similar sort of functionality, Branch objects have a fetch method. I don't know that it's 1-to-1 mapping to the graphite 1.1. functionality, but it seems like I could potentially write some Branch and Leaf type things to deal with that? Do you see any obstacles to doing this in terms of converting the old finder/reader behaviour?

If that doesn't work, is moving events into version 1.1 on the roadmap?

I apologize for being vague, but hopefully some of this is helpful context?

Yes, Chris has been working to merge changes into 1.1.

Hopefully once that's complete you can use all the nifty features in trunk.

Andrew Crowell <email address hidden> wrote:

>Question #171056 on Graphite changed:
>https://answers.launchpad.net/graphite/+question/171056
>
> Status: Needs information => Open
>
>Andrew Crowell gave more information on the question:
>1)
>
>I can't really specify much about our server's storage, but the stats
>server listens for requests and serves back a JSON format in a response.
>
>2)
>
>Okay, so after some investigation the version we were using before
>looking at bazaar, was graphite 1.1. And that does have
>webapp/graphite/finders.py and webapp/graphite/readers.py. We certainly
>didn't have to add those ourselves, but we did extend this to allow
>requests to our stats server.
>
>I noticed that webapp/graphite/storage.py in the trunk version has some
>similar sort of functionality, Branch objects have a fetch method. I
>don't know that it's 1-to-1 mapping to the graphite 1.1. functionality,
>but it seems like I could potentially write some Branch and Leaf type
>things to deal with that? Do you see any obstacles to doing this in
>terms of converting the old finder/reader behaviour?
>
>If that doesn't work, is moving events into version 1.1 on the roadmap?
>
>I apologize for being vague, but hopefully some of this is helpful
>context?
>
>--
>You received this question notification because you are a member of
>graphite-dev, which is an answer contact for Graphite.
>
>_______________________________________________
>Mailing list: https://launchpad.net/~graphite-dev
>Post to : <email address hidden>
>Unsubscribe : https://launchpad.net/~graphite-dev
>More help : https://help.launchpad.net/ListHelp

Right so the reader/finder API is part of the 1.1 branch. I tried maintaining two separate branches for a while but it proved to be unworkable so 1.1 is frozen for now, though all of the good bits (including the refactored storage API with readers & finders) is coming to trunk once 0.9.9 is out the door.

Andy (the original poster) has been working with me, and I am trying reconcile where to go next with Graphite.

We started using Graphite 1.1 (r444) using reader/finder to integrate Graphite with our custom remote datastore and have been pleased with the results.

Now we want to try the lp:~lucio.torre/graphite/add-events, which appears to be merged on the 1.0 branch.

What would be best approach in the long term.

1/ go to the 1.0 branch and re-integrate graphite with our custom remote datastore
2/ try to merge lp:~lucio.torre/graphite/add-events with our existing integration with our customer remote datastore on the 1.1 branch.

When (i.e. calendar time) do you expect 1.0 to have the refactored storage API?

Thanks,
Jon

Personally I think the best approach is to get on trunk once the refactored storage API is integrated. The current priority list is to first release 0.9.9 then work on integrating 1.1 features into trunk. The 0.9.9 release is getting near but we try to iterate through every bug before cutting a release and there are a few left. So my subtle suggestion is to help with the bug queue so we can get 0.9.9 out and then get on the storage refactor :). I do expect 0.9.9 to be released within the next week or so, it's about two weeks overdue already.

Once 0.9.9 is out, I will guesstimate the merge process will take about 2 weeks to get it into trunk. There isn't a firm timetable for the next stable release after 0.9.9 though (probably December/January-ish), as it will be the 1.0 release and will take a few months to iron out bugs from the all the new stuff merging in. So if you're ok with running on trunk for a while (and given that you use 1.1 I assume you are, and that you wear a cowboy hat) you should be fine.

Andrew's challenge involves updating his company's Graphite version while maintaining their existing "load on demand" data retrieval from the stats server. Here are some potential solutions:

Migrate to newer Graphite version with customization:
Use Graphite web app interface: While the Reader/Finder interface is gone, the newer one provides APIs for fetching data on demand. Implement scripts in your stats server to call these APIs instead of using the old "readers.py" and "finders.py" approaches. This lets you
keep Graphite for graphing without pushing data to Carbon.

Customize Carbon data pipeline: Set up Carbon as a receiver but configure it to only store aggregated data (not raw data) to minimize redundancy. Consider using Carbon's whisper database API to query aggregated data from your stats server directly, bypassing the continuous push.
You can also read it here: https://bartlebyandsage.com/