counter and rate data?

Yes, the documentation is certainly lacking in this area but fortunately the functionality is there. The easiest way to get the rate of your counter metrics is to apply the derivative function to your target metrics in the UI.

To do this, first view your metric in the main window. Then click the 'Targets' button to bring up the Targets dialog window. Left click to select your target & right-click for a menu (there's stupid bug still open as to why you can't simply right-click without left-clicking the metric to select it first...). The right-click menu will have an 'Apply Function' sub-menu, which will have many functions to choose from including the one you are looking for, 'Derivative'. Click that and you're function should be transformed to show the rate of change of your counter metric rather than it's usual value. You can un-apply a function by choosing 'Remove outer call' from the 'Apply Function' menu. You can also re-add the same counter metric again to view it's actual values overlayed with their derivative.

Also note that some functions, such as sumSeries() can operate on multiple targets (Ctrl-click to select multiple targets before right clicking).

Oh also forgot to mention you do not need to use the main browser UI in order to apply functions, you can do it directly by modifying the URL of your image. If for example your image's URL is:

http://myserver/render?target=x.y.z&from=-2h

You could apply the derivative function like so:

http://myserver/render?target=derivative(x.y.z)&from=-2h

If you are unsure how to change your URL for other functions, I'd recommend doing it once in the window interface then looking at the image's URL to see what the exact notation is for that particular function (they are all pretty simple and very similar to this one).

Great stuff. Can it be combined with other functions? I was trying to use the average / moving average function but I'm not sure I understand how they work. e.g. I'd like to see the derivative averaged over a period of 10 minutes to pickup the trends I'm looking for?

Actually while I'm there, I was going to suggest a function to translate the time scale by an offset, so you can build graphs to compare a curve against yesterday's, or last week's.

Ah .. moving average controlled by a number of data points rather than a time period is unfortunate, typically my counters report whenever they change, so they are pretty irregular.

With this particular case I'm working on I already needed to change the behavior to regular reporting so that's fine.

Thanks chrismd, that solved my question.

Sorry but I think I have confused you about how movingAverage() works. It is based on time, but the units are determined by your data interval (minutely by default). So it sounds like you have data that is not sent totally regularly and that is totally fine. Unless you tweaked carbon's storage schemas file your database files will be storing data points with 1-minute precision (so most data points will have a value of 'None' if your data is not actually being sent every minute, and if data was sent more frequently than a minute only the last value for each minute would actually get stored, so the database precision sometimes needs to be tweaked). The 'None' data points are ignored by the averaging functions.

So the short version is, by default doing "target=movingAverage(x.y.z, 10)" will give you a moving average of the x.y.z metric with a 10-minute window. But say you configured carbon to store the x.y.* metrics with 5-minute precision instead of 1-minute precision. Then movingAverage(x.y.z, 10) would give you a 50-minute window. I hope that helps.