Graphite

Carbon aggregation and rewrite rules

Asked by Curt Kersey on 2012-10-29

I have a setup where I am receiving metrics via UDP from many hosts – see carbon.conf configuration file below. I need to rewrite some of the metric names before they are saved in the database. In reviewing the documentation, it is possible to do that via the rewrite-rules.conf, which requires that I first enable carbon aggregation. Since I am already receiving metrics via UDP, I would like to have the carbon aggregator to receive those via the UDP protocol, apply any rewrite rules, then store the metrics. In this case, the carbon aggregator is the same as my carbon cache host.

In trying to get this to work, it does not look like carbon aggregator will start on a UDP port. When I have started and stopped the daemon, it never starts a service on the UDP port specified - I have looked at the log messages that get generated and 'ss -ua' command.

Note: the aggregation-rules.conf and rewrite-rules.conf are files that exist, but they are empty – zero bytes.

Questions:
1. Can the aggregator run on a UDP port?
2. Is my configuration file correct?
3. Anything else I should be looking at?

carbon.conf file:
[cache]
LOCAL_DATA_DIR = /raid/whisper/
USER = carbon
MAX_CACHE_SIZE = inf
MAX_UPDATES_PER_SECOND = inf
MAX_CREATES_PER_MINUTE = inf
LINE_RECEIVER_INTERFACE = 0.0.0.0
LINE_RECEIVER_PORT = 2023
ENABLE_UDP_LISTENER = True
UDP_RECEIVER_INTERFACE = 0.0.0.0
UDP_RECEIVER_PORT = 2023
PICKLE_RECEIVER_INTERFACE = 0.0.0.0
PICKLE_RECEIVER_PORT = 2004
USE_INSECURE_UNPICKLER = False
CACHE_QUERY_INTERFACE = 0.0.0.0
CACHE_QUERY_PORT = 7002
USE_FLOW_CONTROL = True
LOG_UPDATES = False
WHISPER_AUTOFLUSH = False

[relay]
LINE_RECEIVER_INTERFACE = 0.0.0.0
LINE_RECEIVER_PORT = 2013
PICKLE_RECEIVER_INTERFACE = 0.0.0.0
PICKLE_RECEIVER_PORT = 2014
RELAY_METHOD = rules
REPLICATION_FACTOR = 1
DESTINATIONS = 127.0.0.1:2004
MAX_DATAPOINTS_PER_MESSAGE = 500
MAX_QUEUE_SIZE = 10000
USE_FLOW_CONTROL = True

[aggregator]
LINE_RECEIVER_INTERFACE = 0.0.0.0
LINE_RECEIVER_PORT = 2003
PICKLE_RECEIVER_INTERFACE = 0.0.0.0
PICKLE_RECEIVER_PORT = 2024
ENABLE_UDP_LISTENER = True
UDP_RECEIVER_INTERFACE = 0.0.0.0
UDP_RECEIVER_PORT = 2003
DESTINATIONS = 127.0.0.1:2023
REPLICATION_FACTOR = 1
MAX_QUEUE_SIZE = 10000
USE_FLOW_CONTROL = True
MAX_DATAPOINTS_PER_MESSAGE = 500
MAX_AGGREGATION_INTERVALS = 5

Thanks,
--Curt

Question information

Language:: English Edit question

Status:: Solved

For:: Graphite Edit question

Assignee:: No assignee Edit question

Solved by:: Curt Kersey

Solved:: 2012-11-13

Last query:: 2012-11-13

Last reply:: 2012-11-05

Revision history for this message

Nicholas Leskiw (nleskiw) said on 2012-10-31:

I tested it, looks like the aggregator sends pickle data, not line data.

Point the aggregator destination to the pickle listener interface and that should solve your problem.

In your case I *believe* it would look like this:

[aggregator]
...
DESTINATIONS = 127.0.0.1:2004
...

Revision history for this message

Curt Kersey (curtkersey) said on 2012-11-05:

Nicholas,
Thanks for testing and getting back to me. I apologize for my delay, but I took time to update software to the latest version in the trunk. Looks like there were some changes along the way (I was running vanilla 0.9.10 release) that helped me to get closer; however, I am still not able to get this working.

I have looked for a more detailed overview of setting up the Carbon Aggregation than is at http://graphite.readthedocs.org/en/1.0/config-carbon.html, but I cannot find it. I want to make sure that I am fundamentally setting it up correctly.

Currently, I have the following in my carbon.conf file. I made a few more changes than you suggested, but only to simplify the configuration. I am not using a relay, so I could remove that section completely to stick with my 'more simple' theme.

[cache]
LOCAL_DATA_DIR = /raid/whisper/
USER = carbon
MAX_CACHE_SIZE = inf
MAX_UPDATES_PER_SECOND = inf
MAX_CREATES_PER_MINUTE = inf
PICKLE_RECEIVER_INTERFACE = 0.0.0.0
PICKLE_RECEIVER_PORT = 2004
USE_INSECURE_UNPICKLER = False
CACHE_QUERY_INTERFACE = 0.0.0.0
CACHE_QUERY_PORT = 7002
USE_FLOW_CONTROL = True
LOG_UPDATES = False
WHISPER_AUTOFLUSH = False

[aggregator]
ENABLE_UDP_LISTENER = True
UDP_RECEIVER_INTERFACE = 0.0.0.0
UDP_RECEIVER_PORT = 2003
DESTINATIONS = 127.0.0.1:2004
REPLICATION_FACTOR = 1
MAX_QUEUE_SIZE = 10000
USE_FLOW_CONTROL = True
MAX_DATAPOINTS_PER_MESSAGE = 500
MAX_AGGREGATION_INTERVALS = 5

In order to start the system, I start the carbon-aggregator.py by typing the following:
/opt/graphite/bin/carbon-aggegator.py start

I can see from the log files that it is starting. If I do a 'ss -a', I can also see the daemons running. I then start sending a diagnostic metric; however, that metric never has a value in the database -- it always has null values when I query it via the render interface. I do not believe the metric is ever getting stored -- either it is not forwarded to the pickle port correctly or it is getting dropped somewhere. I want my carbon cache and carbon aggregator server to be the same.

Relating to the networking daemons that are running -- I can see the Pickler port is running on a TCP port. The connection from the localhost to itself is visible when I do the command 'ss -a -t'. I also get a log message in the carbon-aggregator-a/clients.log that connectionMade.

One more bit of information, I have the following files in the conf directory, but they are just empty files. I was hoping to get this working in a simple case, then add rules to make it more complicated:
    aggregation-rules.conf
    rewrite-rules.conf
    storage-aggregation.conf

Questions:
1- Is it possible to aggregate & store values on the same host?

2- Do I need something in the configuration files I have now that are empty?

3- How can I see what data the pickle port is receiving, so I can debug what is going on?

4- Is there a document to give an example of how to setup the aggregation?

Note: I am happy to contribute a document to show how my setup is working once it is complete.

--Curt

Nicholas,
Thanks for testing and getting back to me.  I apologize for my delay, but I took time to update software to the latest version in the trunk.  Looks like there were some changes along the way (I was running vanilla 0.9.10 release) that helped me to get closer; however, I am still not able to get this working.

I have looked for a more detailed overview of setting up the Carbon Aggregation than is at http://graphite.readthedocs.org/en/1.0/config-carbon.html, but I cannot find it.  I want to make sure that I am fundamentally setting it up correctly.

Currently, I have the following in my carbon.conf file.  I made a few more changes than you suggested, but only to simplify the configuration.  I am not using a relay, so I could remove that section completely to stick with my 'more simple' theme.

In order to start the system, I start the carbon-aggregator.py by typing the following:
    /opt/graphite/bin/carbon-aggegator.py start

I can see from the log files that it is starting.  If I do a 'ss -a', I can also see the daemons running.  I then start sending a diagnostic metric; however, that metric  never has a value in the database -- it always has null values when I query it via the render interface.  I do not believe the metric is ever getting stored -- either it is not forwarded to the pickle port correctly or it is getting dropped somewhere.  I want my carbon cache and carbon aggregator server to be the same.

Relating to the networking daemons that are running -- I can see the Pickler port is running on a TCP port.  The connection from the localhost to itself is visible when I do the command 'ss -a -t'.  I also get a log message in the carbon-aggregator-a/clients.log that connectionMade.

One more bit of information, I have the following files in the conf directory, but they are just empty files.  I was hoping to get this working in a simple case, then add rules to make it more complicated:
    aggregation-rules.conf
    rewrite-rules.conf
    storage-aggregation.conf

Questions:
1- Is it possible to aggregate & store values on the same host?

2- Do I need something in  the configuration files I have now that are empty?

3- How can I see what data the pickle port is receiving, so I can debug what is going on?

4- Is there a document to give an example of how to setup the aggregation?

Note: I am happy to contribute a document to show how my setup is working once it is complete.

--Curt

Revision history for this message

Nicholas Leskiw (nleskiw) said on 2012-11-05:

1. Yes, they can run on the same machine (on different ports, of course.)
2. You shouldn't need anything in those config files to see non-aggregated
metrics sent to aggregator / carbon-cache
3. Unfortunately tcpdump / wireshark is the only option AFAIK
4. There's a bit of info in aggregation-rules.conf.example file.
5. We'd be more than happy to accept any documentation. Thank you for
offering.

I see that you didn't specify a line receiver in carbon cache. Can you
check if it's defaulting to port 2003 and blocking aggregator from
listening on 2003? (Give it 2023 or something.)

That would explain why you a) see something listening and b) don't see any
metrics (because they're UDP but should be TCP). I'm not convinced that's
the issue since you see connections in the aggregator log, but it's worth a
shot.

Otherwise do a tcpdump and trace whatever's going on when you send a single
UDP message to the aggregator.

-Nick
On Nov 5, 2012 2:21 PM, "Curt Kersey" <email address hidden>
wrote:

> Question #212686 on Graphite changed:
> https://answers.launchpad.net/graphite/+question/212686
>
> Status: Answered => Open
>
> Curt Kersey is still having a problem:
> Nicholas,
> Thanks for testing and getting back to me. I apologize for my delay, but
> I took time to update software to the latest version in the trunk. Looks
> like there were some changes along the way (I was running vanilla 0.9.10
> release) that helped me to get closer; however, I am still not able to get
> this working.
>
> I have looked for a more detailed overview of setting up the Carbon
> Aggregation than is at http://graphite.readthedocs.org/en/1.0/config-
> carbon.html, but I cannot find it. I want to make sure that I am
> fundamentally setting it up correctly.
>
> Currently, I have the following in my carbon.conf file. I made a few
> more changes than you suggested, but only to simplify the configuration.
> I am not using a relay, so I could remove that section completely to
> stick with my 'more simple' theme.
>
> [cache]
> LOCAL_DATA_DIR = /raid/whisper/
> USER = carbon
> MAX_CACHE_SIZE = inf
> MAX_UPDATES_PER_SECOND = inf
> MAX_CREATES_PER_MINUTE = inf
> PICKLE_RECEIVER_INTERFACE = 0.0.0.0
> PICKLE_RECEIVER_PORT = 2004
> USE_INSECURE_UNPICKLER = False
> CACHE_QUERY_INTERFACE = 0.0.0.0
> CACHE_QUERY_PORT = 7002
> USE_FLOW_CONTROL = True
> LOG_UPDATES = False
> WHISPER_AUTOFLUSH = False
>
>
> [relay]
> LINE_RECEIVER_INTERFACE = 0.0.0.0
> LINE_RECEIVER_PORT = 2013
> PICKLE_RECEIVER_INTERFACE = 0.0.0.0
> PICKLE_RECEIVER_PORT = 2014
> RELAY_METHOD = rules
> REPLICATION_FACTOR = 1
> DESTINATIONS = 127.0.0.1:2004
> MAX_DATAPOINTS_PER_MESSAGE = 500
> MAX_QUEUE_SIZE = 10000
> USE_FLOW_CONTROL = True
>
>
> [aggregator]
> ENABLE_UDP_LISTENER = True
> UDP_RECEIVER_INTERFACE = 0.0.0.0
> UDP_RECEIVER_PORT = 2003
> DESTINATIONS = 127.0.0.1:2004
> REPLICATION_FACTOR = 1
> MAX_QUEUE_SIZE = 10000
> USE_FLOW_CONTROL = True
> MAX_DATAPOINTS_PER_MESSAGE = 500
> MAX_AGGREGATION_INTERVALS = 5
>
> In order to start the system, I start the carbon-aggregator.py by typing
> the following:
> /opt/graphite/bin/carbon-aggegator.py start
>
> I can see from the log files that it is starting. If I do a 'ss -a', I
> can also see the daemons running. I then start sending a diagnostic
> metric; however, that metric never has a value in the database -- it
> always has null values when I query it via the render interface. I do
> not believe the metric is ever getting stored -- either it is not
> forwarded to the pickle port correctly or it is getting dropped
> somewhere. I want my carbon cache and carbon aggregator server to be
> the same.
>
> Relating to the networking daemons that are running -- I can see the
> Pickler port is running on a TCP port. The connection from the
> localhost to itself is visible when I do the command 'ss -a -t'. I also
> get a log message in the carbon-aggregator-a/clients.log that
> connectionMade.
>
> One more bit of information, I have the following files in the conf
> directory, but they are just empty files. I was hoping to get this working
> in a simple case, then add rules to make it more complicated:
> aggregation-rules.conf
> rewrite-rules.conf
> storage-aggregation.conf
>
> Questions:
> 1- Is it possible to aggregate & store values on the same host?
>
> 2- Do I need something in the configuration files I have now that are
> empty?
>
> 3- How can I see what data the pickle port is receiving, so I can debug
> what is going on?
>
> 4- Is there a document to give an example of how to setup the
> aggregation?
>
> Note: I am happy to contribute a document to show how my setup is
> working once it is complete.
>
> --Curt
>
> --
> You received this question notification because you are a member of
> graphite-dev, which is an answer contact for Graphite.
>
> _______________________________________________
> Mailing list: https://launchpad.net/~graphite-dev
> Post to : <email address hidden>
> Unsubscribe : https://launchpad.net/~graphite-dev
> More help : https://help.launchpad.net/ListHelp
>

I see that you didn't specify a line receiver in carbon cache. Can you
check if it's defaulting to port 2003 and blocking aggregator from
listening on 2003?  (Give it 2023 or something.)

That would explain why you a) see something listening and b) don't see any
metrics (because they're UDP but should be TCP).  I'm not convinced that's
the issue since you see connections in the aggregator log, but it's worth a
shot.

Otherwise do a tcpdump and trace whatever's going on when you send a single
UDP message to the aggregator.

-Nick
 On Nov 5, 2012 2:21 PM, "Curt Kersey" <question212686@answers.launchpad.net>
wrote:

> Question #212686 on Graphite changed:
> https://answers.launchpad.net/graphite/+question/212686
>
>     Status: Answered => Open
>
> Curt Kersey is still having a problem:
> Nicholas,
> Thanks for testing and getting back to me.  I apologize for my delay, but
> I took time to update software to the latest version in the trunk.  Looks
> like there were some changes along the way (I was running vanilla 0.9.10
> release) that helped me to get closer; however, I am still not able to get
> this working.
>
> I have looked for a more detailed overview of setting up the Carbon
> Aggregation than is at http://graphite.readthedocs.org/en/1.0/config-
> carbon.html, but I cannot find it.  I want to make sure that I am
> fundamentally setting it up correctly.
>
> Currently, I have the following in my carbon.conf file.  I made a few
> more changes than you suggested, but only to simplify the configuration.
> I am not using a relay, so I could remove that section completely to
> stick with my 'more simple' theme.
>
> [cache]
> LOCAL_DATA_DIR = /raid/whisper/
> USER = carbon
> MAX_CACHE_SIZE = inf
> MAX_UPDATES_PER_SECOND = inf
> MAX_CREATES_PER_MINUTE = inf
> PICKLE_RECEIVER_INTERFACE = 0.0.0.0
> PICKLE_RECEIVER_PORT = 2004
> USE_INSECURE_UNPICKLER = False
> CACHE_QUERY_INTERFACE = 0.0.0.0
> CACHE_QUERY_PORT = 7002
> USE_FLOW_CONTROL = True
> LOG_UPDATES = False
> WHISPER_AUTOFLUSH = False
>
>
> [relay]
> LINE_RECEIVER_INTERFACE = 0.0.0.0
> LINE_RECEIVER_PORT = 2013
> PICKLE_RECEIVER_INTERFACE = 0.0.0.0
> PICKLE_RECEIVER_PORT = 2014
> RELAY_METHOD = rules
> REPLICATION_FACTOR = 1
> DESTINATIONS = 127.0.0.1:2004
> MAX_DATAPOINTS_PER_MESSAGE = 500
> MAX_QUEUE_SIZE = 10000
> USE_FLOW_CONTROL = True
>
>
> [aggregator]
> ENABLE_UDP_LISTENER = True
> UDP_RECEIVER_INTERFACE = 0.0.0.0
> UDP_RECEIVER_PORT = 2003
> DESTINATIONS = 127.0.0.1:2004
> REPLICATION_FACTOR = 1
> MAX_QUEUE_SIZE = 10000
> USE_FLOW_CONTROL = True
> MAX_DATAPOINTS_PER_MESSAGE = 500
> MAX_AGGREGATION_INTERVALS = 5
>
> In order to start the system, I start the carbon-aggregator.py by typing
> the following:
>     /opt/graphite/bin/carbon-aggegator.py start
>
> I can see from the log files that it is starting.  If I do a 'ss -a', I
> can also see the daemons running.  I then start sending a diagnostic
> metric; however, that metric  never has a value in the database -- it
> always has null values when I query it via the render interface.  I do
> not believe the metric is ever getting stored -- either it is not
> forwarded to the pickle port correctly or it is getting dropped
> somewhere.  I want my carbon cache and carbon aggregator server to be
> the same.
>
> Relating to the networking daemons that are running -- I can see the
> Pickler port is running on a TCP port.  The connection from the
> localhost to itself is visible when I do the command 'ss -a -t'.  I also
> get a log message in the carbon-aggregator-a/clients.log that
> connectionMade.
>
> One more bit of information, I have the following files in the conf
> directory, but they are just empty files.  I was hoping to get this working
> in a simple case, then add rules to make it more complicated:
>     aggregation-rules.conf
>     rewrite-rules.conf
>     storage-aggregation.conf
>
> Questions:
> 1- Is it possible to aggregate & store values on the same host?
>
> 2- Do I need something in  the configuration files I have now that are
> empty?
>
> 3- How can I see what data the pickle port is receiving, so I can debug
> what is going on?
>
> 4- Is there a document to give an example of how to setup the
> aggregation?
>
> Note: I am happy to contribute a document to show how my setup is
> working once it is complete.
>
> --Curt
>
> --
> You received this question notification because you are a member of
> graphite-dev, which is an answer contact for Graphite.
>
> _______________________________________________
> Mailing list: https://launchpad.net/~graphite-dev
> Post to     : graphite-dev@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~graphite-dev
> More help   : https://help.launchpad.net/ListHelp
>

Revision history for this message

Curt Kersey (curtkersey) said on 2012-11-13:

My problem is solved. The biggest issue is carbon-cache using its default configuration if something is not in the configuration file. In this case, carbon-cache was trying to use the same port as carbon-aggregator. Depending on which one I started first was the one that got running.

I am updating the README.md file to add a section on use of carbon-aggregator, which will hopefully make this and a couple of other things more clear to other new users.

Thanks,
--Curt

To post a message you must log in.

Ask a question

Edit question

Graphite

Carbon aggregation and rewrite rules

Question information

Related bugs

Related FAQ:

Subscribers