Data does not get collected

Asked by GunsNRoses

I am dealing with a really weird problem . I have setup graphite and managed to run the client example without a problem, data was collected as intended.

Once I tried to push some data from a remote machine, however, data seems to be dropped/rejected/missing.

I can telnet to the remote server on the graphite ip/port, raw telnet messages do not work.

The carbon logs in debug mode show the following:

16/04/2013 21:11:23 :: [listener] MetricLineReceiver connection with REMOTE_SERVER_IP:47704 established
16/04/2013 21:11:23 :: [listener] MetricLineReceiver connection with REMOTE_SERVER_IP:47704 closed cleanly
16/04/2013 21:11:23 :: [console] Sorted 1 cache queues in 0.000020 seconds
16/04/2013 21:11:34 :: [listener] MetricLineReceiver connection with REMOTE_SERVER_IP:47718 established
16/04/2013 21:11:34 :: [listener] MetricLineReceiver connection with REMOTE_SERVER_IP:47718 closed cleanly
16/04/2013 21:11:34 :: [console] Sorted 1 cache queues in 0.000014 seconds
16/04/2013 21:11:45 :: [listener] MetricLineReceiver connection with REMOTE_SERVER_IP:47731 established
16/04/2013 21:11:45 :: [listener] MetricLineReceiver connection with REMOTE_SERVER_IP:47731 closed cleanly
16/04/2013 21:11:45 :: [console] Sorted 1 cache queues in 0.000025 seconds
16/04/2013 21:11:52 :: [console] /opt/graphite/conf/storage-aggregation.conf not found, ignoring.
16/04/2013 21:11:53 :: [console] Sorted 13 cache queues in 0.000033 seconds
16/04/2013 21:11:56 :: [listener] MetricLineReceiver connection with REMOTE_SERVER_IP:47741 established
16/04/2013 21:11:56 :: [listener] MetricLineReceiver connection with REMOTE_SERVER_IP:47741 closed cleanly
16/04/2013 21:11:56 :: [console] Sorted 1 cache queues in 0.000028 seconds

When checking carbon's internal stats, I see there are update operations, metrics received operations, commited points operations, but there are no points per updates, which seems weird.

The thing that is bugging me the most is that the data push works from localhost, it does not work from a remote machine. Again telnetting from the remote machine is ok, so I don't think it is a firewall issue.

I tried to push to the following schemas every 15 seconds:

[default_1min_for_1day]
pattern = .*
retentions = 60s:1d

[schema_name]
priority = 100
pattern = ^pigeon\.
retentions = 10s:14d

I tried messages that end with "\n" and with "\r\n" both do not work.

Anyone dealt with something similar?

The versions I am using is 9.10 stable.

Question information

Language:
English Edit question
Status:
Solved
For:
Graphite Edit question
Assignee:
No assignee Edit question
Solved by:
GunsNRoses
Solved:
Last query:
Last reply:
Revision history for this message
GunsNRoses (ablagoev) said :
#1

Finally found the problem. Be sure to check the server time on both client/server. If the client/server are out of sync results can be distorted. An easy way to check both servers is to run this in bash and check the corresponding values:

date +%s

In my case the client was 2 hours behind the server and the rest becomes pretty logical. Hard to debug though. :)