Data points arriving late

Asked by Borom Ir

Hi!

I'm trying to figure out Graphite's behaviour when the data points arrive "late".

My storage schema is very simple:
[default]
pattern = .*
retentions = 60s:7d, 5m:90d

And I'm sending data to RabbitMQ every minute (so this matches my storage schema).
Things work pretty well at the moment, but I'm trying to figure out 1 edge case that is driving me crazy (probably just missing some configuration).

From time to time, the network fails and Graphite can't connect to RabbitMQ (or I can just force the situation and stop Carbon for a while). This should be ok, as my metrics keep collecting in a queue which Graphite can process when connectivity is restored.

My problem is that once the connectivity is restored (or I start the Carbon agent again) and Graphite starts processing all these "buffered" messages, it doesn't store these metrics in whisper files, it seems like it just ignores these old timestamps and once it catches up with the current time, it will start inserting values in whisper files again, leaving huge gaps (I'm talking about hours here) in my whisper files (using whisper-fetch I can see all have the value "None").

I have checked the data in the queue and it is correct and has the right timestamp (well, actually my metrics are sent every 60s but not exactlly at HH::MM::00, some metrics will arrive at HH::MM::13, other will arrive at HH::MM::43, but I guess this is ok for an 60s interval), so:

1) Does Graphite require a continuous flow of data to work? I mean...what happens if It receives data points late? My understanding was that it didn't matter as long as my data point arrived with a Timestamp.
2) How does it behave when it receives data points out of order? That is...what happens if I receive first timestamp T3, then T1 and then T2? My understanding is that this was irrelevant

Does anyone have a clue or could point me out in the right direction so I can keep debugging this issue? I have run out of ideas...

Thanks!

Question information

Language:
English Edit question
Status:
Solved
For:
Graphite Edit question
Assignee:
No assignee Edit question
Solved by:
Sidnei da Silva
Solved:
Last query:
Last reply:
Revision history for this message
Best Sidnei da Silva (sidnei) said :
#1

Yes, if there's a timestamp in the message to be written out to the whisper file, it does not matter if it's delayed.

I suspect what's happening is:

- Lots of metrics get queued in rabbitmq
- You've set a maximum number of messages to be queue on carbon
- When carbon reconnects to rabbitmq, it picks up all queued messages
- The carbon queue overflows
- Metrics that don't fit in carbon's queue get dropped on the floor

Revision history for this message
Borom Ir (boromlaunchpad) said :
#2

Thanks Sidnei da Silva, that solved my question.