Upgrade from 0.9.8 to 0.9.9
I am having a number of problems trying to upgrade from 0.9.8 to 0.9.9.
I am using diamond to send my metrics in to graphite and that hasn't changed. They were working prior to the upgrade.
I have it configured like so...
1. Each diamond server sends to a central relay in each datacenter.
2. The relay sends the metrics along to a cache server in the main datacenter.
When I start the relay I can see diamond connecting ( pickle receiver port 2014 ).
Starting carbon-relay (instance a)
11/02/2012 00:39:50 :: [console] Log opened.
11/02/2012 00:39:50 :: [console] twistd 11.0.0 (/usr/local/
11/02/2012 00:39:50 :: [console] reactor class: twisted.
11/02/2012 00:39:50 :: [console] twisted.
11/02/2012 00:39:50 :: [console] Starting factory <twisted.
11/02/2012 00:39:50 :: [console] twisted.
11/02/2012 00:39:50 :: [console] Starting factory <twisted.
11/02/2012 00:39:50 :: [console] Starting factory CarbonClientFac
11/02/2012 00:39:50 :: [clients] CarbonClientFac
11/02/2012 00:39:50 :: [clients] CarbonClientPro
11/02/2012 00:39:53 :: [listener] MetricPickleRec
11/02/2012 00:40:06 :: [listener] MetricPickleRec
11/02/2012 00:40:09 :: [listener] MetricPickleRec
11/02/2012 00:40:10 :: [listener] MetricPickleRec
11/02/2012 00:40:35 :: [listener] MetricPickleRec
11/02/2012 00:40:36 :: [listener] MetricPickleRec
11/02/2012 00:40:50 :: [console] Unhandled error in Deferred:
11/02/2012 00:40:50 :: [console] Unhandled Error
Traceback (most recent call last):
File "/usr/local/
self.mainLoop()
File "/usr/local/
self.
File "/usr/local/
call.
File "/usr/local/
d = defer.maybeDefe
--- <exception caught here> ---
File "/usr/local/
result = f(*args, **kw)
File "/usr/local/
record(
exceptions.
Also I keep getting the above error which looks like it is coming from the section of code where it is reporting internal cache metrics, although it looks like it only happens once.
So it appears that diamond and the relay are talking ok to each other and it looks like the relay and the cache are talking to each other.
But I never see any updates on the cache server.
Incidentally the query below should be "Infrastructur.
==> /var/log/
11/02/2012 00:41:50 :: [127.0.0.1:57739] cache query for "astructure.
11/02/2012 00:41:50 :: [127.0.0.1:57739] cache query for "astructure.
Also the console log on the cache server only ever updates the 13 internal carbon metrics.
==> /var/log/
11/02/2012 00:50:02 :: Sorted 13 cache queues in 0.000044 seconds
If I remove the carbon directory where the whisper files live it gets recreated and I see the tree for all of my old metrics in the webui but the whisper files don't ever get updated and consequently neither does the graph.
I'm not sure where it went wrong but it is definitely off the rails.
One other thing I've been rolling my own rpm's and completely removed all old 0.9.8 code from the source and only put the 0.9.9 source in the new rpm's. I completely removed all rpms for graphite-web, carbon, and whisper and did installs with the new rpms. Any idea where I should start looking?
Thanks!
Cody
Question information
- Language:
- English Edit question
- Status:
- Answered
- For:
- Graphite Edit question
- Assignee:
- No assignee Edit question
- Last query:
- Last reply:
Can you help with this problem?
Provide an answer of your own, or ask Cody Stevens for more information if necessary.