Verifying data sent from relay to cache.

For reference here is our carbon.conf

[cache]
USER = carbon
MAX_CACHE_SIZE = 500000
MAX_UPDATES_PER_SECOND = 1000
MAX_CREATES_PER_MINUTE = 1000
LOG_UPDATES = false
LINE_RECEIVER_INTERFACE = 0.0.0.0
LINE_RECEIVER_PORT = 2013
PICKLE_RECEIVER_INTERFACE = 0.0.0.0
PICKLE_RECEIVER_PORT = 2014
CACHE_QUERY_INTERFACE = 127.0.0.1
CACHE_QUERY_PORT = 7002
USE_INSECURE_UNPICKLER = false

[relay]
USER = carbon
LINE_RECEIVER_INTERFACE = 0.0.0.0
LINE_RECEIVER_PORT = 2003
PICKLE_RECEIVER_INTERFACE = 0.0.0.0
PICKLE_RECEIVER_PORT = 2004
RELAY_METHOD = consistent-hashing
REPLICATION_FACTOR = 1
DESTINATIONS = 127.0.0.1:2014

[aggregator]
USER = carbon
DESTINATIONS = 127.0.0.1:2014
LINE_RECEIVER_PORT = 2023
PICKLE_RECEIVER_PORT = 2024
RULES = []

So, according to config above setup is :
1. Relay listening for metrics on port 2003 (line protocol) and 2004 (pickle protocol)
2. Relay sending metrics to 127.0.0.1:2014
3. Cache listening for metrics on port 2013 (line protocol) and 2014 (pickle protocol), so, it should accept metrics from the relay and write them to whisper files.
4. Aggregator listening metrics on port 2013 (line protocol) and 2014 (pickle protocol) and sending them to cache too (127.0.0.1:2014).

So, you need to check
1. does anything coming to port 2003?
2. does anything coming to port 2014?

After looking into this some more and learning the architecture as well:

I've tried sending data directly from Statsd to Cache on the receiver and pickle port(not 100% where it should go).

The whisper file is created, but then starts writing None after an initial metric or two.

I'm struggling to find out why connectivity is in place, but Carbon just stops writing to the db file. I'm following the retention listed in the git example for carbon:

[carbon]
pattern = ^carbon\.
retentions = 60:90d

[default_1min_for_1day]
pattern = .*
retentions = 60s:1d

And my XFILESFACTOR = .1

If anything I feel like writes of 0.00 should be coming through not Null/None, which leads me to believe something is wrong with Carbon/Whisper or it's getting hammered too hard to catch up?

Graphite is caching metrics in a cache, so, they can be flushed on disk much later - but still should be accessible in web-interface.

Once the processes are restarted the whisper file is written to instantly and over the course of two writes acts normally and of course it shows up on the graphite.

After those few writes to the whisper file, nothing gets written besides None and the graphs never display this information.

Did your graphite-web using carbonlink connection (127.0.0.1:7002)? Could you please show your local_settings.py config?

No for carbonlink and local_settings.py is as follows:

TIME_ZONE = 'America/Chicago'

DATABASES = {
    'default': {
        'NAME': 'graphite',
        'ENGINE': 'django.db.backends.mysql',
        'USER': 'xxxxxxx',
        'PASSWORD': 'xxxxxxxx',
        'HOST': 'localhost',
        'PORT': '3306'
    }
}

CLUSTER_SERVERS = ''

MEMCACHE_HOSTS = ''

Should be default, but could you please add `CARBONLINK_HOSTS=['127.0.0.1:7002']` and restart Apache?

Added and restarted, doesn't seem to be a change for the whisper files or graph either one.

An update to this:

I edited the StatsD conf to point to carbon-cache instead of relay and everything seems to be working fine. I'm not sure what made relay stop functioning or what is wrong with it, but at least I know which part of the flow is broken.

This question was expired because it remained in the 'Open' state without activity for the last 15 days.