Logstash config to send to Graphite:
input {
...
...
}
output {
stdout { debug => true debug_format => "json"}
if [type] == "TEST-LOG" {
graphite {
host => "10.50.150.111"
port => 2003
debug => true
metrics => [ "Test.%{service_name}.%{host_name}.metric1", "%{metric1}", "Test.%{service_name}.%{host_name}.metric2", "%{metric2}", "Test.%{service_name}.%{host_name}.metric3", "%{metric3}", "Test.%{service_name}.%{host_name}.metric4", "%{metric4}" ]
}
}
}
Result output from stdout -- Logstash filter look fine as expected.
{"message":"2013-11-14 00:00:00,host-01,serviceA,132,2256.2,2.1,205.439394\r","@timestamp":"2013-11-14T00:00:00.000Z","@version":"1","type":"Test-LOG","host":"host-x","path":"/shipper/raw/Test.csv","logdate":"2013-11-14 00:00:00","host_name":"host-01","service_name":"serviceA","metric1":"132","metric2":"2256.2","metric3":"2.1","metric4":"205.439394\r"}
{"message":"2013-11-14 00:00:00,host-02,serviceA,164,8123,0,620.006098\r","@timestamp":"2013-11-14T00:00:00.000Z","@version":"1","type":"Test-LOG","host":"host-x","path":"/shipper/raw/Test.csv","logdate":"2013-11-14 00:00:00","host_name":"host-01","service_name":"serviceA","metric1":"164","metric2":"8123","metric3":"0","metric4":"620.006098\r"}
{"message":"2013-11-14 00:00:00,host-03,serviceA,4622,0,27.412987\r","@timestamp":"2013-11-14T00:00:00.000Z","@version":"1","type":"Test-LOG","host":"host-x","path":"/shipper/raw/Test.csv","logdate":"2013-11-14 00:00:00","host_name":"host-01","service_name":"serviceA","metric1":"2310","metric2":"4622","metric3":"0","metric4":"27.412987\r"}
{"message":"2013-11-14 00:00:00,host-04,serviceA,296,2914,0,93.162162\r","@timestamp":"2013-11-14T00:00:00.000Z","@version":"1","type":"Test-LOG","host":"host-x","path":"/shipper/raw/Test.csv","logdate":"2013-11-14 00:00:00","host_name":"host-01","service_name":"serviceA","metric1":"296","metric2":"2914","metric3":"0","metric4":"93.162162\r"}
Result metric show in Whisper in Graphite -- A lot of data missing as random matric missing
Test.serviceA.host-01.metric1
Test.serviceA.host-01.metric2
Test.serviceA.host-02.metric1
Test.serviceA.host-02.metric2
Test.serviceA.host-02.metric3
**Metric data in Whisper random missing if re-run Logstash with the same output from stdout
Expected result
Test.serviceA.host-01.metric1
Test.serviceA.host-01.metric2
Test.serviceA.host-01.metric3
Test.serviceA.host-01.metric4
Test.serviceA.host-02.metric1
Test.serviceA.host-02.metric2
Test.serviceA.host-02.metric3
Test.serviceA.host-02.metric4
Test.serviceA.host-03.metric1
Test.serviceA.host-03.metric2
Test.serviceA.host-03.metric3
Test.serviceA.host-03.metric4
Test.serviceA.host-04.metric1
Test.serviceA.host-04.metric2
Test.serviceA.host-04.metric3
Test.serviceA.host-04.metric4
How to check where metric data lost is?
How to fix these?
Could anyone help to advise?
Thanks.
