Graphite emitter not working

Hi,

I am using druid 0.8.3 and graphite-emitter 0.9.0 extension. All the metrics emitted have slash in their name and this seems to be causing an issue in storing the metrics by graphite.

The metrics show up only while the corresponding wsp files get created for the first time. Data points for the same metrics coming later don’t show up. Has anyone faced the same issue or have been able to publish the metrics successfully to graphite in the production?

Thanks,

Varsha

Hi Varsha
Can you provide more insights like configs and what make you think that a slash is breaking the storing ?

I have used this with slash and it work for me.

B-Slim

Hi Slim,

Following are the configs I am using :

druid.emitter=graphite

druid.emitter.graphite.hostname=127.0.0.1

druid.emitter.graphite.port=2001

druid.emitter.graphite.eventConverter={“type”:“all”, “namespacePrefix”: “druid.test”, “ignoreHostname”:true, “ignoreServiceName”:true}

druid.monitoring.monitors=[“com.metamx.metrics.JvmMonitor”,“io.druid.client.cache.CacheMonitor”,“io.druid.server.metrics.HistoricalMetricsMonitor”]

This is what I have observed - I can see the datapoints in graphite for a particular metric only for when the whisper file gets created for a first time(The every first time this metric is published). I don’t see further datapoints though a tcp dump shows that the metrics are actually getting published from Druid broker. So I feel due to the ‘/’ present in the metric name, graphite file creation succeeds because ‘/’ get treated as directory paths but for further metrics it can’t resolve.

Hence for metrics like historical’s query/segment/time at the query id level I do see the metrics once but for other metrics like jvm/mem/used I see the datapoints just initially and nothing after that. Deleting this metric directory(graphite’s storage directory) will again show the new datapoints for that time interval maybe because it has to create the file again.

Is it working for you for metrics like jvm/mem/used emitted for example by the broker? Please let me know if I am missing something.

Thanks,

Varsha

I was able to emit druid metrics to graphite without any issues. One difference I see from ur config is that,
I used a pickled port while sending metrics to graphite. Think the default pickled port for graphite is 2004.

Hi,

I tried verifying again by sending metrics manually instead of testing with the druid emitter. The pickled port in my case is now 2120.

Below is the output for metric name with dots :

varsha@work:~$ export GRAPHITE_HOST=127.0.0.1

varsha@work:~$ export GRAPHITE_PORT=2120

varsha@work:~$ export GRAPHITE_WEB=http://127.0.0.1:9100

varsha@work:~$ export METRIC_PREFIX=druid.pickle.test.dot

varsha@work:~$ export METRIC_VALUE=999

varsha@work:~$ export METRIC_NAME=$METRIC_PREFIX.test.x.y.z.foo.bar

varsha@work:~$ export FROM_TS=date -v-360S +%s

varsha@work:~$ export DATE_TS=date -v-240S +%s

varsha@work:~$ ./picklesend.py $GRAPHITE_HOST $GRAPHITE_PORT $METRIC_NAME $METRIC_VALUE $DATE_TS

[(‘druid.pickle.test.dot.test.x.y.z.foo.bar’, (1462371800, 999.0))]

varsha@work:~$ sleep 15

varsha@work:~$ curl $GRAPHITE_WEB’/render/?target=’$METRIC_PREFIX’.test.x.y.z.foo.bar&format=json&from=’$FROM_TS

[{“target”: “druid.pickle.test.dot.test.x.y.z.foo.bar”, “datapoints”: [[null, 1462371720], [999.0, 1462371780], [null, 1462371840], [null, 1462371900], [null, 1462371960], [null, 1462372020]]}]

varsha@work:~$ DATE_TS=date -v-180S +%s

varsha@work:~$ ./picklesend.py $GRAPHITE_HOST $GRAPHITE_PORT $METRIC_NAME $METRIC_VALUE $DATE_TS

[(‘druid.pickle.test.dot.test.x.y.z.foo.bar’, (1462371897, 999.0))]

varsha@work:~$ sleep 15

varsha@work:~$ curl $GRAPHITE_WEB’/render/?target=’$METRIC_PREFIX’.test.x.y.z.foo.bar&format=json&from=’$FROM_TS

[{“target”: “druid.pickle.test.dot.test.x.y.z.foo.bar”, “datapoints”: [[null, 1462371720], [999.0, 1462371780], [999.0, 1462371840], [null, 1462371900], [null, 1462371960], [null, 1462372020], [null, 1462372080]]}]

varsha@work:~$ DATE_TS=date -v-120S +%s

varsha@work:~$ ./picklesend.py $GRAPHITE_HOST $GRAPHITE_PORT $METRIC_NAME $METRIC_VALUE $DATE_TS

[(‘druid.pickle.test.dot.test.x.y.z.foo.bar’, (1462371994, 999.0))]

varsha@work:~$ sleep 15

varsha@work:~$ curl $GRAPHITE_WEB’/render/?target=’$METRIC_PREFIX’.test.x.y.z.foo.bar&format=json&from=’$FROM_TS

[{“target”: “druid.pickle.test.dot.test.x.y.z.foo.bar”, “datapoints”: [[null, 1462371720], [999.0, 1462371780], [999.0, 1462371840], [null, 1462371900], [999.0, 1462371960], [null, 1462372020], [null, 1462372080], [null, 1462372140]]}]

Below is the output for metric name with slashes :

varsha@work:~$ export GRAPHITE_HOST=127.0.0.1

varsha@work:~$ export GRAPHITE_PORT=2120

varsha@work:~$ export GRAPHITE_WEB=http://127.0.0.1:9100

varsha@work:~$ export METRIC_PREFIX=druid.pickle.test.slash

varsha@work:~$ export METRIC_VALUE=999

varsha@work:~$ export METRIC_NAME=$METRIC_PREFIX.test.x.y.z/foo/bar

varsha@work:~$ export FROM_TS=date -v-360S +%s

varsha@work:~$ export DATE_TS=date -v-240S +%s

varsha@work:~$ ./picklesend.py $GRAPHITE_HOST $GRAPHITE_PORT $METRIC_NAME $METRIC_VALUE $DATE_TS

[(‘druid.pickle.test.slash.test.x.y.z/foo/bar’, (1462373891, 999.0))]

varsha@work:~$ sleep 15

varsha@work:~$ curl $GRAPHITE_WEB’/render/?target=’$METRIC_PREFIX’.test.x.y.z.foo.bar&format=json&from=’$FROM_TS

[{“target”: “druid.pickle.test.slash.test.x.y.z.foo.bar”, “datapoints”: [[null, 1462373820], [999.0, 1462373880], [null, 1462373940], [null, 1462374000], [null, 1462374060], [null, 1462374120]]}]

varsha@work:~$ DATE_TS=date -v-180S +%s

varsha@work:~$ ./picklesend.py $GRAPHITE_HOST $GRAPHITE_PORT $METRIC_NAME $METRIC_VALUE $DATE_TS

[(‘druid.pickle.test.slash.test.x.y.z/foo/bar’, (1462373970, 999.0))]

varsha@work:~$ sleep 15

varsha@work:~$ curl $GRAPHITE_WEB’/render/?target=’$METRIC_PREFIX’.test.x.y.z.foo.bar&format=json&from=’$FROM_TS

[{“target”: “druid.pickle.test.slash.test.x.y.z.foo.bar”, “datapoints”: [[null, 1462373820], [999.0, 1462373880], [null, 1462373940], [null, 1462374000], [null, 1462374060], [null, 1462374120]]}]

varsha@work:~$ DATE_TS=date -v-120S +%s

varsha@work:~$ ./picklesend.py $GRAPHITE_HOST $GRAPHITE_PORT $METRIC_NAME $METRIC_VALUE $DATE_TS

[(‘druid.pickle.test.slash.test.x.y.z/foo/bar’, (1462374047, 999.0))]

varsha@work:~$ sleep 15

varsha@work:~$ curl $GRAPHITE_WEB’/render/?target=’$METRIC_PREFIX’.test.x.y.z.foo.bar&format=json&from=’$FROM_TS

[{“target”: “druid.pickle.test.slash.test.x.y.z.foo.bar”, “datapoints”: [[null, 1462373820], [999.0, 1462373880], [null, 1462373940], [null, 1462374000], [null, 1462374060], [null, 1462374120], [null, 1462374180]]}]

Got 3 datapoints for metric name with a dot but found only the first datapoint for the metric name with slash.

Below are the commands I ran to verify this and I have attached the picklesend.py file. Any suggestions on what I might be missing for observing this behavior?

export GRAPHITE_HOST=127.0.0.1

export GRAPHITE_PORT=2120

export GRAPHITE_WEB=http://127.0.0.1:9100

export METRIC_PREFIX=druid.pickle.test.slash

export METRIC_VALUE=999

export METRIC_NAME=$METRIC_PREFIX.test.x.y.z/foo/bar

export FROM_TS=date -v-360S +%s

export DATE_TS=date -v-240S +%s

./picklesend.py $GRAPHITE_HOST $GRAPHITE_PORT $METRIC_NAME $METRIC_VALUE $DATE_TS

sleep 15

curl $GRAPHITE_WEB’/render/?target=’$METRIC_PREFIX’.test.x.y.z.foo.bar&format=json&from=’$FROM_TS

DATE_TS=date -v-180S +%s

./picklesend.py $GRAPHITE_HOST $GRAPHITE_PORT $METRIC_NAME $METRIC_VALUE $DATE_TS

sleep 15

curl $GRAPHITE_WEB’/render/?target=’$METRIC_PREFIX’.test.x.y.z.foo.bar&format=json&from=’$FROM_TS

DATE_TS=date -v-120S +%s

./picklesend.py $GRAPHITE_HOST $GRAPHITE_PORT $METRIC_NAME $METRIC_VALUE $DATE_TS

sleep 15

curl $GRAPHITE_WEB’/render/?target=’$METRIC_PREFIX’.test.x.y.z.foo.bar&format=json&from=’$FROM_TS

Thanks,

Varsha

picklesend.py (965 Bytes)

are you sure the carbon server configured to retain data ?

It’s being working fine since a week after replacing the slashes with dots.

Did you have to add any extra configs apart from the default graphite metrics to specifically allow slashes?

Hi Varsha,
i am sorry still can figure out what you are trying to say (i have tested this with slashes and it is working).

is the replacement happened within Carbon server or somewhere else ?

Hi Rajini -

Installing Graphite is not a straight forward way - especially, when you are behind a corporate firewall and had to deal with dependency hell with python packages. Can you please shed some light on what source did you use to install Graphite. I mean steps…

Hi
i have used Pip http://graphite.readthedocs.io/en/latest/install-pip.html

you can find a variety of other methods online like installing out of the source code… http://www.infoq.com/articles/graphite-intro

Good luck

What about Apache configurations? What changes should you make to make it work. I was able to bring up a developer instance of Graphite but I was not able to hook it up with Apache.