I’ve integrated the graphite-emitter metrics extension. Everything worked fine up to a point where metrics (on one specific server) were not being transmitted anymore. Both middle-manager and the historical reside on this specific server. The logs completely stopped spilling to the console as well. Are there any hints as to what could be the cause of this? Memory issues? The historical was still functioning properly and answering queries, but no metrics where being emitted.
More so, I’ve configured both the monitoring “emissionPeriod” and the graphite-emitter “flushPeriod” to 10s each but still metrics get dumped every 10s up to 5mins without consistency (and even then, not all metrics are being transmitted as I can see according to the data points in Grafana).
I’d appreciate any help
this seems to be an issue with the main process it self, if the logs are not flowing and neither metrics that’s means that some background process died or blocked on something.
Can you check logs for exception or/and a thread dump to check what is happening.
I was under the impression that the metrics are constantly transmitted per emission period (especially JVM statistics). Seems they are not (only being transmitted when there are actual queries taking place). This causes some kind of confusion as it’s quite unexpected.
Even though, at the time of metrics not being transmitted I ran a few queries and didn’t see any metrics being displayed in Grafana (had to restart the service). I currently cannot seem to reproduce this at the moment and will keep an eye on it.
I looked at the logs and didn’t find anything at the time, sadly. Will resurrect this thread if I find anything.