Data are fluctuating by historical node

Hi Guys ,

I am facing some fluctuating data while taking from historical node.

Some time its showing correct and then after some time it showing less data with same query and interval.

2015-03-25 07:08:12,592 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2015-03-25T07:08:12.592Z”,“service”:“xxx”,“host”:“xxx:80”,“metric”:“cache/delta/misses”,“value”:6}]

2015-03-25 07:08:12,592 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2015-03-25T07:08:12.592Z”,“service”:“xxx”,“host”:“xxx:80”,“metric”:“cache/total/misses”,“value”:203053}]

above lines are showing something about it ? What is mean of this?

Thanks Jitesh.

Those logs are simply related to the cached data being available or not.

Can you share more about the data that is not being consistent per query? What was the ingestion spec, do you have any load rules, and are there any entries in the Historical node logs that are WARN or ERROR ?

I realized that may not have been a clear explanation. The results of a query can be cached so when a second query comes in that asks for the same thing, it can pull from the cached results instead of re-calculating them.

Thanks charles.

First query is also showing wrong result.

After I replaced historical node It’s showing correct and not fluctuating.

All memory config was right but don’t know why it was not showing correct result before node replaced.

Thanks

Jitesh

Hi Jitesh, if you are interested in tracing the root issue, it may be interesting to look at the logs of the historical and see if you see any exceptions, and also in broker logs as well.

Hi guys,

I replaced the node and its working good.But we know this is not solution of this problem. As fangjin said I am tracing root issue of this.

If you guys have suggestions on this Please let me know.

Thanks.

Jitesh

Hi Jitesh, there are several things that may cause this problem.

I would try to isolate the time range which causes issues and try to understand which segments are problematic. From there, it’d be interesting to go through the broker logs to see what the broker thinks about what is happening with those segments.