Accuracies of metrics in druid

I am ingesting a topic in kafka through tranquility. I am only getting an accuracy of about 95% in the metrics even though there are extremely few failedCounts in the tranquility logs.
What might be the other sources which I could change to get a better accuracy?

Hi Rishi,

Can you provide more details on what type of query you are doing and what is type of aggregate you are using ?

Fwiw, topN, hyperUnique, approxHistogram are approximate.

If you want exact results use groupBy or timeSeries queries with Sum, max, min or count aggregators.

Another possible cause for mismatch in results can be due to events getting dropped during ingestion as there might be some events which are outside windowPeriod.

You can try increasing windowPeriod to avoid this.

Hey Rishi,

One thing to double check is whether you are counting your events properly, see “Counting the number of ingested events” here: http://druid.io/docs/latest/ingestion/schema-design.html

I am using pivot to query metrics. I have a count and several other metrics defined in my tranquility realtime ingestion spec. My window period is 30 min. I am still getting inaccuracies.

Hey Rishi,

If nothing is failing (no Druid tasks failing and sentCount == receivedCount in tranquility-kafka) then you should not see missing data. Could you confirm that there are no failed tasks and that un-sent messages don’t account for the discrepancy you see? If those look okay, then could you check whether a direct Druid query that does a longSum over your count metric (or plyql query that does a “SELECT SUM(count)”) gives you counts that make more sense?

It would be really strange if you see no task failures, and sentCount == receivedCount, but also a wrong sum of counts. Strange enough that I would try looking at the system you are getting the “official” count from and confirm that system is counting properly.

Hey Gian,

I checked and there were no indexing tasks failing in the interval I queried. Also there were no failedCounts in the interval I was querying. But there is still a large discrepancy between what I got in pivot and what I should get.

I tried direct Druid query that does a longSum over count metric and that gave the same result as pivot was giving me.

Hi Rishi, can you post the query you tried?

To check the count I tried
{

“queryType”: “timeseries”,

“dataSource”: “Prebid_Auction_Decision_23MAR”,

“granularity”: “all”,

“aggregations”: [

{ “type”: “longSum”, “name”: “count”, “fieldName”: “count” }

],

“intervals”: [ “2016-03-24T10:00:00.000/2016-03-24T14:00:00.000” ]

}

Hi Rishi, can you post your indexing spec as well?