We encountered the following issue on the indexing service of Druid-0.8.1.
Our indexing tasks produced a heap dump now and then. When we analyzed this further, we found that few TOPN and groupBy queries had set the limit to 1Million. By analyzing the heap dump, It seems Druid Indexing service pre-creates objects based on the limit value in the query which caused memory pressure/gc cleanup and finally ended up with data lose.
In our case based on the user’s filter conditions, we expect only 10-20 rows as an output but since user is not familiar with the significance of each key, he set limit as 1M.
We know that if user doesn’t set any limit then Druid applies a default value 1000 as limit that’s good strategy but even if user sets the limit, can Druid check this value and cap it to “X” if it’s more than that?
something like introducing a config like “druid.query.search.maxSearchLimit” in Indexing service and taking min(druid.query.search.maxSearchLimit, limit value set in the query)
Do we have some solutions available in later versions? if not, is this a valid concern to be considered?
Can anyone please share your view for the above stated problem?
The query engines don’t pre-allocate space up to the max limit, but it’s possible you had some queries that really did get 1M rows back for some reason. There isn’t a config to cap the limits, but even if there was, it doesn’t really completely solve the problem since the intermediate result set might need to be larger than your final limit. (such as if you have an inner query, or if you’re sorting on a metric)
To solve this problem, in Druid 0.9.2 we added a new groupBy engine (v2) that allows you to cap memory usage, and optionally spill to disk for queries that exceed the memory limit. If you have stability problems due to queries running OOME then I’d suggest giving that a try.
Thanks Gian for the reply. Sure let us give a try with new version.
for the issue with an existing version, user did a topN query with the threshold as 1M for the dimension which has only 2Lakhs as cardinality. This query had been fired for each and every minute and after serving 3-4 queries, indexing tasks crashed with leaving heapdump. While analyzing the heap dump we got a trace that it created 1M object. Here is the excerpt of the heap dump and any pointer on this will help us to plan/corrrect the setting properly.
Instance Counts for All Classes (including platform)
1013028 instances of class [Ljava.lang.String;
364058 instances of class io.druid.segment.incremental.
362576 instances of class io.druid.query.aggregation.DoubleSumAggregator
Is this on a realtime task? What was the OOME message exactly: was it “java heap space” or something else?
Yes, it was from one of the tasks of Indexing service.
Yes, it was “java heap space”
Here is the exact error from one of my indexing service tasks
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid133152.hprof …
Heap dump file created [38224102525 bytes in 437.232 secs]
Please advise on this.
Any pointer on this would help us to decide the things further. Can you please advise us on this?
Since this is an indexing service task and you were doing topN queries, the objects you saw (io.druid.segment.incremental.IncrementalIndex$1, DoubleSumAggregator) were probably indexing related and not query related. They could have been query related if you were doing groupBy v1 queries (it uses IncrementalIndex) but not for topNs.
The first thing I’d do is make sure that memory use is well configured. The big draws on your memory are going to be:
Realtime data being loaded: maxRowsInMemory * (2 + maxPendingPersists)
Any memory needed for query result merging, which will mostly scale with the max query result set size for topN and groupBy v1
So, some stuff you can try:
Lower maxRowsInMemory and/or maxPendingPersists
Upgrade Druid; you’re on an older version. Memory use improves with almost every release. One big improvement in the latest (0.9.2) is the groupBy v2 engine, which can be optionally enabled and substantially reduces heap pressure from groupBys.
Avoid issuing queries with thresholds/limits higher than you need.
Thanks for your continuous support.
We got queries with higher threshold for both TopN and groupBy v1 which exhausted the indexing tasks memory. By stopping the topN queries alone, we didn’t get any benefit and the problem continued. Finally, when we reset the threshold to 1k from 1M, for groupBy queries, we completely stopped the issue.
Can you please share more insights on how groupBy query works on this context and how we can handle the situation where user sets limit/threshold as higher values say 1M.
I am curious to get to know your view on this.
Sure thing. See “Implementation details” at: http://druid.io/docs/latest/querying/groupbyquery.html#implementation-details
Pre 0.9.2 there, was only one engine, and it’s the same as the one now called “v1”. In particular it does all of its result merging on heap using an IncrementalIndex, so the large amount of IncrementalIndex objects you saw could definitely have been groupBy related. You can mitigate the situation by setting druid.query.groupBy.maxResults to some lower value than the default. If you set it low enough then you should stop seeing OOMEs.
Or you can upgrade to 0.9.2 and switch to the “v2” engine, which uses off heap memory and generally has better control over its memory usage.