I use druid extension Lookups-cache-global for loading lookups.Lately we have been having issues with loading lookups in druid.
was expected as we load around 180 lookups of which 5-6 of them have around 10 million entries and others around million entries.
This has led to a lot of historicals going down(high gc pauses,not able to query).
How can I improve the performance of loading lookups.
Hi Sharath: loading 5~6 10-million row and more million-row lookup tabes is not what lookup designed up to. Is it possible to pre-process the data before ingesting them to Druid?
Hope this helps.
I don’t think I would be able to do that.But that problem started after I upgraded druid from 0.9 to 0.12.3.
Before upgrading druid,historicals used to recover after a brief slow down when loading lookups.
I think this has to with change in the way lookups were loaded.there seems to be a lazy loading of lookups.
Can you suggest anything for decreasing the GC pauses?
I have thought about the following optimisations
(1)Increasing the heap size
(2)Experimenting with different garbage collectors
We ran into the same problem, after using (or better tried to use) two lookups with about 1,6 million entries.
We added a few properties to jvm settings, to geht GC Log and HeapDumps for postmortem analyzing.
I think increasing Heap could solve the problem. The question is: how long?
I added a picture from a heapdump created after historical node died.
1 and 2: 380MB and 353MB ConcurrentHashMap which holds data from two lookups (2x 1.6 million entries)
3: Thread with groupBy query which uses a lookup (LookupExtractionFn) and needs about 354MB
After looking at point 3 in more detail, I found out that the thread holds a complete copy of hashmap from point 3 (!!)
…for me that means, if you have e.g. five threads handling queries with lookups, you have 5 copies of hashmap in heap. ==> you need a lot of heap
==> Lookups are not made for that, what we both want to do with it
thanks for your reply.Will have a look at the information your provided.
For what it’s worth, we have been prototyping an off heap lookup implementation that should scale to way higher sizes than is practical in the JVM heap. Since it is still in the prototype stage there isn’t much to share yet but we intend to discuss it more in the future. The idea is sort of similar to https://github.com/yahoo/maha/tree/master/druid-lookups. You might also enjoy checking that one out.
well thats right, and we tried to use offHeap cache. It worked well with a few hundred thousend entries, but caused errors with >1 million entries as you can see here