Middle manager OutOfMemoryError when publishing segments

Druid version: 0.9.1.1

Data insert type: Kafka Indexing Service

Middle manager runtime.properties:

druid.service=druid/middleManager
druid.port=8091

Number of tasks per middleManager

druid.worker.capacity=6

Task launch parameters

druid.indexer.runner.javaOpts=-server -Xmx4g -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
druid.indexer.task.baseTaskDir=/opt/druid/task

HTTP server threads

druid.server.http.numThreads=25

Processing threads and buffers

druid.processing.buffer.sizeBytes=268435456
druid.processing.numThreads=2

Hadoop indexing

druid.indexer.task.hadoopWorkingPath=/opt/hadoop
druid.indexer.task.defaultHadoopCoordinates=[“org.apache.hadoop:hadoop-client:2.3.0”]

``

Middle manager jvm.params:

JVM_PARAMS=-server -Xms64m -Xmx64m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/opt/druid/tmp/ -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager

``

Kafka indexing task tuning config:

“tuningConfig”: {
“type”: “kafka”,
“maxRowsInMemory”: 10000,
“maxRowsPerSegment”: 500000,
“intermediatePersistPeriod”: “PT10M”,
“basePersistDirectory”: “/opt/druid/tmp/1498645844254-0”,
“maxPendingPersists”: 0,
“indexSpec”: {
“bitmap”: {
“type”: “concise”
},
“dimensionCompression”: null,
“metricCompression”: null
},
“buildV9Directly”: false,
“reportParseExceptions”: false,
“handoffConditionTimeout”: 0
}

``

Task duration: first it was set to 1 hour, but later we tried with 10 minutes -> same error

The error that occurs:

2017-06-28T10:43:22,723 INFO [appenderator_merge_0] io.druid.segment.CompressedPools - Allocating new littleEndByteBuf[58,134]
2017-06-28T10:43:22,723 INFO [appenderator_merge_0] io.druid.segment.CompressedPools - Allocating new littleEndByteBuf[58,135]
2017-06-28T10:43:23,621 WARN [task-runner-0-priority-0] io.druid.segment.realtime.appenderator.FiniteAppenderatorDriver - Failed publishAll (try 1), retrying in 2,459ms.
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Direct buffer memory
at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) ~[guava-16.0.1.jar:?]
at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) ~[guava-16.0.1.jar:?]
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-16.0.1.jar:?]
at io.druid.segment.realtime.appenderator.FiniteAppenderatorDriver.publishAll(FiniteAppenderatorDriver.java:417) [druid-server-0.9.1.1.jar:0.9.1.1]
at io.druid.segment.realtime.appenderator.FiniteAppenderatorDriver.finish(FiniteAppenderatorDriver.java:256) [druid-server-0.9.1.1.jar:0.9.1.1]
at io.druid.indexing.kafka.KafkaIndexTask.run(KafkaIndexTask.java:504) [druid-kafka-indexing-service-0.9.1.1.jar:0.9.1.1]
at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) [druid-indexing-service-0.9.1.1.jar:0.9.1.1]
at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) [druid-indexing-service-0.9.1.1.jar:0.9.1.1]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:693) ~[?:1.8.0_91]
at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123) ~[?:1.8.0_91]
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) ~[?:1.8.0_91]
at io.druid.segment.CompressedPools$4.get(CompressedPools.java:100) ~[druid-processing-0.9.1.1.jar:0.9.1.1]
at io.druid.segment.CompressedPools$4.get(CompressedPools.java:93) ~[druid-processing-0.9.1.1.jar:0.9.1.1]
at io.druid.collections.StupidPool.take(StupidPool.java:64) ~[druid-common-0.9.1.1.jar:0.9.1.1]
at io.druid.segment.CompressedPools.getByteBuf(CompressedPools.java:108) ~[druid-processing-0.9.1.1.jar:0.9.1.1]
at io.druid.segment.data.CompressedObjectStrategy.fromByteBuffer(CompressedObjectStrategy.java:286) ~[druid-processing-0.9.1.1.jar:0.9.1.1]
at io.druid.segment.data.CompressedObjectStrategy.fromByteBuffer(CompressedObjectStrategy.java:42) ~[druid-processing-0.9.1.1.jar:0.9.1.1]
at io.druid.segment.data.GenericIndexed$BufferIndexed._get(GenericIndexed.java:225) ~[druid-processing-0.9.1.1.jar:0.9.1.1]
at io.druid.segment.data.GenericIndexed$1.get(GenericIndexed.java:300) ~[druid-processing-0.9.1.1.jar:0.9.1.1]
at io.druid.segment.data.CompressedFloatsIndexedSupplier$CompressedIndexedFloats.loadBuffer(CompressedFloatsIndexedSupplier.java:295) ~[druid-processing-0.9.1.1.jar:0.9.1.1]
at io.druid.segment.data.CompressedFloatsIndexedSupplier$1.get(CompressedFloatsIndexedSupplier.java:86) ~[druid-processing-0.9.1.1.jar:0.9.1.1]
at io.druid.segment.column.IndexedFloatsGenericColumn.getFloatSingleValueRow(IndexedFloatsGenericColumn.java:73) ~[druid-processing-0.9.1.1.jar:0.9.1.1]
at io.druid.segment.QueryableIndexIndexableAdapter$2$1.next(QueryableIndexIndexableAdapter.java:271) ~[druid-processing-0.9.1.1.jar:0.9.1.1]
at io.druid.segment.QueryableIndexIndexableAdapter$2$1.next(QueryableIndexIndexableAdapter.java:177) ~[druid-processing-0.9.1.1.jar:0.9.1.1]
at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48) ~[guava-16.0.1.jar:?]
at com.google.common.collect.Iterators$PeekingImpl.peek(Iterators.java:1162) ~[guava-16.0.1.jar:?]
at com.metamx.common.guava.MergeIterator$1.compare(MergeIterator.java:46) ~[java-util-0.27.9.jar:?]
at com.metamx.common.guava.MergeIterator$1.compare(MergeIterator.java:42) ~[java-util-0.27.9.jar:?]
at java.util.PriorityQueue.siftUpUsingComparator(PriorityQueue.java:669) ~[?:1.8.0_91]
at java.util.PriorityQueue.siftUp(PriorityQueue.java:645) ~[?:1.8.0_91]
at java.util.PriorityQueue.offer(PriorityQueue.java:344) ~[?:1.8.0_91]
at java.util.PriorityQueue.add(PriorityQueue.java:321) ~[?:1.8.0_91]
at com.metamx.common.guava.MergeIterator.(MergeIterator.java:55) ~[java-util-0.27.9.jar:?]
at com.metamx.common.guava.MergeIterable.iterator(MergeIterable.java:49) ~[java-util-0.27.9.jar:?]
at io.druid.collections.CombiningIterable.iterator(CombiningIterable.java:95) ~[druid-common-0.9.1.1.jar:0.9.1.1]
at io.druid.segment.IndexMerger.makeIndexFiles(IndexMerger.java:817) ~[druid-processing-0.9.1.1.jar:0.9.1.1]
at io.druid.segment.IndexMerger.merge(IndexMerger.java:423) ~[druid-processing-0.9.1.1.jar:0.9.1.1]
at io.druid.segment.IndexMerger.mergeQueryableIndex(IndexMerger.java:244) ~[druid-processing-0.9.1.1.jar:0.9.1.1]
at io.druid.segment.IndexMerger.mergeQueryableIndex(IndexMerger.java:217) ~[druid-processing-0.9.1.1.jar:0.9.1.1]
at io.druid.segment.realtime.appenderator.AppenderatorImpl.mergeAndPush(AppenderatorImpl.java:724) ~[druid-server-0.9.1.1.jar:0.9.1.1]
at io.druid.segment.realtime.appenderator.AppenderatorImpl.access$1100(AppenderatorImpl.java:107) ~[druid-server-0.9.1.1.jar:0.9.1.1]
at io.druid.segment.realtime.appenderator.AppenderatorImpl$7.apply(AppenderatorImpl.java:630) ~[druid-server-0.9.1.1.jar:0.9.1.1]
at io.druid.segment.realtime.appenderator.AppenderatorImpl$7.apply(AppenderatorImpl.java:618) ~[druid-server-0.9.1.1.jar:0.9.1.1]
at com.google.common.util.concurrent.Futures$1.apply(Futures.java:713) ~[guava-16.0.1.jar:?]
at com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:861) ~[guava-16.0.1.jar:?]
… 3 more

``

I listed the most important configs for the middle manager (maybe I forgot some?), and we have problems when the indexing tasks finishes and starts publishing the segments to deep storage.

We are generating a lot of data that have about 800 metrics and 200 dimensions. So we assume the problem lies there, in the huge amount of metrics and dimensions.

But we do not know what configs we could/should tweak to fix this problem.

So can you help us with some instructions what to try?

Thanks,

Drazen

Try setting -XX:MaxDirectMemorySize a bit higher.