Compacting tasks failed by Direct buffer memory

we have used druid service for a long time, for some datasources, there are too many small segments. we want to merge and compact these segments. we used it as below:

{

“type” : “compact”,

“dataSource” : “some_datasource”,

“interval” : “2018-08-21/2018-09-20”,

“tuningConfig” : {

  "type": "index",

  "numShards" : 1,

  "forceExtendableShardSpecs" : true

}

}

all tasks except for compacting tasks are all “SUCCESS” status, there is exception in the compact tasks logs, how should we do?

2018-10-20T12:45:53,175 ERROR [task-runner-0-priority-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running task[AbstractTask{id=‘compact_shopee_all_s0__microsite_test_2018-10-20T01:33:06.052Z’, groupId=‘compact_shopee_all_s0__microsite_test_2018-10-20T01:33:06.052Z’, taskResource=TaskResource{availabilityGroup=‘compact_shopee_all_s0__microsite_test_2018-10-20T01:33:06.052Z’, requiredCapacity=1}, dataSource=‘shopee_all_s0__microsite_test’, context={}}]

java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Direct buffer memory

at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.1.jar:?]

at io.druid.indexing.common.task.IndexTask.generateAndPublishSegments(IndexTask.java:738) ~[druid-indexing-service-0.12.2.jar:0.12.2]

at io.druid.indexing.common.task.IndexTask.run(IndexTask.java:264) ~[druid-indexing-service-0.12.2.jar:0.12.2]

at io.druid.indexing.common.task.CompactionTask.run(CompactionTask.java:213) ~[druid-indexing-service-0.12.2.jar:0.12.2]

at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444) [druid-indexing-service-0.12.2.jar:0.12.2]

at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416) [druid-indexing-service-0.12.2.jar:0.12.2]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_121]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]

at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]

Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Direct buffer memory

at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) ~[guava-16.0.1.jar:?]

at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) ~[guava-16.0.1.jar:?]

at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-16.0.1.jar:?]

at io.druid.segment.realtime.appenderator.BatchAppenderatorDriver.pushAndClear(BatchAppenderatorDriver.java:141) ~[druid-server-0.12.2.jar:0.12.2]

at io.druid.segment.realtime.appenderator.BatchAppenderatorDriver.pushAllAndClear(BatchAppenderatorDriver.java:124) ~[druid-server-0.12.2.jar:0.12.2]

at io.druid.indexing.common.task.IndexTask.generateAndPublishSegments(IndexTask.java:714) ~[druid-indexing-service-0.12.2.jar:0.12.2]

… 8 more

Caused by: java.lang.OutOfMemoryError: Direct buffer memory

at java.nio.Bits.reserveMemory(Bits.java:693) ~[?:1.8.0_121]

at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123) ~[?:1.8.0_121]

at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) ~[?:1.8.0_121]

at io.druid.segment.CompressedPools$4.get(CompressedPools.java:105) ~[druid-processing-0.12.2.jar:0.12.2]

at io.druid.segment.CompressedPools$4.get(CompressedPools.java:98) ~[druid-processing-0.12.2.jar:0.12.2]

at io.druid.collections.StupidPool.makeObjectWithHandler(StupidPool.java:112) ~[druid-common-0.12.2.jar:0.12.2]

at io.druid.collections.StupidPool.take(StupidPool.java:103) ~[druid-common-0.12.2.jar:0.12.2]

at io.druid.segment.CompressedPools.getByteBuf(CompressedPools.java:113) ~[druid-processing-0.12.2.jar:0.12.2]

at io.druid.segment.data.DecompressingByteBufferObjectStrategy.fromByteBuffer(DecompressingByteBufferObjectStrategy.java:49) ~[druid-processing-0.12.2.jar:0.12.2]

at io.druid.segment.data.DecompressingByteBufferObjectStrategy.fromByteBuffer(DecompressingByteBufferObjectStrategy.java:28) ~[druid-processing-0.12.2.jar:0.12.2]

at io.druid.segment.data.GenericIndexed$BufferIndexed.bufferedIndexedGet(GenericIndexed.java:427) ~[druid-processing-0.12.2.jar:0.12.2]

at io.druid.segment.data.GenericIndexed$2.get(GenericIndexed.java:573) ~[druid-processing-0.12.2.jar:0.12.2]

at io.druid.segment.data.CompressedVSizeColumnarIntsSupplier$CompressedVSizeColumnarInts.loadBuffer(CompressedVSizeColumnarIntsSupplier.java:367) ~[druid-processing-0.12.2.jar:0.12.2]

at io.druid.segment.data.CompressedVSizeColumnarIntsSupplier$CompressedVSizeColumnarInts.get(CompressedVSizeColumnarIntsSupplier.java:340) ~[druid-processing-0.12.2.jar:0.12.2]

at io.druid.segment.column.SimpleDictionaryEncodedColumn.getSingleValueRow(SimpleDictionaryEncodedColumn.java:79) ~[druid-processing-0.12.2.jar:0.12.2]

at io.druid.segment.StringDimensionHandler.getEncodedKeyComponentFromColumn(StringDimensionHandler.java:186) ~[druid-processing-0.12.2.jar:0.12.2]

at io.druid.segment.StringDimensionHandler.getEncodedKeyComponentFromColumn(StringDimensionHandler.java:35) ~[druid-processing-0.12.2.jar:0.12.2]

at io.druid.segment.QueryableIndexIndexableAdapter$2$1.next(QueryableIndexIndexableAdapter.java:266) ~[druid-processing-0.12.2.jar:0.12.2]

at io.druid.segment.QueryableIndexIndexableAdapter$2$1.next(QueryableIndexIndexableAdapter.java:187) ~[druid-processing-0.12.2.jar:0.12.2]

at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48) ~[guava-16.0.1.jar:?]

at com.google.common.collect.Iterators$PeekingImpl.peek(Iterators.java:1162) ~[guava-16.0.1.jar:?]

at io.druid.java.util.common.guava.MergeIterator$1.compare(MergeIterator.java:48) ~[java-util-0.12.2.jar:0.12.2]

at io.druid.java.util.common.guava.MergeIterator$1.compare(MergeIterator.java:44) ~[java-util-0.12.2.jar:0.12.2]

at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48) ~[guava-16.0.1.jar:?]

at com.google.common.collect.Iterators$PeekingImpl.peek(Iterators.java:1162) ~[guava-16.0.1.jar:?]

at io.druid.java.util.common.guava.MergeIterator$1.compare(MergeIterator.java:48) ~[java-util-0.12.2.jar:0.12.2]

at io.druid.java.util.common.guava.MergeIterator$1.compare(MergeIterator.java:44) ~[java-util-0.12.2.jar:0.12.2]

at java.util.PriorityQueue.siftUpUsingComparator(PriorityQueue.java:669) ~[?:1.8.0_121]

at java.util.PriorityQueue.siftUp(PriorityQueue.java:645) ~[?:1.8.0_121]

at java.util.PriorityQueue.offer(PriorityQueue.java:344) ~[?:1.8.0_121]

at java.util.PriorityQueue.add(PriorityQueue.java:321) ~[?:1.8.0_121]

at io.druid.java.util.common.guava.MergeIterator.(MergeIterator.java:57) ~[java-util-0.12.2.jar:0.12.2]

at io.druid.java.util.common.guava.MergeIterable.iterator(MergeIterable.java:52) ~[java-util-0.12.2.jar:0.12.2]

at io.druid.collections.CombiningIterable.iterator(CombiningIterable.java:95) ~[druid-common-0.12.2.jar:0.12.2]

at io.druid.segment.IndexMergerV9.mergeIndexesAndWriteColumns(IndexMergerV9.java:456) ~[druid-processing-0.12.2.jar:0.12.2]

at io.druid.segment.IndexMergerV9.makeIndexFiles(IndexMergerV9.java:209) ~[druid-processing-0.12.2.jar:0.12.2]

at io.druid.segment.IndexMergerV9.merge(IndexMergerV9.java:837) ~[druid-processing-0.12.2.jar:0.12.2]

at io.druid.segment.IndexMergerV9.mergeQueryableIndex(IndexMergerV9.java:710) ~[druid-processing-0.12.2.jar:0.12.2]

at io.druid.segment.IndexMergerV9.mergeQueryableIndex(IndexMergerV9.java:688) ~[druid-processing-0.12.2.jar:0.12.2]

at io.druid.segment.realtime.appenderator.AppenderatorImpl.mergeAndPush(AppenderatorImpl.java:652) ~[druid-server-0.12.2.jar:0.12.2]

at io.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$push$0(AppenderatorImpl.java:563) ~[druid-server-0.12.2.jar:0.12.2]

at com.google.common.util.concurrent.Futures$1.apply(Futures.java:713) ~[guava-16.0.1.jar:?]

at com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:861) ~[guava-16.0.1.jar:?]

… 3 more

2018-10-20T12:45:53,177 INFO [task-runner-0-priority-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [compact_shopee_all_s0__microsite_test_2018-10-20T01:33:06.052Z] status changed to [FAILED].

2018-10-20T12:45:53,179 INFO [task-runner-0-priority-0] io.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {

“id” : “compact_shopee_all_s0__microsite_test_2018-10-20T01:33:06.052Z”,

“status” : “FAILED”,

“duration” : 11562146

}

2018-10-20T12:45:53,184 INFO [main] io.druid.cli.CliPeon - Thread [Thread[pool-4-thread-3,5,main]] is non daemon.

2018-10-20T12:45:53,184 INFO [main] io.druid.cli.CliPeon - Thread [Thread[pool-4-thread-1,5,main]] is non daemon.

2018-10-20T12:45:53,184 INFO [main] io.druid.cli.CliPeon - Thread [Thread[pool-4-thread-2,5,main]] is non daemon.

Hi,

would you tell me how many data you have for the interval of “2018-08-21/2018-09-20”? I think the task is failing because it’s too large to merge to a single segment.

Can you set targetPartitionSize instead of numShards or simply remove numShards to use the default value for targetPartitionSize?

It would generate multiple segments depending on the input segment size and targetPartitionSize.

Or, do you need to set numShards to 1?

Jihoon

correct, had fixed it.