Direct Memory Usage With Sketches

I’m attempting to convert some of our hyperUnique metrics to thetaSketches, but it appears to be using a lot more direct memory as a result, far more than the docs suggest.

Peon Memory Settings:

-XX:MaxDirectMemorySize=4g

druid.indexer.fork.property.druid.processing.buffer.sizeBytes=524288000

druid.indexer.fork.property.druid.processing.numMergeBuffer=2

druid.indexer.fork.property.druid.processing.numThreads=1

Using the formula in the docs (druid.processing.numThreads + druid.processing.numMergeBuffers + 1) * druid.processing.buffer.sizeBytes,

I get (2 + 1 + 1) * 524288000 = 2097152000

Actual usage though is much higher:

2019-07-26T16:45:47,461 INFO [MonitorScheduler-0] org.apache.druid.java.util.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2019-07-26T16:45:47.461Z”,“service”:“druid/middleManager”,“host”:“c3puanalytics4.premierinc.com:8100”,“version”:“0.15.0-incubating”,“metric”:“jvm/bufferpool/capacity”,“value”:4294949846,“bufferpoolName”:“direct”,“dataSource”:[“analytics-data-daily-temp”],“id”:[“compact_analytics-data-daily-temp_2019-07-26T16:40:40.513Z”]}]

2019-07-26T16:45:47,461 INFO [MonitorScheduler-0] org.apache.druid.java.util.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2019-07-26T16:45:47.461Z”,“service”:“druid/middleManager”,“host”:“c3puanalytics4.premierinc.com:8100”,“version”:“0.15.0-incubating”,“metric”:"jvm/bufferpool/used",“value”:4294949847,“bufferpoolName”:“direct”,“dataSource”:[“analytics-data-daily-temp”],“id”:[“compact_analytics-data-daily-temp_2019-07-26T16:40:40.513Z”]}]

Dimension conversions complete as expected:

2019-07-26T15:16:21,579 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMerger - Allocated [12,432] bytes of dictionary merging direct buffers

2019-07-26T15:16:21,580 INFO [appenderator_merge_0] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[user_type] conversions with cardinality[3] in 9 millis.

2019-07-26T15:16:21,580 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMergerV9 - Completed dim conversions in 9,724 millis.

Starts allocation a ton of buffers and eventually fails:

2019-07-26T15:16:22,641 INFO [appenderator_merge_0] org.apache.druid.segment.CompressedPools - Allocating new littleEndByteBuf[63]

2019-07-26T15:16:22,641 INFO [appenderator_merge_0] org.apache.druid.segment.CompressedPools - Allocating new littleEndByteBuf[64]

2019-07-26T15:16:30,461 INFO [appenderator_merge_0] org.apache.druid.segment.CompressedPools - Allocating new littleEndByteBuf[93,213]

2019-07-26T15:16:30,461 INFO [appenderator_merge_0] org.apache.druid.segment.CompressedPools - Allocating new littleEndByteBuf[93,214]

2019-07-26T15:16:30,461 INFO [appenderator_merge_0] org.apache.druid.segment.CompressedPools - Allocating new littleEndByteBuf[93,215]

2019-07-26T15:16:30,461 INFO [appenderator_merge_0] org.apache.druid.segment.CompressedPools - Allocating new littleEndByteBuf[93,216]

2019-07-26T15:16:30,462 INFO [appenderator_merge_0] org.apache.druid.segment.CompressedPools - Allocating new littleEndByteBuf[93,217]

2019-07-26T15:16:34,460 INFO [task-runner-0-priority-0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Shutting down…

2019-07-26T15:16:34,463 INFO [appenderator_persist_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[analytics-data-daily-temp_2019-06-28T00:00:00.000Z_2019-07-08T00:00:00.000Z_2019-07-26T15:10:39.625Z].

2019-07-26T15:16:34,558 ERROR [task-runner-0-priority-0] org.apache.druid.indexing.common.task.IndexTask - Encountered exception in BUILD_SEGMENTS.

java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Direct buffer memory

at org.apache.druid.indexing.common.task.IndexTask.generateAndPublishSegments(IndexTask.java:1033) ~[druid-indexing-service-0.15.0-incubating.jar:0.15.0-incubating]

at org.apache.druid.indexing.common.task.IndexTask.run(IndexTask.java:467) [druid-indexing-service-0.15.0-incubating.jar:0.15.0-incubating]

at org.apache.druid.indexing.common.task.CompactionTask.run(CompactionTask.java:333) [druid-indexing-service-0.15.0-incubating.jar:0.15.0-incubating]

at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:419) [druid-indexing-service-0.15.0-incubating.jar:0.15.0-incubating]

at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:391) [druid-indexing-service-0.15.0-incubating.jar:0.15.0-incubating]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_161]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]

Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Direct buffer memory

at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) ~[guava-16.0.1.jar:?]

at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) ~[guava-16.0.1.jar:?]

at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-16.0.1.jar:?]

at org.apache.druid.segment.realtime.appenderator.BatchAppenderatorDriver.pushAndClear(BatchAppenderatorDriver.java:141) ~[druid-server-0.15.0-incubating.jar:0.15.0-incubating]

at org.apache.druid.segment.realtime.appenderator.BatchAppenderatorDriver.pushAllAndClear(BatchAppenderatorDriver.java:124) ~[druid-server-0.15.0-incubating.jar:0.15.0-incubating]

at org.apache.druid.indexing.common.task.IndexTask.generateAndPublishSegments(IndexTask.java:1002) ~[druid-indexing-service-0.15.0-incubating.jar:0.15.0-incubating]

… 8 more

Caused by: java.lang.OutOfMemoryError: Direct buffer memory

at java.nio.Bits.reserveMemory(Bits.java:695) ~[?:1.8.0_161]

at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123) ~[?:1.8.0_161]

at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) ~[?:1.8.0_161]

at org.apache.druid.segment.CompressedPools$4.get(CompressedPools.java:105) ~[druid-processing-0.15.0-incubating.jar:0.15.0-incubating]

at org.apache.druid.segment.CompressedPools$4.get(CompressedPools.java:98) ~[druid-processing-0.15.0-incubating.jar:0.15.0-incubating]

at org.apache.druid.collections.StupidPool.makeObjectWithHandler(StupidPool.java:116) ~[druid-core-0.15.0-incubating.jar:0.15.0-incubating]

at org.apache.druid.collections.StupidPool.take(StupidPool.java:107) ~[druid-core-0.15.0-incubating.jar:0.15.0-incubating]

at org.apache.druid.segment.CompressedPools.getByteBuf(CompressedPools.java:113) ~[druid-processing-0.15.0-incubating.jar:0.15.0-incubating]

at org.apache.druid.segment.data.DecompressingByteBufferObjectStrategy.fromByteBuffer(DecompressingByteBufferObjectStrategy.java:49) ~[druid-processing-0.15.0-incubating.jar:0.15.0-incubating]

at org.apache.druid.segment.data.DecompressingByteBufferObjectStrategy.fromByteBuffer(DecompressingByteBufferObjectStrategy.java:28) ~[druid-processing-0.15.0-incubating.jar:0.15.0-incubating]

at org.apache.druid.segment.data.GenericIndexed$BufferIndexed.bufferedIndexedGet(GenericIndexed.java:444) ~[druid-processing-0.15.0-incubating.jar:0.15.0-incubating]

at org.apache.druid.segment.data.GenericIndexed$2.get(GenericIndexed.java:599) ~[druid-processing-0.15.0-incubating.jar:0.15.0-incubating]

at org.apache.druid.segment.data.CompressedVSizeColumnarIntsSupplier$CompressedVSizeColumnarInts.loadBuffer(CompressedVSizeColumnarIntsSupplier.java:346) ~[druid-processing-0.15.0-incubating.jar:0.15.0-incubating]

at org.apache.druid.segment.data.CompressedVSizeColumnarIntsSupplier$CompressedVSizeColumnarInts.get(CompressedVSizeColumnarIntsSupplier.java:319) ~[druid-processing-0.15.0-incubating.jar:0.15.0-incubating]

at org.apache.druid.segment.column.StringDictionaryEncodedColumn$1SingleValueQueryableDimensionSelector.getRowValue(StringDictionaryEncodedColumn.java:218) ~[druid-processing-0.15.0-incubating.jar:0.15.0-incubating]

at org.apache.druid.segment.column.StringDictionaryEncodedColumn$1SingleValueQueryableDimensionSelector.getRow(StringDictionaryEncodedColumn.java:212) ~[druid-processing-0.15.0-incubating.jar:0.15.0-incubating]

at org.apache.druid.segment.selector.settable.SettableDimensionValueSelector.setValueFrom(SettableDimensionValueSelector.java:47) ~[druid-processing-0.15.0-incubating.jar:0.15.0-incubating]

at org.apache.druid.segment.QueryableIndexIndexableAdapter$RowIteratorImpl.setRowPointerValues(QueryableIndexIndexableAdapter.java:319) ~[druid-processing-0.15.0-incubating.jar:0.15.0-incubating]

at org.apache.druid.segment.QueryableIndexIndexableAdapter$RowIteratorImpl.moveToNext(QueryableIndexIndexableAdapter.java:298) ~[druid-processing-0.15.0-incubating.jar:0.15.0-incubating]

at org.apache.druid.segment.ForwardingRowIterator.moveToNext(ForwardingRowIterator.java:62) ~[druid-processing-0.15.0-incubating.jar:0.15.0-incubating]

at org.apache.druid.segment.ForwardingRowIterator.moveToNext(ForwardingRowIterator.java:62) ~[druid-processing-0.15.0-incubating.jar:0.15.0-incubating]

at org.apache.druid.segment.MergingRowIterator.lambda$new$0(MergingRowIterator.java:84) ~[druid-processing-0.15.0-incubating.jar:0.15.0-incubating]

at java.util.stream.IntPipeline$9$1.accept(IntPipeline.java:343) ~[?:1.8.0_161]

at java.util.stream.Streams$RangeIntSpliterator.forEachRemaining(Streams.java:110) ~[?:1.8.0_161]

at java.util.Spliterator$OfInt.forEachRemaining(Spliterator.java:693) ~[?:1.8.0_161]

at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_161]

at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_161]

at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:545) ~[?:1.8.0_161]

at java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260) ~[?:1.8.0_161]

at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:438) ~[?:1.8.0_161]

at org.apache.druid.segment.MergingRowIterator.(MergingRowIterator.java:92) ~[druid-processing-0.15.0-incubating.jar:0.15.0-incubating]

at org.apache.druid.segment.RowCombiningTimeAndDimsIterator.(RowCombiningTimeAndDimsIterator.java:108) ~[druid-processing-0.15.0-incubating.jar:0.15.0-incubating]

at org.apache.druid.segment.IndexMergerV9.lambda$merge$2(IndexMergerV9.java:910) ~[druid-processing-0.15.0-incubating.jar:0.15.0-incubating]

at org.apache.druid.segment.IndexMergerV9.makeMergedTimeAndDimsIterator(IndexMergerV9.java:1031) ~[druid-processing-0.15.0-incubating.jar:0.15.0-incubating]

at org.apache.druid.segment.IndexMergerV9.makeIndexFiles(IndexMergerV9.java:180) ~[druid-processing-0.15.0-incubating.jar:0.15.0-incubating]

at org.apache.druid.segment.IndexMergerV9.merge(IndexMergerV9.java:915) ~[druid-processing-0.15.0-incubating.jar:0.15.0-incubating]

at org.apache.druid.segment.IndexMergerV9.mergeQueryableIndex(IndexMergerV9.java:833) ~[druid-processing-0.15.0-incubating.jar:0.15.0-incubating]

at org.apache.druid.segment.IndexMergerV9.mergeQueryableIndex(IndexMergerV9.java:811) ~[druid-processing-0.15.0-incubating.jar:0.15.0-incubating]

at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.mergeAndPush(AppenderatorImpl.java:726) ~[druid-server-0.15.0-incubating.jar:0.15.0-incubating]

at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$push$1(AppenderatorImpl.java:630) ~[druid-server-0.15.0-incubating.jar:0.15.0-incubating]

at com.google.common.util.concurrent.Futures$1.apply(Futures.java:713) ~[guava-16.0.1.jar:?]

at com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:861) ~[guava-16.0.1.jar:?]

… 3 more

Steve,

The formula is just the baseline/guideline and this will change based on your query pattern.

Rommel Garcia

Thanks Rommel,

I realize it’s just an estimate, but I’m overshooting it by such a large margin, I’m wondering if the docs need to be updated. I’m still failing at 8gb, when 2 should be sufficient (according to the estimate).

Thanks,

Steve

What is the key size you are using for thetaSketches?

Rommel Garcia

I have a total of 9 sketches. All 9 initially used the default size. Reducing most to 1024 / 2048, with a couple at 4096 has solved the issue.