0.11.0 version groupby IndexOutOfBoundsException

Hi, I update druid to 0.11.0 (imply 2.4.0)
the same groupby query return error.

{

“error” : “Unknown exception”,

“errorMessage” : “A record of size [1048744] cannot be written to the array buffer at offset[2025319760] because it exceeds the buffer capacity[536808629]. Try increasing druid.processing.buffer.sizeBytes”,

“errorClass” : “io.druid.java.util.common.ISE”,

``

query 0.10.1 is ok

my historical config

druid.service=druid/historical

druid.port=8083

HTTP server threads

druid.server.http.numThreads=40

Processing threads and buffers

druid.processing.buffer.sizeBytes=536870912

druid.processing.numMergeBuffers=4

druid.processing.numThreads=11

druid.processing.tmpDir=var/druid/processing

druid.query.groupBy.maxOnDiskStorage=10000000000

Segment storage

druid.segmentCache.locations=[{“path”:“var/druid/segment-cache”,“maxSize”:600000000000}]

druid.server.maxSize=600000000000

Query cache

druid.historical.cache.useCache=true

druid.historical.cache.populateCache=true

druid.cache.type=caffeine

druid.cache.sizeInBytes=4000000000

``

error msg

2017-12-28T15:33:08,772 ERROR [processing-2] io.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2 - Exception with one of the sequences!

io.druid.java.util.common.ISE: A record of size [1048744] cannot be written to the array buffer at offset[2025319760] because it exceeds the buffer capacity[536808629]. Try increasing druid.processing.buffer.sizeBytes

at io.druid.query.groupby.epinephelinae.BufferArrayGrouper.aggregate(BufferArrayGrouper.java:156) ~[druid-processing-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.query.groupby.epinephelinae.IntGrouper.aggregate(IntGrouper.java:33) ~[druid-processing-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.query.groupby.epinephelinae.GroupByQueryEngineV2$ArrayAggregateIterator.aggregateSingleValueDims(GroupByQueryEngineV2.java:621) ~[druid-processing-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.query.groupby.epinephelinae.GroupByQueryEngineV2$ArrayAggregateIterator.aggregateSingleValueDims(GroupByQueryEngineV2.java:601) ~[druid-processing-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.query.groupby.epinephelinae.GroupByQueryEngineV2$GroupByEngineIterator.initNewDelegate(GroupByQueryEngineV2.java:304) ~[druid-processing-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.query.groupby.epinephelinae.GroupByQueryEngineV2$GroupByEngineIterator.hasNext(GroupByQueryEngineV2.java:351) ~[druid-processing-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:45) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.ConcatSequence$1.accumulate(ConcatSequence.java:49) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.ConcatSequence$1.accumulate(ConcatSequence.java:45) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.MappingAccumulator.accumulate(MappingAccumulator.java:40) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.FilteringAccumulator.accumulate(FilteringAccumulator.java:41) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.MappingAccumulator.accumulate(MappingAccumulator.java:40) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:46) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.FilteredSequence.accumulate(FilteredSequence.java:45) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.ConcatSequence.accumulate(ConcatSequence.java:43) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.LazySequence.accumulate(LazySequence.java:40) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.LazySequence.accumulate(LazySequence.java:40) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.query.spec.SpecificSegmentQueryRunner$2.accumulate(SpecificSegmentQueryRunner.java:86) ~[druid-processing-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.query.spec.SpecificSegmentQueryRunner.doNamed(SpecificSegmentQueryRunner.java:172) ~[druid-processing-0.11.0-iap1.jar:0.11.0-iap1]

``

Hi, I update druid to 0.11.0 (imply 2.4.0)
the same groupby query return error.

{

“error” : “Unknown exception”,

“errorMessage” : “A record of size [1048744] cannot be written to the array buffer at offset[2025319760] because it exceeds the buffer capacity[536808629]. Try increasing druid.processing.buffer.sizeBytes”,

“errorClass” : “io.druid.java.util.common.ISE”,

``

query 0.10.1 is ok

my historical config

druid.service=druid/historical

druid.port=8083

HTTP server threads

druid.server.http.numThreads=40

Processing threads and buffers

druid.processing.buffer.sizeBytes=536870912

druid.processing.numMergeBuffers=4

druid.processing.numThreads=11

druid.processing.tmpDir=var/druid/processing

druid.query.groupBy.maxOnDiskStorage=10000000000

Segment storage

druid.segmentCache.locations=[{“path”:“var/druid/segment-cache”,“maxSize”:600000000000}]

druid.server.maxSize=600000000000

Query cache

druid.historical.cache.useCache=true

druid.historical.cache.populateCache=true

druid.cache.type=caffeine

druid.cache.sizeInBytes=4000000000

``

error msg

2017-12-28T15:33:08,772 ERROR [processing-2] io.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2 - Exception with one of the sequences!

io.druid.java.util.common.ISE: A record of size [1048744] cannot be written to the array buffer at offset[2025319760] because it exceeds the buffer capacity[536808629]. Try increasing druid.processing.buffer.sizeBytes

at io.druid.query.groupby.epinephelinae.BufferArrayGrouper.aggregate(BufferArrayGrouper.java:156) ~[druid-processing-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.query.groupby.epinephelinae.IntGrouper.aggregate(IntGrouper.java:33) ~[druid-processing-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.query.groupby.epinephelinae.GroupByQueryEngineV2$ArrayAggregateIterator.aggregateSingleValueDims(GroupByQueryEngineV2.java:621) ~[druid-processing-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.query.groupby.epinephelinae.GroupByQueryEngineV2$ArrayAggregateIterator.aggregateSingleValueDims(GroupByQueryEngineV2.java:601) ~[druid-processing-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.query.groupby.epinephelinae.GroupByQueryEngineV2$GroupByEngineIterator.initNewDelegate(GroupByQueryEngineV2.java:304) ~[druid-processing-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.query.groupby.epinephelinae.GroupByQueryEngineV2$GroupByEngineIterator.hasNext(GroupByQueryEngineV2.java:351) ~[druid-processing-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:45) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.ConcatSequence$1.accumulate(ConcatSequence.java:49) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.ConcatSequence$1.accumulate(ConcatSequence.java:45) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.MappingAccumulator.accumulate(MappingAccumulator.java:40) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.FilteringAccumulator.accumulate(FilteringAccumulator.java:41) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.MappingAccumulator.accumulate(MappingAccumulator.java:40) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:46) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.FilteredSequence.accumulate(FilteredSequence.java:45) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.ConcatSequence.accumulate(ConcatSequence.java:43) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.LazySequence.accumulate(LazySequence.java:40) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.LazySequence.accumulate(LazySequence.java:40) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.query.spec.SpecificSegmentQueryRunner$2.accumulate(SpecificSegmentQueryRunner.java:86) ~[druid-processing-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[java-util-0.11.0-iap1.jar:0.11.0-iap1]

at io.druid.query.spec.SpecificSegmentQueryRunner.doNamed(SpecificSegmentQueryRunner.java:172) ~[druid-processing-0.11.0-iap1.jar:0.11.0-iap1]

``

increase druid.processing.buffer.sizeBytes or druid.processing.numMergeBuffers .

在 2017年12月28日星期四 UTC+8下午3:34:33,Zhenyuan Gao写道:

My question is why 0.10.1 version not having this problem

在 2017年12月28日星期四 UTC+8下午5:27:09,zhou…@gmail.com写道: