Failed Segment Merge - exceptionMessage=Index[1] >= size[1]

Ran into a strange error today after turning auto merging on for our coordinators. Many of the merge tasks completed just fine but about 1/3 terminated with this exception:

com.metamx.common.IAE: Index[1] >= size[1]

    at io.druid.segment.data.GenericIndexed$BufferIndexed._get(GenericIndexed.java:198) ~[druid-processing-0.8.3.jar:0.8.3]

    at io.druid.segment.data.GenericIndexed$1.get(GenericIndexed.java:296) ~[druid-processing-0.8.3.jar:0.8.3]

    at io.druid.segment.data.CachingIndexed.get(CachingIndexed.java:84) ~[druid-processing-0.8.3.jar:0.8.3]

    at io.druid.segment.column.SimpleDictionaryEncodedColumn.lookupName(SimpleDictionaryEncodedColumn.java:76) ~[druid-processing-0.8.3.jar:0.8.3]

    at io.druid.segment.QueryableIndexIndexableAdapter$1.get(QueryableIndexIndexableAdapter.java:150) ~[druid-processing-0.8.3.jar:0.8.3]

    at io.druid.segment.QueryableIndexIndexableAdapter$1.get(QueryableIndexIndexableAdapter.java:134) ~[druid-processing-0.8.3.jar:0.8.3]

    at io.druid.segment.QueryableIndexIndexableAdapter$3.seek(QueryableIndexIndexableAdapter.java:388) ~[druid-processing-0.8.3.jar:0.8.3]

    at io.druid.segment.IndexMerger.makeIndexFiles(IndexMerger.java:870) ~[druid-processing-0.8.3.jar:0.8.3]

    at io.druid.segment.IndexMerger.append(IndexMerger.java:480) ~[druid-processing-0.8.3.jar:0.8.3]

    at io.druid.segment.IndexMerger.append(IndexMerger.java:407) ~[druid-processing-0.8.3.jar:0.8.3]

    at io.druid.indexing.common.task.AppendTask.merge(AppendTask.java:130) ~[druid-indexing-service-0.8.3.jar:0.8.3]

    at io.druid.indexing.common.task.MergeTaskBase.run(MergeTaskBase.java:152) [druid-indexing-service-0.8.3.jar:0.8.3]

    at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:285) [druid-indexing-service-0.8.3.jar:0.8.3]

    at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:265) [druid-indexing-service-0.8.3.jar:0.8.3]

    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_31]

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_31]

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_31]

    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_31]

``

Skimming through the code it looks like the offset requested is larger than the data:

We are currently running 0.8.3 but some of the segments hosted by the cluster have been around since we were running.

Anyone else run into this before?

Hey Drew,

I believe this was fixed in Druid 0.9.0, could you give that a try and report back if you still have problems?