What is "version 2 indexed" in terms of druid loading segments?

https://github.com/apache/incubator-druid/blob/bca40dcdaf3177b95d01187f413e1c0c6865964b/processing/src/main/java/org/apache/druid/segment/data/GenericIndexed.java#L130

I am seeing an issue after upgrading druid versions where one of my datasources won’t have all segments loaded by historicals because of a potential versioning issue in the segments. The stack trace is as follows:

2019-06-12T17:42:46,230 ERROR [ZkCoordinator] org.apache.druid.server.coordination.SegmentLoadDropHandler - Failed to load segment for dataSource: {class=org.apache.druid.server.coordination.SegmentLoadDropHandler, exceptionType=class org.apache.druid.segment.loading.SegmentLoadingException, exceptionMessage=Exception loading segment[sapphire-stage-druid-metrics_2019-05-21T14:00:00.000Z_2019-05-21T15:00:00.000Z_2019-05-21T14:00:14.673Z], segment=DataSegment{size=7112133889, shardSpec=NumberedShardSpec{partitionNum=0, partitions=0}, metrics=[count, value_sum, value_min, value_max], dimensions=[feed, service, host, version, metric, dataSource, duration, hasFilters, id, interval, segment, type, clusterName, memKind, poolKind, poolName, bufferpoolName, gcGen, gcName, gcGenSpaceName, context, remoteAddress, success, server, taskId, taskType, tier, priority, taskStatus], version=‘2019-05-21T14:00:14.673Z’, loadSpec={type=>hdfs, path=>hdfs://xxxxx/druid/sapphire-stage/data/sapphire-stage-druid-metrics/20190521T140000.000Z_20190521T150000.000Z/2019-05-21T14_00_14.673Z/0_index.zip}, interval=2019-05-21T14:00:00.000Z/2019-05-21T15:00:00.000Z, dataSource=‘sapphire-stage-druid-metrics’, binaryVersion=‘9’}}

org.apache.druid.segment.loading.SegmentLoadingException: Exception loading segment[sapphire-stage-druid-metrics_2019-05-21T14:00:00.000Z_2019-05-21T15:00:00.000Z_2019-05-21T14:00:14.673Z]

   at org.apache.druid.server.coordination.SegmentLoadDropHandler.loadSegment(SegmentLoadDropHandler.java:265) ~[druid-server-0.13.0.jar:0.13.0]

   at org.apache.druid.server.coordination.SegmentLoadDropHandler.addSegment(SegmentLoadDropHandler.java:307) [druid-server-0.13.0.jar:0.13.0]

   at org.apache.druid.server.coordination.SegmentChangeRequestLoad.go(SegmentChangeRequestLoad.java:47) [druid-server-0.13.0.jar:0.13.0]

   at org.apache.druid.server.coordination.ZkCoordinator$1.childEvent(ZkCoordinator.java:118) [druid-server-0.13.0.jar:0.13.0]

   at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:520) [curator-recipes-4.0.0.jar:4.0.0]

   at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:514) [curator-recipes-4.0.0.jar:4.0.0]

   at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:93) [curator-framework-4.0.0.jar:4.0.0]

   at org.apache.curator.shaded.com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:296) [curator-client-4.0.0.jar:?]

   at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:85) [curator-framework-4.0.0.jar:4.0.0]

   at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:512) [curator-recipes-4.0.0.jar:4.0.0]

   at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35) [curator-recipes-4.0.0.jar:4.0.0]

   at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:771) [curator-recipes-4.0.0.jar:4.0.0]

   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_73]

   at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_73]

   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_73]

   at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_73]

   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_73]

   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_73]

   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_73]

Caused by: org.apache.druid.java.util.common.IAE: use read(ByteBuffer buffer, ObjectStrategy strategy, SmooshedFileMapper fileMapper) to read version 2 indexed.

   at org.apache.druid.segment.data.GenericIndexed.read(GenericIndexed.java:131) ~[druid-processing-0.13.0.jar:0.13.0]

   at org.apache.druid.segment.data.CompressedVSizeColumnarIntsSupplier.fromByteBuffer(CompressedVSizeColumnarIntsSupplier.java:161) ~[druid-processing-0.13.0.jar:0.13.0]

   at org.apache.druid.segment.data.V3CompressedVSizeColumnarMultiIntsSupplier.fromByteBuffer(V3CompressedVSizeColumnarMultiIntsSupplier.java:67) ~[druid-processing-0.13.0.jar:0.13.0]

   at org.apache.druid.segment.serde.DictionaryEncodedColumnPartSerde$1.readMultiValuedColumn(DictionaryEncodedColumnPartSerde.java:381) ~[druid-processing-0.13.0.jar:0.13.0]

   at org.apache.druid.segment.serde.DictionaryEncodedColumnPartSerde$1.read(DictionaryEncodedColumnPartSerde.java:309) ~[druid-processing-0.13.0.jar:0.13.0]

   at org.apache.druid.segment.column.ColumnDescriptor.read(ColumnDescriptor.java:106) ~[druid-processing-0.13.0.jar:0.13.0]

   at org.apache.druid.segment.IndexIO$V9IndexLoader.deserializeColumn(IndexIO.java:618) ~[druid-processing-0.13.0.jar:0.13.0]

   at org.apache.druid.segment.IndexIO$V9IndexLoader.load(IndexIO.java:593) ~[druid-processing-0.13.0.jar:0.13.0]

   at org.apache.druid.segment.IndexIO.loadIndex(IndexIO.java:187) ~[druid-processing-0.13.0.jar:0.13.0]

   at org.apache.druid.segment.loading.MMappedQueryableSegmentizerFactory.factorize(MMappedQueryableSegmentizerFactory.java:48) ~[druid-processing-0.13.0.jar:0.13.0]

   at org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegment(SegmentLoaderLocalCacheManager.java:123) ~[druid-server-0.13.0.jar:0.13.0]

   at org.apache.druid.server.SegmentManager.getAdapter(SegmentManager.java:196) ~[druid-server-0.13.0.jar:0.13.0]

   at org.apache.druid.server.SegmentManager.loadSegment(SegmentManager.java:157) ~[druid-server-0.13.0.jar:0.13.0]

   at org.apache.druid.server.coordination.SegmentLoadDropHandler.loadSegment(SegmentLoadDropHandler.java:261) ~[druid-server-0.13.0.jar:0.13.0]

   ... 18 more

``

What version were you running previously? The code throwing the error was added in 0.10.0 to make it possible to handle larger columns: https://github.com/apache/incubator-druid/pull/3743 and hasn’t changed much since then. I’m wondering what might have changed such that it works before for you but not now.

By the way, IIRC the official Druid 0.13.0 jars should just be “0.13.0-incubating” not “0.13.0”. So you might have a version that has been patched somehow.

Gian

We were running druid 0.11.0 previously and had never been running pre 0.11.0 on this cluster. I believe for druid 13 we had a tweak in our build process to strip the incubating from all the pom files for some reason. But any patches applied internally by us do not relate to index building or segment loading.

I’ve not confirmed this but I think there’s possibly a bug with handling of multivalue+multipart String columns, it looks like CompressedVSizeColumnarIntsSupplier.fromByteBuffer should account for the multipart column case where it needs a SmooshedFileMapper, perhaps passed down from DictionaryEncodedColumnPartSerde.

I opened an issue: https://github.com/apache/incubator-druid/issues/7943