Migrating from Azure to S3

Hello,

I’m trying to migrate a DB from Azure Blob storage to S3. I have changed the Metadata DB to reflect the new paths and configs. However, I’m getting the following error in the Historical node:

io.druid.segment.loading.SegmentLoadingException: Exception loading segment[mydatasource_2019-04-22T14:00:00.000Z_2019-04-22T15:00:00.000Z_2019-04-22T14:00:39.410Z_1]

at io.druid.server.coordination.SegmentLoadDropHandler.loadSegment(SegmentLoadDropHandler.java:265) ~[druid-server-0.12.2.jar:0.12.2]

at io.druid.server.coordination.SegmentLoadDropHandler.addSegment(SegmentLoadDropHandler.java:307) [druid-server-0.12.2.jar:0.12.2]

at io.druid.server.coordination.SegmentChangeRequestLoad.go(SegmentChangeRequestLoad.java:47) [druid-server-0.12.2.jar:0.12.2]

at io.druid.server.coordination.ZkCoordinator$1.childEvent(ZkCoordinator.java:118) [druid-server-0.12.2.jar:0.12.2]

at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:520) [curator-recipes-4.0.0.jar:4.0.0]

at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:514) [curator-recipes-4.0.0.jar:4.0.0]

at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:93) [curator-framework-4.0.0.jar:4.0.0]

at org.apache.curator.shaded.com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:296) [curator-client-4.0.0.jar:?]

at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:85) [curator-framework-4.0.0.jar:4.0.0]

at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:512) [curator-recipes-4.0.0.jar:4.0.0]

at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35) [curator-recipes-4.0.0.jar:4.0.0]

at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:771) [curator-recipes-4.0.0.jar:4.0.0]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_60]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_60]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_60]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_60]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_60]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_60]

at java.lang.Thread.run(Thread.java:745) [?:1.8.0_60]

Caused by: java.lang.IllegalArgumentException: Could not resolve type id ‘azure’ into a subtype of [simple type, class io.druid.segment.loading.LoadSpec]

at [Source: N/A; line: -1, column: -1]

at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:2774) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:2700) ~[jackson-databind-2.4.6.jar:2.4.6]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocation(SegmentLoaderLocalCacheManager.java:204) ~[druid-server-0.12.2.jar:0.12.2]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocationWithStartMarker(SegmentLoaderLocalCacheManager.java:193) ~[druid-server-0.12.2.jar:0.12.2]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadSegmentWithRetry(SegmentLoaderLocalCacheManager.java:151) ~[druid-server-0.12.2.jar:0.12.2]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegmentFiles(SegmentLoaderLocalCacheManager.java:133) ~[druid-server-0.12.2.jar:0.12.2]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegment(SegmentLoaderLocalCacheManager.java:108) ~[druid-server-0.12.2.jar:0.12.2]

at io.druid.server.SegmentManager.getAdapter(SegmentManager.java:196) ~[druid-server-0.12.2.jar:0.12.2]

at io.druid.server.SegmentManager.loadSegment(SegmentManager.java:157) ~[druid-server-0.12.2.jar:0.12.2]

at io.druid.server.coordination.SegmentLoadDropHandler.loadSegment(SegmentLoadDropHandler.java:261) ~[druid-server-0.12.2.jar:0.12.2]

… 18 more

Caused by: com.fasterxml.jackson.databind.JsonMappingException: Could not resolve type id ‘azure’ into a subtype of [simple type, class io.druid.segment.loading.LoadSpec]

at [Source: N/A; line: -1, column: -1]

at com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:148) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.DeserializationContext.unknownTypeException(DeserializationContext.java:862) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.jsontype.impl.TypeDeserializerBase._findDeserializer(TypeDeserializerBase.java:167) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:99) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:84) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:132) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.impl.TypeWrappedDeserializer.deserialize(TypeWrappedDeserializer.java:41) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:2769) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:2700) ~[jackson-databind-2.4.6.jar:2.4.6]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocation(SegmentLoaderLocalCacheManager.java:204) ~[druid-server-0.12.2.jar:0.12.2]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocationWithStartMarker(SegmentLoaderLocalCacheManager.java:193) ~[druid-server-0.12.2.jar:0.12.2]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadSegmentWithRetry(SegmentLoaderLocalCacheManager.java:151) ~[druid-server-0.12.2.jar:0.12.2]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegmentFiles(SegmentLoaderLocalCacheManager.java:133) ~[druid-server-0.12.2.jar:0.12.2]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegment(SegmentLoaderLocalCacheManager.java:108) ~[druid-server-0.12.2.jar:0.12.2]

at io.druid.server.SegmentManager.getAdapter(SegmentManager.java:196) ~[druid-server-0.12.2.jar:0.12.2]

at io.druid.server.SegmentManager.loadSegment(SegmentManager.java:157) ~[druid-server-0.12.2.jar:0.12.2]

at io.druid.server.coordination.SegmentLoadDropHandler.loadSegment(SegmentLoadDropHandler.java:261) ~[druid-server-0.12.2.jar:0.12.2]

… 18 more

I think this means that the actual data in the deep storage references the fact that it’s on Azure.

How can I get over this problem?

Thanks

Hi,

I haven’t the solution but hit a similar issue today :

I moved my deep storage from one bucket to another.

Segment loading was ok, as querying was.

But when I tried to index a new datasource from those segments, the name of the old bucket appeared in the error stack.

I was also surprised that datas seem to be deeply linked to where they are ingested.

On my side, I had the possibility to reindex datas from original source, but I’m interested in knowing if there are others possibilities to solve that.

Guillaume

Upgrading to version 0.14.1-incubating fixes the issue. I’m still verifying that everything works, but at least I see the segments being loaded.

Great news Sebastian - four our information and other’s, what version did you upgrade from?

Cheers,

Daniel

Ok, so mine is still present as I already am in 0.14.1

Ah, yes. I was using 0.12.2