Ingestion tasks failing with IllegalArgumentException

Hi,

We have been experiencing this issue but couldn’t seem to find the root cause of it. All the indexing tasks seems to be failing

  • Logs:
    2018-11-09T15:45:10,861 ERROR [task-runner-0-priority-0] io.druid.indexing.common.task.MergeTaskBase - Exception merging[test-requests]: {class=io.druid.indexing.common.task.MergeTaskBase, exceptionType=class java.lang.IllegalArgumentException, exceptionMessage=Instantiation of [simple type, class io.druid.segment.loading.LocalLoadSpec] value failed: [/data/druid/deep/test-requests/2018-09-21T11:00:00.000Z_2018-10-31T14:00:00.000Z/2018-10-31T14:21:43.268Z/0/index.zip] does not exist, interval=2018-09-21T11:00:00.000Z/2018-11-09T15:00:00.000Z}

java.lang.IllegalArgumentException: Instantiation of [simple type, class io.druid.segment.loading.LocalLoadSpec] value failed: [/data/druid/deep/test-requests/2018-09-21T11:00:00.000Z_2018-10-31T14:00:00.000Z/2018-10-31T14:21:43.268Z/0/index.zip] does not exist

at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:2774) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:2700) ~[jackson-databind-2.4.6.jar:2.4.6]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocation(SegmentLoaderLocalCacheManager.java:205) ~[druid-server-0.10.1.jar:0.10.1]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocationWithStartMarker(SegmentLoaderLocalCacheManager.java:195) ~[druid-server-0.10.1.jar:0.10.1]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadSegmentWithRetry(SegmentLoaderLocalCacheManager.java:154) ~[druid-server-0.10.1.jar:0.10.1]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegmentFiles(SegmentLoaderLocalCacheManager.java:130) ~[druid-server-0.10.1.jar:0.10.1]

at io.druid.indexing.common.TaskToolbox.fetchSegments(TaskToolbox.java:214) ~[druid-indexing-service-0.10.1.jar:0.10.1]

at io.druid.indexing.common.task.MergeTaskBase.run(MergeTaskBase.java:155) [druid-indexing-service-0.10.1.jar:0.10.1]

at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) [druid-indexing-service-0.10.1.jar:0.10.1]

at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) [druid-indexing-service-0.10.1.jar:0.10.1]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_144]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_144]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_144]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]

Caused by: com.fasterxml.jackson.databind.JsonMappingException: Instantiation of [simple type, class io.druid.segment.loading.LocalLoadSpec] value failed: [/data/druid/deep/test-requests/2018-09-21T11:00:00.000Z_2018-10-31T14:00:00.000Z/2018-10-31T14:21:43.268Z/0/index.zip] does not exist

at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.wrapException(StdValueInstantiator.java:405) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith(StdValueInstantiator.java:234) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.impl.PropertyBasedCreator.build(PropertyBasedCreator.java:167) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:398) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1064) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:264) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:156) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:126) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:113) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:84) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:132) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.impl.TypeWrappedDeserializer.deserialize(TypeWrappedDeserializer.java:41) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:2769) ~[jackson-databind-2.4.6.jar:2.4.6]

… 13 more

Caused by: java.lang.IllegalArgumentException: [/data/druid/deep/test-requests/2018-09-21T11:00:00.000Z_2018-10-31T14:00:00.000Z/2018-10-31T14:21:43.268Z/0/index.zip] does not exist

at com.google.common.base.Preconditions.checkArgument(Preconditions.java:148) ~[guava-16.0.1.jar:?]

at io.druid.segment.loading.LocalLoadSpec.(LocalLoadSpec.java:51) ~[druid-server-0.10.1.jar:0.10.1]

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_144]

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_144]

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_144]

at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_144]

at com.fasterxml.jackson.databind.introspect.AnnotatedConstructor.call(AnnotatedConstructor.java:125) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith(StdValueInstantiator.java:230) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.impl.PropertyBasedCreator.build(PropertyBasedCreator.java:167) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:398) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1064) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:264) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:156) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:126) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:113) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:84) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:132) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.deser.impl.TypeWrappedDeserializer.deserialize(TypeWrappedDeserializer.java:41) ~[jackson-databind-2.4.6.jar:2.4.6]

at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:2769) ~[jackson-databind-2.4.6.jar:2.4.6]

… 13 more

2018-11-09T15:45:10,866 INFO [task-runner-0-priority-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [merge_test-requests_bdd572cb7e9d4d83752bb158970d34710efe6685_2018-11-09T15:44:59.482Z] status changed to [FAILED].

2018-11-09T15:45:10,868 INFO [task-runner-0-priority-0] io.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {

“id” : “merge_test-requests_bdd572cb7e9d4d83752bb158970d34710efe6685_2018-11-09T15:44:59.482Z”,

“status” : “FAILED”,

“duration” : 33

}

It looks like the task is not able to read input segments from deep storage, I would recommend checking that the segment file is present and accessible from the middle manager node where that task is being run.

Thanks,

Jon

Hey Jon,

Thanks for the reply.

I was able to see that segment is being created in another path and deep storage doesn’t have that segment.

2018-11-12T14:00:55,147 INFO [main] io.druid.indexing.worker.executor.ExecutorLifecycle - Running with task: {

“type” : “append”,

“id” : “merge_test-requests_75a016d378f18e97af34fc7f22f2894d3d1142e5_2018-11-12T14:00:44.001Z”,

“dataSource” : “test-requests”,

“segments” : [ {

“dataSource” : “test-requests”,

“interval” : “2018-09-21T11:00:00.000Z/2018-10-31T14:00:00.000Z”,

“version” : “2018-10-31T14:21:43.268Z”,

“loadSpec” : {

“type” : “local”,

“path” : “/data/druid/deep/test-requests/2018-09-21T11:00:00.000Z_2018-10-31T14:00:00.000Z/2018-10-31T14:21:43.268Z/0/index.zip”

},

“dimensions” : “env,host,site”,

“metrics” : “bytesSum,count,durationSum,unique_ips,unique_users”,

“shardSpec” : {

“type” : “none”

},

“binaryVersion” : 9,

“size” : 1525630,

“identifier” : “test-requests_2018-09-21T11:00:00.000Z_2018-10-31T14:00:00.000Z_2018-10-31T14:21:43.268Z”

}, {

“dataSource” : “test-requests”,

“interval” : “2018-10-31T15:00:00.000Z/2018-10-31T16:00:00.000Z”,

“version” : “2018-10-31T15:00:00.000Z”,

“loadSpec” : {

“type” : “local”,

“path” : “/auto/stage-data/druid/data/test-requests/2018-10-31T15:00:00.000Z_2018-10-31T16:00:00.000Z/2018-10-31T15:00:00.000Z/0/index.zip”

The deep storage mentioned in properties file is deep_storage_path => ‘/auto/stage-data/druid/data’.