Getting exception on hadoop batch ingestion of CSV file

Hi,

Currently we are using druid .10 from hdp cluster 2.6.5.

Problem:

After submitting the hadoop batch ingestion task we are getting the below exception logs from the coordinator task.

Kindly suggest what is missing here as we are trying this from very long time as facing same issue.

java.lang.RuntimeException: java.lang.reflect.InvocationTargetException

at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.1.jar:?]

at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:218) ~[druid-indexing-service-0.10.1.2.6.5.0-292.jar:0.10.1.2.6.5.0-292]

at io.druid.indexing.common.task.HadoopIndexTask.run(HadoopIndexTask.java:178) ~[druid-indexing-service-0.10.1.2.6.5.0-292.jar:0.10.1.2.6.5.0-292]

at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) [druid-indexing-service-0.10.1.2.6.5.0-292.jar:0.10.1.2.6.5.0-292]

at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) [druid-indexing-service-0.10.1.2.6.5.0-292.jar:0.10.1.2.6.5.0-292]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_112]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_112]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_112]

at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]

Caused by: java.lang.reflect.InvocationTargetException

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_112]

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_112]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_112]

at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]

at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:215) ~[druid-indexing-service-0.10.1.2.6.5.0-292.jar:0.10.1.2.6.5.0-292]

… 7 more

Caused by: io.druid.java.util.common.ISE: Job[class io.druid.indexer.DetermineHashedPartitionsJob] failed!

at io.druid.indexer.JobHelper.runJobs(JobHelper.java:389) ~[druid-indexing-hadoop-0.10.1.2.6.5.0-292.jar:0.10.1.2.6.5.0-292]

at io.druid.indexer.HadoopDruidDetermineConfigurationJob.run(HadoopDruidDetermineConfigurationJob.java:91) ~[druid-indexing-hadoop-0.10.1.2.6.5.0-292.jar:0.10.1.2.6.5.0-292]

at io.druid.indexing.common.task.HadoopIndexTask$HadoopDetermineConfigInnerProcessing.runTask(HadoopIndexTask.java:308) ~[druid-indexing-service-0.10.1.2.6.5.0-292.jar:0.10.1.2.6.5.0-292]

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_112]

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_112]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_112]

at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]

at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:215) ~[druid-indexing-service-0.10.1.2.6.5.0-292.jar:0.10.1.2.6.5.0-292]

… 7 more

2019-02-11T11:55:04,130 INFO [task-runner-0-priority-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_hadoop_hadoop_eblock_device_performance_2019-02-11T11:54:52.268Z] status changed to [FAILED].

2019-02-11T11:55:04,132 INFO [task-runner-0-priority-0] io.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {

“id” : “index_hadoop_hadoop_eblock_device_performance_2019-02-11T11:54:52.268Z”,

“status” : “FAILED”,

“duration” : 8523

}

Hi Mohammed:

Can you check the YARN task log and let us know the error message there?

Please also show us the ingestion spec, as well as the what’s under path ‘./dist/druid/hadoop-dependencies’

Thanks

Ming

Hi Mohammed

Review / grep through overlord / middlemanager logs to find what happened to this job:
index_hadoop_hadoop_eblock_device_performance_2019-02-11T11:54:52.268Z

That might give some clues as to what happened.

Hi Mohammed,

Could you please share the index file you are using and also the csv file sample ?

Thanks,

Heta