Io.druid.indexing failed

TROUBLESHOOTING Google Groups
#1

I am ingesting some data from hdfs by druid,but then i saw the task failed in my log.Please help me.

2019-06-20T07:29:43,734 ERROR [task-runner-0-priority-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running task[HadoopIndexTask{id=index_hadoop_weatherReport_2019-06-20T07:29:24.505Z, type=index_hadoop, dataSource=weatherReport}]
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
	at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.1.jar:?]
	at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:222) ~[druid-indexing-service-0.12.1.3.0.0.0-1634.jar:0.12.1.3.0.0.0-1634]
	at io.druid.indexing.common.task.HadoopIndexTask.run(HadoopIndexTask.java:184) ~[druid-indexing-service-0.12.1.3.0.0.0-1634.jar:0.12.1.3.0.0.0-1634]
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444) [druid-indexing-service-0.12.1.3.0.0.0-1634.jar:0.12.1.3.0.0.0-1634]
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416) [druid-indexing-service-0.12.1.3.0.0.0-1634.jar:0.12.1.3.0.0.0-1634]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_112]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_112]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_112]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_112]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_112]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_112]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
	at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:219) ~[druid-indexing-service-0.12.1.3.0.0.0-1634.jar:0.12.1.3.0.0.0-1634]
	... 7 more
Caused by: io.druid.java.util.common.ISE: Job[class io.druid.indexer.DetermineHashedPartitionsJob] failed!
	at io.druid.indexer.JobHelper.runJobs(JobHelper.java:390) ~[druid-indexing-hadoop-0.12.1.3.0.0.0-1634.jar:0.12.1.3.0.0.0-1634]
	at io.druid.indexer.HadoopDruidDetermineConfigurationJob.run(HadoopDruidDetermineConfigurationJob.java:91) ~[druid-indexing-hadoop-0.12.1.3.0.0.0-1634.jar:0.12.1.3.0.0.0-1634]
	at io.druid.indexing.common.task.HadoopIndexTask$HadoopDetermineConfigInnerProcessing.runTask(HadoopIndexTask.java:325) ~[druid-indexing-service-0.12.1.3.0.0.0-1634.jar:0.12.1.3.0.0.0-1634]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_112]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_112]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_112]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
	at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:219) ~[druid-indexing-service-0.12.1.3.0.0.0-1634.jar:0.12.1.3.0.0.0-1634]
	... 7 more
2019-06-20T07:29:43,742 INFO [task-runner-0-priority-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_hadoop_weatherReport_2019-06-20T07:29:24.505Z] status changed to [FAILED].
2019-06-20T07:29:43,746 INFO [task-runner-0-priority-0] io.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {
  "id" : "index_hadoop_weatherReport_2019-06-20T07:29:24.505Z",
  "status" : "FAILED",
  "duration" : 9725
}
#2

Hi,

The error indicates that druid cannot determine the partitioning.

Maybe you’re missing some parameters in your spec.

Could you please post your ingestion spec ?

#3

{

“type”: “index_hadoop”,

“spec”: {

“dataSchema”: {

“dataSource”: “weatherReport”,

“parser”: {

“type”: “hadoopyString”,

“parseSpec”: {

“format”: “csv”,

“timestampSpec”: {

“column”: “timestamp”,

“format”: “auto”

},

“columns”: [“timestamp”,“vehicleId”,“vehicleType”,“manufactureId”,“adCode”,“cityChange”,“provinceChange”,“mileage”, “driveSeconds”,“maxSpeed”,“onlineSeconds”,“rAccCount”,“rDecCount”,“rTurnCount”,“condCode”,“temperature”,“humidity”,“pm25”,“airQuality”,“pop”],

“dimensionsSpec”: {

“dimensions”: [“vehicleId”,“manufactureId”,“vehicleType”,“adCode”,“condCode”,“temperature”,“humidity”,“pm25”,“airQuality”,“pop”]

}

}

},

“metricsSpec”: [

{“type”: “count”,“name”: “count”},

{“type”: “longSum”,“name”: “mileage”,“fieldName”: “mileage”},

{“type”: “longSum”,“name”: “driveSeconds”,“fieldName”: “driveSeconds”},

{“type”: “longSum”, “name”: “rAccCount”, “fieldName”: “rAccCount”},

{“type”: “longSum”, “name”: “rDecCount”, “fieldName”: “rDecCount”},

{“type”: “longSum”, “name”: “rTurnCount”, “fieldName”: “rTurnCount”},

{“type”: “longMax”, “name”: “maxSpeed”, “fieldName”: “maxSpeed”},

{“type”: “longSum”, “name”: “onlineSeconds”, “fieldName”: “onlineSeconds”},

{“type”: “longSum”, “name”: “cityChange”, “fieldName”: “cityChange”},

{“type”: “longSum”, “name”: “provinceChange”, “fieldName”: “provinceChange”}

],

“granularitySpec”: {

“type”: “uniform”,

“segmentGranularity”: “DAY”,

“queryGranularity”: “DAY”,

“rollup”: true,

“intervals” : [“2019-05-01/2019-06-01”]

}

},

“ioConfig”: {

“type”: “hadoop”,

“inputSpec”: {

“type”: “static”,

“paths”: “hdfs://testcluster/user/hue/vehicle-driving-weather-aggregate-data/report_201905/,hdfs://testcluster/user/hue/vehicle-alarm-weather-aggregate-data/report_201905/”

}

},

“tuningConfig”: {

“type”: “hadoop”,

“partitionsSpec”: {

“type”: “hashed”,

“targetPartitionSize”: 5000000

}

}

}

}

在 2019年6月20日星期四 UTC+8下午3:41:33，Guillaume Lhermenier写道：

#4 
exitCode: -1000
Failing this attempt.Diagnostics: [2019-06-20 15:29:43.459]Application application_1561000919975_0017 initialization failed (exitCode=255) with output: main : command provided 0
main : run as user is druid
main : requested yarn user is druid
Requested user druid is not whitelisted and has id 113,which is below the minimum allowed 1000

For more detailed output, check the application tracking page: http://master:8088/cluster/app/application_1561000919975_0017 Then click on links to logs of each attempt.
. Failing the application.
2019-06-20T07:29:43,713 INFO [task-runner-0-priority-0] org.apache.hadoop.mapreduce.Job - Counters: 0
2019-06-20T07:29:43,715 ERROR [task-runner-0-priority-0] io.druid.indexer.DetermineHashedPartitionsJob - Job failed: job_1561000919975_0017
2019-06-20T07:29:43,715 INFO [task-runner-0-priority-0] io.druid.indexer.JobHelper - Deleting path[/tmp/druid-indexing/weatherReport/2019-06-20T072924.505Z_2c535efa8b4a437686f47dbe4a298ab5]
2019-06-20T07:29:43,734 ERROR [task-runner-0-priority-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running task[HadoopIndexTask{id=index_hadoop_weatherReport_2019-06-20T07:29:24.505Z, type=index_hadoop, dataSource=weatherReport}]
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
	at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.1.jar:?]
	at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:222) ~[druid-indexing-service-0.12.1.3.0.0.0-1634.jar:0.12.1.3.0.0.0-1634]
	at io.druid.indexing.common.task.HadoopIndexTask.run(HadoopIndexTask.java:184) ~[druid-indexing-service-0.12.1.3.0.0.0-1634.jar:0.12.1.3.0.0.0-1634]
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444) [druid-indexing-service-0.12.1.3.0.0.0-1634.jar:0.12.1.3.0.0.0-1634]
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416) [druid-indexing-service-0.12.1.3.0.0.0-1634.jar:0.12.1.3.0.0.0-1634]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_112]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_112]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_112]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_112]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_112]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_112]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
	at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:219) ~[druid-indexing-service-0.12.1.3.0.0.0-1634.jar:0.12.1.3.0.0.0-1634]
	... 7 more
Caused by: io.druid.java.util.common.ISE: Job[class io.druid.indexer.DetermineHashedPartitionsJob] failed!
	at io.druid.indexer.JobHelper.runJobs(JobHelper.java:390) ~[druid-indexing-hadoop-0.12.1.3.0.0.0-1634.jar:0.12.1.3.0.0.0-1634]
	at io.druid.indexer.HadoopDruidDetermineConfigurationJob.run(HadoopDruidDetermineConfigurationJob.java:91) ~[druid-indexing-hadoop-0.12.1.3.0.0.0-1634.jar:0.12.1.3.0.0.0-1634]
	at io.druid.indexing.common.task.HadoopIndexTask$HadoopDetermineConfigInnerProcessing.runTask(HadoopIndexTask.java:325) ~[druid-indexing-service-0.12.1.3.0.0.0-1634.jar:0.12.1.3.0.0.0-1634]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_112]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_112]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_112]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
	at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:219) ~[druid-indexing-service-0.12.1.3.0.0.0-1634.jar:0.12.1.3.0.0.0-1634]
	... 7 more
2019-06-20T07:29:43,742 INFO [task-runner-0-priority-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_hadoop_weatherReport_2019-06-20T07:29:24.505Z] status changed to [FAILED].
2019-06-20T07:29:43,746 INFO [task-runner-0-priority-0] io.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {
  "id" : "index_hadoop_weatherReport_2019-06-20T07:29:24.505Z",
  "status" : "FAILED",
  "duration" : 9725

在 2019年6月20日星期四 UTC+8下午3:41:33，Guillaume Lhermenier写道：

#5

Can you help me? I am very anxious.

在 2019年6月20日星期四 UTC+8下午3:41:33，Guillaume Lhermenier写道：

#6 
main : run as user is druid
main : requested yarn user is druid
Requested user druid is not whitelisted and has id 113,which is below the minimum allowed 1000

It looks like there are some permissions issues, you’d probably want to check with your Hadoop cluster admin and get those resolved.

#7

Hi,I am very glad to receive your reply. We had resolve it by adjust the parameters of MapReduce. We had limit the resouce of MapReduce before. By the way,my family name is Wei(魏)，we belong to a family.Ha ha ha ha

在 2019年6月22日星期六 UTC+8上午2:30:08，Jonathan Wei写道：