Hi team!
I have trouble with reindexing task.
Here is my reindexing json
{
"type": "index_hadoop",
"spec": {
"dataSchema": {
"dataSource": "myDataSource"
},
"ioConfig": {
"type": "hadoop",
"inputSpec": {
"type": "dataSource",
"ingestionSpec": {
"dataSource": "myDataSource",
"intervals": ["2017-09-26T17:00:00Z/PT1H"]
}
}
},
"tuningConfig": {
"type": "hadoop",
"jobProperties": {
"mapreduce.job.queuename": "root.druid.batch",
"mapreduce.job.classloader": "true",
"mapreduce.job.classloader.system.classes": "-javax.validation.,java.,javax.,org.apache.commons.logging.,org.apache.log4j.,org.apache.hadoop.",
"mapreduce.map.memory.mb": 4096,
"mapreduce.map.java.opts": "-server -Xmx4096m -Duser.timezone=UTC -Dfile.encoding=UTF-8",
"mapreduce.reduce.memory.mb": 8192,
"mapreduce.reduce.java.opts": "-server -Xmx8g -Duser.timezone=UTC -Dfile.encoding=UTF-8"
},
"partitionsSpec": {
"type": "dimension",
"partitionDimension": "pd",
"targetPartitionSize": 8333333
},
"buildV9Directly": "true"
}
},
"hadoopDependencyCoordinates": ["org.apache.hadoop:hadoop-client:2.6.0"]
}
This is the indexer log from overlord web-console ui.
2017-09-26T10:32:50,958 INFO [task-runner-0-priority-0] io.druid.indexing.common.task.HadoopIndexTask - Starting a hadoop determine configuration job...
2017-09-26T10:32:50,988 INFO [task-runner-0-priority-0] io.druid.indexer.path.DatasourcePathSpec - Found total [36] segments for [myDataSource] in interval [[2017-09-26T17:00:00.000Z/2017-09-26T18:00:00.000Z]]
2017-09-26T10:32:50,988 WARN [task-runner-0-priority-0] io.druid.segment.indexing.DataSchema - No parser has been specified
2017-09-26T10:32:50,989 ERROR [task-runner-0-priority-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running task[HadoopIndexTask{id=index_hadoop_myDataSource_2017-09-26T10:32:43.389Z, type=index_hadoop, dataSource=myDataSource}]
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.1.jar:?]
at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:211) ~[druid-indexing-service-0.10.0.jar:0.10.0]
at io.druid.indexing.common.task.HadoopIndexTask.run(HadoopIndexTask.java:176) ~[druid-indexing-service-0.10.0.jar:0.10.0]
at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) [druid-indexing-service-0.10.0.jar:0.10.0]
at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) [druid-indexing-service-0.10.0.jar:0.10.0]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_60]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_60]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_60]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_60]
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_60]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_60]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_60]
at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_60]
at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:208) ~[druid-indexing-service-0.10.0.jar:0.10.0]
... 7 more
Caused by: java.lang.NullPointerException
at io.druid.indexer.path.DatasourcePathSpec.addInputPaths(DatasourcePathSpec.java:117) ~[druid-indexing-hadoop-0.10.0.jar:0.10.0]
at io.druid.indexer.HadoopDruidIndexerConfig.addInputPaths(HadoopDruidIndexerConfig.java:389) ~[druid-indexing-hadoop-0.10.0.jar:0.10.0]
at io.druid.indexer.JobHelper.ensurePaths(JobHelper.java:337) ~[druid-indexing-hadoop-0.10.0.jar:0.10.0]
at io.druid.indexer.HadoopDruidDetermineConfigurationJob.run(HadoopDruidDetermineConfigurationJob.java:55) ~[druid-indexing-hadoop-0.10.0.jar:0.10.0]
at io.druid.indexing.common.task.HadoopIndexTask$HadoopDetermineConfigInnerProcessing.runTask(HadoopIndexTask.java:306) ~[druid-indexing-service-0.10.0.jar:0.10.0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_60]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_60]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_60]
at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_60]
at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:208) ~[druid-indexing-service-0.10.0.jar:0.10.0]
... 7 more
This task executed via following command
curl -v -L -XPOST -H'Content-Type: application/json' -d @$jsonFile http://$overlord:$overlordPort/druid/indexer/v1/task
Can you plz tell me where is the invalid point?
I read following document several times but i could not found flaws.
And can you give some full sample json?
It would be very useful.
Thanks, have a nice day.
