Ingest data failed

**Please help me.I am ingest data from hdfs by druid. But it failed.**


Caused by: io.druid.java.util.common.ISE: Job[class io.druid.indexer.DetermineHashedPartitionsJob] failed!


java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.1.jar:?]
at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:222) ~[druid-indexing-service-0.12.1.3.0.0.0-1634.jar:0.12.1.3.0.0.0-1634]
at io.druid.indexing.common.task.HadoopIndexTask.run(HadoopIndexTask.java:184) ~[druid-indexing-service-0.12.1.3.0.0.0-1634.jar:0.12.1.3.0.0.0-1634]
at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444) [druid-indexing-service-0.12.1.3.0.0.0-1634.jar:0.12.1.3.0.0.0-1634]
at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416) [druid-indexing-service-0.12.1.3.0.0.0-1634.jar:0.12.1.3.0.0.0-1634]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_112]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_112]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_112]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_112]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_112]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_112]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:219) ~[druid-indexing-service-0.12.1.3.0.0.0-1634.jar:0.12.1.3.0.0.0-1634]
… 7 more
Caused by: io.druid.java.util.common.ISE: Job[class io.druid.indexer.DetermineHashedPartitionsJob] failed!
at io.druid.indexer.JobHelper.runJobs(JobHelper.java:390) ~[druid-indexing-hadoop-0.12.1.3.0.0.0-1634.jar:0.12.1.3.0.0.0-1634]
at io.druid.indexer.HadoopDruidDetermineConfigurationJob.run(HadoopDruidDetermineConfigurationJob.java:91) ~[druid-indexing-hadoop-0.12.1.3.0.0.0-1634.jar:0.12.1.3.0.0.0-1634]
at io.druid.indexing.common.task.HadoopIndexTask$HadoopDetermineConfigInnerProcessing.runTask(HadoopIndexTask.java:325) ~[druid-indexing-service-0.12.1.3.0.0.0-1634.jar:0.12.1.3.0.0.0-1634]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_112]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_112]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_112]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:219) ~[druid-indexing-service-0.12.1.3.0.0.0-1634.jar:0.12.1.3.0.0.0-1634]
… 7 more
2019-06-20T10:14:00,959 INFO [task-runner-0-priority-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_hadoop_weatherReport_2019-06-20T10:13:39.104Z] status changed to [FAILED].
2019-06-20T10:14:00,962 INFO [task-runner-0-priority-0] io.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {
“id” : “index_hadoop_weatherReport_2019-06-20T10:13:39.104Z”,
“status” : “FAILED”,
“duration” : 9893

My spec:

{
“type”: “index_hadoop”,
“spec”: {
“dataSchema”: {
“dataSource”: “weatherReport”,
“parser”: {
“type”: “hadoopyString”,
“parseSpec”: {
“format”: “csv”,
“timestampSpec”: {
“column”: “timestamp”,
“format”: “auto”
},
“columns”: [“timestamp”,“vehicleId”,“vehicleType”,“manufactureId”,“adCode”,“cityChange”,“provinceChange”,“mileage”, “driveSeconds”,“maxSpeed”,“onlineSeconds”,“rAccCount”,“rDecCount”,“rTurnCount”,“condCode”,“temperature”,“humidity”,“pm25”,“airQuality”,“pop”],
“dimensionsSpec”: {
“dimensions”: [“vehicleId”,“manufactureId”,“vehicleType”,“adCode”,“condCode”,“temperature”,“humidity”,“pm25”,“airQuality”,“pop”]
}
}
},
“metricsSpec”: [
{“type”: “count”,“name”: “count”},
{“type”: “longSum”,“name”: “mileage”,“fieldName”: “mileage”},
{“type”: “longSum”,“name”: “driveSeconds”,“fieldName”: “driveSeconds”},
{“type”: “longSum”, “name”: “rAccCount”, “fieldName”: “rAccCount”},
{“type”: “longSum”, “name”: “rDecCount”, “fieldName”: “rDecCount”},
{“type”: “longSum”, “name”: “rTurnCount”, “fieldName”: “rTurnCount”},
{“type”: “longMax”, “name”: “maxSpeed”, “fieldName”: “maxSpeed”},
{“type”: “longSum”, “name”: “onlineSeconds”, “fieldName”: “onlineSeconds”},
{“type”: “longSum”, “name”: “cityChange”, “fieldName”: “cityChange”},
{“type”: “longSum”, “name”: “provinceChange”, “fieldName”: “provinceChange”}

        ],
        "granularitySpec": {
            "type": "uniform",
            "segmentGranularity": "DAY",
            "queryGranularity": "DAY",
            "rollup": true,
            "intervals" : ["2019-05-01/2019-06-01"]
        }
    },
    "ioConfig": {
        "type": "hadoop",
        "inputSpec": {
            "type": "static",
            "paths": "hdfs://testcluster/user/hue/vehicle-driving-weather-aggregate-data/report_201905/*,hdfs://testcluster/user/hue/vehicle-alarm-weather-aggregate-data/report_201905/*"
        }
    },
    "tuningConfig": {
        "type": "hadoop",
        "partitionsSpec": {
            "type": "hashed",
            "targetPartitionSize": 5000000
        },
        "forceExtendableShardSpecs" : true

    }
}

}

Hi Scoffi,
Is your hadoop cluster apache hadoop or Hortonworks HDP distribution or Cloudera CDH distribution?

Can you try by putting these in job properties section of your ingestion spec. This is just an template. You might have to change values according to your cluster?

Thanks for your help.I am very glad to receive your reply. And i had resolve it by adjusting the parameters of resources of MapReduce. We had limit the resource before. By the way, what’s your google email.

在 2019年6月22日星期六 UTC+8上午3:30:00,Siva Mannem写道: