Hadoop Index task is starting in Local Mode

For some reason I am not able to start hadoop index task on Hadoop cluster. The task starts and successfully finishes but in local mode.

Please see the attached are my spec and log files from the coordinator

task invocation:

curl -X ‘POST’ -H ‘Content-Type:application/json’ -d @taskspec.json :8090/druid/indexer/v1/task

Spec File: Attached

Log File : Attached

both druid.extensions.hadoopDependenciesDir and “hadoopDependencyCoordinates” are pointing to hadoop-cleint:2.7.3

I am not sure what I am missing to enable it to run on hadoop cluster. Any pointers in resolving this would be greatly appreciated



tasklog.log (523 KB)

taskspec.json (2.06 KB)

Hi, would you check that your hadoop-related configurations are properly in your classpath?
The simplest way to ensure those files are in your classpath is to copy *-site.xml files to ${DRUID_ROOT}/conf/druid/_common.


2017년 5월 4일 (목) 오전 10:45, Mallik Bathula mbathula@gmail.com님이 작성:

Thx Jihoon,
I figured it out few minutes ago , it turns out that my mapred-site.xml was not synced properly.

However I ran into a new issue

  1. “java.io.FileNotFoundException: /hadoop/yarn/log/application_1464036814491_0009/container_e04_1464036814491_0009_01_000001 (Is a directory)”
  2. and was able to resolve it by restarting the coordinator and overload with hadoop/yarn class paths