Issue with indexing service using the Hadoop cluster

Hey folks, we are facing the following issue when we try to index data against our Hadoop (hadoop-2.4.0.2.1.2.0-402) cluster and was wondering if anybody could help:

2015-08-25T01:53:32,828 ERROR [task-runner-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running task[HadoopIndexTask{id=index_hadoop_dtmrt_pci_byr_ltime_2015-08-25T01:53:18.757Z, type=index_hadoop, dataSource=dtmrt_pci_byr_ltime}]

java.lang.RuntimeException: java.lang.reflect.InvocationTargetException

at com.google.api.client.repackaged.com.google.common.base.Throwables.propagate(Throwables.java:160) ~[google-http-client-1.15.0-rc.jar:?]

at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:132) ~[druid-indexing-service-0.8.0.jar:0.8.0]

at io.druid.indexing.common.task.HadoopIndexTask.run(HadoopIndexTask.java:188) ~[druid-indexing-service-0.8.0.jar:0.8.0]

at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:235) [druid-indexing-service-0.8.0.jar:0.8.0]

at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:214) [druid-indexing-service-0.8.0.jar:0.8.0]

at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_40]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_40]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_40]

at java.lang.Thread.run(Thread.java:724) [?:1.7.0_40]

Caused by: java.lang.reflect.InvocationTargetException

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.7.0_40]

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[?:1.7.0_40]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.7.0_40]

at java.lang.reflect.Method.invoke(Method.java:606) ~[?:1.7.0_40]

at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:129) ~[druid-indexing-service-0.8.0.jar:0.8.0]

… 7 more

Caused by: java.lang.RuntimeException: org.apache.hadoop.security.AccessControlException: Permission denied: user=tim, access=WRITE, inode="/":hdfs:hdfs:drwxr-xr-x

at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:265)

at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:251)

at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:232)

at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:176)

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5513)

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5495)

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:5469)

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2288)

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2241)

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2194)

at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:526)

at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:354)

at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)

at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)

at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)

at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)

at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:415)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)

at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

at io.druid.indexer.IndexGeneratorJob.run(IndexGeneratorJob.java:192) ~[druid-indexing-hadoop-0.8.0.jar:0.8.0]

at io.druid.indexer.JobHelper.runJobs(JobHelper.java:179) ~[druid-indexing-hadoop-0.8.0.jar:0.8.0]

at io.druid.indexer.HadoopDruidIndexerJob.run(HadoopDruidIndexerJob.java:96) ~[druid-indexing-hadoop-0.8.0.jar:0.8.0]

at io.druid.indexing.common.task.HadoopIndexTask$HadoopIndexGeneratorInnerProcessing.runTask(HadoopIndexTask.java:241) ~[druid-indexing-service-0.8.0.jar:0.8.0]

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.7.0_40]

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[?:1.7.0_40]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.7.0_40]

at java.lang.reflect.Method.invoke(Method.java:606) ~[?:1.7.0_40]

at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:129) ~[druid-indexing-service-0.8.0.jar:0.8.0]

… 7 more

``

The following are the different configs and commands we are doing:

Index task command:

curl -X ‘POST’ -H ‘Content-Type:application/json’ -d @dtmrt_byr_ltime.json localhost:8080/druid/indexer/v1/task

``

Overlord command:

java -Xmx2g -Duser.timezone=UTC -Dfile.encoding=UTF-8 -classpath lib/*:config/overlord:config/_common:$(hadoop classpath) io.druid.cli.Main server overlord

``

Middle Manager command:

java -Xmx2g -Duser.timezone=UTC -Dfile.encoding=UTF-8 -classpath lib/*:config/_common:$(hadoop classpath) io.druid.cli.Main server middleManager

``

common.runtime.properties (excerpt):

druid.storage.type=hdfs

druid.storage.storageDirectory=hdfs:///tmp/pci/

``

Index spec JSON:

{

“type”: “index_hadoop”,

“spec”: {

“dataSchema”: {

“dataSource”: “dtmrt_pci_byr_ltime”,

“parser”: {

“type”: “string”,

“parseSpec”: {

“format”: “tsv”,

“delimiter”: “\t”,

“timestampSpec”: {

“column”: “segment_date”,

“format”: “yyyy-MM-dd”

},

“dimensionsSpec”: {

“dimensions”: [

“rcvr_id”,

“sndr_id”,

“indy_key”,

“cntry_code”,

“prnt_merch_key”,

“txn_segments”

],

“dimensionExclusions”: ,

“spatialDimensions”:

},

“columns”: [

“segment_date”,

“rcvr_id”,

“sndr_id”,

“indy_key”,

“cntry_code”,

“prnt_merch_key”,

“txn_segments”,

“lf_txn”,

“lf_tpv”,

“buyer_lt_months”

]

}

},

“metricsSpec”: [

{

“type”: “doubleSum”,

“name”: “lf_txn”,

“fieldName”: “lf_txn”

},

{

“type”: “doubleSum”,

“name”: “lf_tpv”,

“fieldName”: “lf_tpv”

},

{

“type”: “doubleSum”,

“name”: “buyer_lt_months”,

“fieldName”: “buyer_lt_months”

},

{

“type”: “hyperUnique”,

“name”: “sndr_id_hunq”,

“fieldName”: “sndr_id”

}

],

“granularitySpec”: {

“type”: “uniform”,

“segmentGranularity”: “DAY”,

“queryGranularity”: “NONE”,

“intervals”: [

“2015-06-01/2015-06-30”

]

}

},

“ioConfig”: {

“type”: “hadoop”,

“inputSpec”: {

“type”: “static”,

“paths”: “hdfs:///tmp/pci/dtmrt_bry_ltime_t/*”

}

},

“tuningConfig”: {

“type”: “hadoop”

}

}

}

``

Thanks,

Tim

The error is here:

Caused by: java.lang.RuntimeException: org.apache.hadoop.security.AccessControlException: Permission denied: user=tim, access=WRITE, inode="/":hdfs:hdfs:drwxr-xr-x

You’ll need to fix your permissions on HDFS.

But why is it looking at the root “/” directory? Is this a configuration setting somewhere?

Druid will try to create both of these directories on HDFS if they do not already exist. If the user you’re running as does not have permission to create them, you’ll have to either grant the user permission or create them some other way before running the Druid indexer.

  • druid.indexer.task.hadoopWorkingPath (by default, /tmp/druid-indexing)

  • druid.storage.storageDirectory (if using HDFS deep storage)

So, the /tmp directory already exists.
The storageDirectory is also pointing to /tmp.

What I do not understand is what directory/file is it trying to create on the root “/” directory?

If all your paths are located under /tmp, and /tmp exists, nothing should be trying to create things under the root. Can you attach full logs from the task that failed? There might be something interesting in there somewhere.