Task failed on the overlord and logs are empty.

Hello,

I’m new on druid, i set up a cluster with ec2 instances with the following roles:

  • Zookeeper

  • Mysql (Metadata storage)

  • Overlord

  • Middle Manager

  • Coordinator

  • Broker

  • Historical

The communication between all nodes and Zookeeper / Mysql seems to work.

So now my cluster is up, i try to submit a task to the Overlord to have some data !

Here is my task:

{

“type” : “index_hadoop”,

“spec” : {

“dataSchema” : {

“dataSource” : “druid”,

“parser” : {

“type” : “hadoopyString”,

“parseSpec” : {

“format” : “tsv”,

“timestampSpec” : {

“column” : “dateenr”,

“format” : “yyyyMMdd”

},

“columns” : [],

“delimiter”:"|",

“dimensionsSpec” : {

“dimensions”: [],

“dimensionExclusions” : ,

“spatialDimensions” :

}

}

},

“metricsSpec” : [

{

“type” : “count”,

“name” : “count”

}

],

“granularitySpec” : {

“type” : “uniform”,

“segmentGranularity” : “DAY”,

“queryGranularity” : “NONE”,

“intervals” : [ “2016-04-01/2016-04-02” ]

}

},

“ioConfig” : {

“type” : “hadoop”,

“inputSpec” : {

“type” : “static”,

“paths” : “s3n://MyBucket/KeyToMyTsvGzip”

}

},

“tuningConfig” : {

“type”: “hadoop”

}

}

}

When i try to submit these one to the overlord, the task is created and instantly fail.

I can handle the fact that my task fail, i was pretty prepared for that, but the fact is i have no logs to help me debug.

I set up the log lines for s3 in my overlord and in my middle manager.

Maybe the problem could come from the overlord:

When the overord start:

2016-05-23T15:34:27,437 WARN [pool-12-thread-1] io.druid.indexing.overlord.RemoteTaskRunner - No worker selections strategy set. Using default.

When a task fail and i try to access the logs:

2016-05-23T15:36:40,544 WARN [qtp310350177-33] io.druid.indexing.overlord.http.OverlordResource - Failed to stream log for task index_hadoop_druid_2016-05-23T15:34:27.355Z

java.io.IOException: Failed to stream logs from: Druid/Logs/index_hadoop_druid_2016-05-23T15:34:27.355Z/log

[…]

I don’t know if my configuration is bad or if it is my task, Any ideas ?

Thanks,

Ben

Here’s my command for submitting the task:
curl -X ‘POST’ -H ‘Content-Type:application/json’ -d @TaskTestDruid.json http://:8090/druid/indexer/v1/task

My task is in file “TaskTestDruid.json”.

Hello,

My config was bad on the common.runtime.properties file.

I had my s3 access key and password surrounded by quotes, it was wrong ! :slight_smile:

Hi Benjamin, glad to see you figured it out.

Looking at your cluster, you can probably colocate many of the services together so use less hardware.

Yeah but in some days/weeks, we want to use it in the company so we’ll need more power than that i think but i did not find any “recommended” configuration so i really don’t now …
If you have any advice i’m totally open !

You can take a look at how Imply does colocation:
http://imply.io/docs/latest/