Druid 0.12.0 - LockAcquireAction Exception

Hey everyone,

I have build the Druid 0.12.0 for HDP 2.6.0-235 from the source. Everything is working fine, even the first task to determine indexes and intervals completed successfully. But I’m getting the following the exception while the submitting for the second task:

2018-03-16T01:58:39,233 INFO [task-runner-0-priority-0] io.druid.indexing.common.actions.RemoteTaskActionClient - Submitting action for task[index_hadoop_mediated_wireless_2018-03-16T01:55:13.841Z] to overlord: [LockAcquireAction{lockType=EXCLUSIVE, interval=2018-03-04T00:00:00.000Z/2018-03-05T00:00:00.000Z, timeoutMs=300000}].
2018-03-16T01:58:39,234 INFO [task-runner-0-priority-0] io.druid.java.util.http.client.pool.ChannelResourceFactory - Generating: http://n2pl-pa-hdd230:8090
2018-03-16T01:58:39,253 WARN [task-runner-0-priority-0] io.druid.indexing.common.actions.RemoteTaskActionClient - Exception submitting action for task[index_hadoop_mediated_wireless_2018-03-16T01:55:13.841Z]
io.druid.java.util.common.IOE: Scary HTTP status returned: 500 Server Error. Check your overlord logs for exceptions.
	at io.druid.indexing.common.actions.RemoteTaskActionClient.submit(RemoteTaskActionClient.java:95) [druid-indexing-service-0.12.0.jar:0.12.0]
	at io.druid.indexing.common.task.HadoopIndexTask.run(HadoopIndexTask.java:210) [druid-indexing-service-0.12.0.jar:0.12.0]
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:444) [druid-indexing-service-0.12.0.jar:0.12.0]
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:416) [druid-indexing-service-0.12.0.jar:0.12.0]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_144]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_144]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_144]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
2018-03-16T01:58:39,256 INFO [task-runner-0-priority-0] io.druid.indexing.common.actions.RemoteTaskActionClient - Will try again in [PT5.038S].

It will be great if anyone can help me with the possible causes for exception in LockAcquireAction.

Hi Akul,

Have you had a chance to check overlord logs? This error means that the overlord failed to process lockAcquireAction. The detailed reason should be logged in overlord logs.

Jihoon

2018년 3월 15일 (목) 오후 7:35, Akul Narang akul.narang1011@gmail.com님이 작성:

Hey Jihoon,

I got nothing in the overlord logs. Just a failure status for task.

What can be the possibilities for this error or what kind of lock is it trying to acquire ?
I’m using HDFS as deep storage. Is it on hdfs or local filesystem or Mysql ?

Any suggestions would be appreciated.