Indexing task pendding with running status (BUG)

HI,

I am running two druid data server(historical node with a middle manager). they are running with 3 workers on each machine. the overlord panel shows me 2 tasks in running state and in which the second task is a replica of first. It also shows me one task in pending tasks list but if I check the status of the task then it shows as running state and surprisingly it also shown combinedly 3 worker capacity is being used. The pending task is there from last 10 hours. I am running tranquillity Kafka ingestion using hourly granularity.

**running tasks**

index_realtime_task_2018-02-22T18:00:00.000Z_0_0
2018-02-22T18:00:00.023Z
2018-02-22T18:00:00.027Z
server1
8100
-1

index_realtime_task_2018-02-22T18:00:00.000Z_0_1
2018-02-22T18:00:00.023Z
2018-02-22T18:00:00.042Z
server2
8100
-1

Showing 1 to 2 of 2 entries

FirstPrevious1NextLast

pending task

index_realtime_task_2018-02-22T06:00:00.000Z_0_1
2018-02-22T06:52:13.548Z
2018-02-22T06:52:13.548Z
null
-1
-1**
this pending task status**

{"task":"index_realtime_tasks_2018-02-22T06:00:00.000Z_0_1","status":{"id":"index_realtime_tasks_2018-02-22T06:00:00.000Z_0_1","status":"RUNNING","duration":-1}}

Remote workers

http
server1:8091
server1
3
0
1
[“task-2018-02-22T18:00:00.000Z-0000”]
[“index_realtime_task_2018-02-22T18:00:00.000Z_0_0”]
2018-02-22T18:16:07.925Z
null
http
server2:8091
server2
3
0
2
[“task-2018-02-22T06:00:00.000Z-0000”,“task-2018-02-22T18:00:00.000Z-0000”]
[“index_realtime_task_2018-02-22T18:00:00.000Z_0_1”,“index_realtime_task_2018-02-22T06:00:00.000Z_0_1”]
2018-02-22T18:16:05.953Z
null

Hi,
Which version of druid are you using ?

can you also share overlord logs and middlemanager logs for server2 for this duration ?

Hi Nishant, I am using Druid 0.11.0-iap2 with imply 2.4.3. I ahveset loglevel as Error only below is overlord.log error and the historical.log file only have a segment load fail issue of 29-12-2017. this issue arised when I restarted imply(druid) to change middle manager worker capacity from 2 to 3 . and in first restart, mysql blocked the host beacaue of multiple connection then I stoped imply(druid) and restarted it again after restarting mysql.

overloard.log

2018-02-22 06:38:17,274 Thread-41 ERROR Unable to register shutdown hook because JVM is shutting down. java.lang.IllegalStateException: Not started
at io.druid.common.config.Log4jShutdown.addShutdownCallback(Log4jShutdown.java:47)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.addShutdownCallback(Log4jContextFactory.java:273)
at org.apache.logging.log4j.core.LoggerContext.setUpShutdownHook(LoggerContext.java:256)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:216)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:145)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:41)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:182)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:103)
at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:43)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:42)
at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:29)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:253)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:265)
at org.apache.curator.utils.CloseableUtils.(CloseableUtils.java:33)
at org.apache.curator.ConnectionState.close(ConnectionState.java:119)
at org.apache.curator.CuratorZookeeperClient.close(CuratorZookeeperClient.java:227)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.close(CuratorFrameworkImpl.java:381)
at io.druid.curator.CuratorModule$1.stop(CuratorModule.java:98)
at io.druid.java.util.common.lifecycle.Lifecycle.stop(Lifecycle.java:335)
at io.druid.java.util.common.lifecycle.Lifecycle$1.run(Lifecycle.java:366)
at java.lang.Thread.run(Thread.java:748)

Thu Feb 22 06:54:37 UTC 2018 WARN: Establishing SSL connection without server’s identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and $
Thu Feb 22 06:54:38 UTC 2018 WARN: Establishing SSL connection without server’s identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and