MiddleManager ignore request of unkown task

Hi,

Our MiddleManager can not run any task but keep ignoring request from overlord.

MiddleManager works fine after started. But with more and more unknown task to cancel, middleManager can not run any task, until we restart it.

Druid Version : 0.10.1

log:
2019-02-25T17:13:36,241 INFO [qtp908435478-155] io.druid.indexing.overlord.ForkingTaskRunner - Ignoring request to cancel unknown task: index_kafka_DL_WENZHOU_PROD_c948fd89f6fbb54_blkfipnl

2019-02-25T17:13:36,242 INFO [qtp908435478-207] io.druid.indexing.overlord.ForkingTaskRunner - Ignoring request to cancel unknown task: index_kafka_DL_CHANGXINGFADIAN_PROD_8819383baae781b_nenfgngm

2019-02-25T17:13:36,245 INFO [qtp908435478-68] io.druid.indexing.overlord.ForkingTaskRunner - Ignoring request to cancel unknown task: index_kafka_DL_ZHENHAI_RANQI_REDAIN_PROD_be4c68e126d121d_haefilci

2019-02-25T17:13:36,248 INFO [qtp908435478-74] io.druid.indexing.overlord.ForkingTaskRunner - Ignoring request to cancel unknown task: index_kafka_DL_ZHOUSHAN_PROD_da36ccc08c6376f_dgehajao

2019-02-25T17:13:36,254 INFO [qtp908435478-123] io.druid.indexing.overlord.ForkingTaskRunner - Ignoring request to cancel unknown task: index_kafka_DL_WENZHOURANJI_PROD_a306b7ce9a81324_akoiilfp

2019-02-25T17:13:36,256 INFO [qtp908435478-148] io.druid.indexing.overlord.ForkingTaskRunner - Ignoring request to cancel unknown task: index_kafka_DL_BEILUN_PROD_374024c13a880ac_fdfiljhd

2019-02-25T17:13:36,259 INFO [qtp908435478-172] io.druid.indexing.overlord.ForkingTaskRunner - Ignoring request to cancel unknown task: index_kafka_DL_ZHENHAI_PROD_2c5f6119e6b0608_iedkkbhi

2019-02-25T17:13:36,261 INFO [qtp908435478-85] io.druid.indexing.overlord.ForkingTaskRunner - Ignoring request to cancel unknown task: index_kafka_WEEK_DL_JIAHUA_PROD_0f627c9e47f3d8f_dbpnfgjd

2019-02-25T17:13:36,263 INFO [qtp908435478-43] io.druid.indexing.overlord.ForkingTaskRunner - Ignoring request to cancel unknown task: index_kafka_DL_JINHUARANJI_PROD_6baf21f0b0b64ae_omniopmj

2019-02-25T17:13:36,264 INFO [qtp908435478-176] io.druid.indexing.overlord.ForkingTaskRunner - Ignoring request to cancel unknown task: index_kafka_DL_TAIER_PROD_66e12554bf1df19_fjkhdcol

2019-02-25T17:14:36,205 INFO [qtp908435478-173] io.druid.indexing.overlord.ForkingTaskRunner - Ignoring request to cancel unknown task: index_kafka_INDICATOR_TIANRANQI_YUNXING_PROD_e28585086e48d90_poofbpcb

2019-02-25T17:14:36,210 INFO [qtp908435478-157] io.druid.indexing.overlord.ForkingTaskRunner - Ignoring request to cancel unknown task: index_kafka_DL_WENZHOU_PROD_c948fd89f6fbb54_blkfipnl

2019-02-25T17:14:36,212 INFO [qtp908435478-196] io.druid.indexing.overlord.ForkingTaskRunner - Ignoring request to cancel unknown task: index_kafka_DL_CHANGXINGFADIAN_PROD_8819383baae781b_nenfgngm

2019-02-25T17:14:36,214 INFO [qtp908435478-207] io.druid.indexing.overlord.ForkingTaskRunner - Ignoring request to cancel unknown task: index_kafka_DL_ZHENHAI_RANQI_REDAIN_PROD_be4c68e126d121d_haefilci

2019-02-25T17:14:36,217 INFO [qtp908435478-80] io.druid.indexing.overlord.ForkingTaskRunner - Ignoring request to cancel unknown task: index_kafka_DL_ZHOUSHAN_PROD_da36ccc08c6376f_dgehajao

2019-02-25T17:14:36,224 INFO [qtp908435478-68] io.druid.indexing.overlord.ForkingTaskRunner - Ignoring request to cancel unknown task: index_kafka_DL_WENZHOURANJI_PROD_a306b7ce9a81324_akoiilfp

2019-02-25T17:14:36,225 INFO [qtp908435478-155] io.druid.indexing.overlord.ForkingTaskRunner - Ignoring request to cancel unknown task: index_kafka_DL_BEILUN_PROD_374024c13a880ac_fdfiljhd

2019-02-25T17:14:36,228 INFO [qtp908435478-61] io.druid.indexing.overlord.ForkingTaskRunner - Ignoring request to cancel unknown task: index_kafka_DL_ZHENHAI_PROD_2c5f6119e6b0608_iedkkbhi

2019-02-25T17:14:36,231 INFO [qtp908435478-123] io.druid.indexing.overlord.ForkingTaskRunner - Ignoring request to cancel unknown task: index_kafka_WEEK_DL_JIAHUA_PROD_0f627c9e47f3d8f_dbpnfgjd

2019-02-25T17:14:36,232 INFO [qtp908435478-74] io.druid.indexing.overlord.ForkingTaskRunner - Ignoring request to cancel unknown task: index_kafka_DL_JINHUARANJI_PROD_6baf21f0b0b64ae_omniopmj

2019-02-25T17:14:36,234 INFO [qtp908435478-148] io.druid.indexing.overlord.ForkingTaskRunner - Ignoring request to cancel unknown task: index_kafka_DL_TAIER_PROD_66e12554bf1df19_fjkhdcol

2019-02-25T17:15:05,052 INFO [qtp908435478-85] io.druid.indexing.overlord.ForkingTaskRunner - Ignoring request to cancel unknown task: index_kafka_INDICATOR_TIANRANQI_YUNXING_PROD_e28585086e48d90_poofbpcb

2019-02-25T17:15:05,057 INFO [qtp908435478-211] io.druid.indexing.overlord.ForkingTaskRunner - Ignoring request to cancel unknown task: index_kafka_DL_WENZHOU_PROD_c948fd89f6fbb54_blkfipnl

2019-02-25T17:15:05,059 INFO [qtp908435478-169] io.druid.indexing.overlord.ForkingTaskRunner - Ignoring request to cancel unknown task: index_kafka_DL_CHANGXINGFADIAN_PROD_8819383baae781b_nenfgngm

2019-02-25T17:15:05,062 INFO [qtp908435478-176] io.druid.indexing.overlord.ForkingTaskRunner - Ignoring request to cancel

But if we restart the middleManager, no unknown task any more.

How can task become unknown? How to cancel the unknown task?

Thanks!

Wenny

Hi :

Can you confirm both zookeeper and meta DB are stable, and Druid’s connection to them are good?

Yes, we have 4 middleManagers, 2 of them are stable, and the other two have the problem.

在 2019年2月27日星期三 UTC+8上午7:42:10,Ming F写道:

We use mysql as the metadata DB. The metadata DB is used not only for druid, but also for other components, such as HDP, ambari, and so on.

在 2019年2月27日星期三 UTC+8下午3:43:15,sandp…@gmail.com写道: