Indexing task rejected

imply 2.5.11, only one 4 core 16G node

Error log from middlemanager:

2018-06-01T12:27:23,771 ERROR [forking-task-runner-15] com.google.common.util.concurrent.ExecutionList - RuntimeException while executing runnable com.google.common.util.concurrent.Futures$4@4f5c9bc8 with executor com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService@1a22baeb

java.util.concurrent.RejectedExecutionException: Task io.druid.indexing.worker.WorkerTaskManager$$Lambda$37/1407054970@15379431 rejected from java.util.concurrent.ThreadPoolExecutor@1430fb4[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 223]

at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) ~[?:1.8.0_171]

at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) ~[?:1.8.0_171]

at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) ~[?:1.8.0_171]

at java.util.concurrent.Executors$DelegatedExecutorService.execute(Executors.java:668) ~[?:1.8.0_171]

at io.druid.indexing.worker.WorkerTaskManager.submitNoticeToExec(WorkerTaskManager.java:179) ~[druid-indexing-service-0.12.0-iap10.jar:0.12.0-iap10]

at io.druid.indexing.worker.WorkerTaskManager.access$000(WorkerTaskManager.java:77) ~[druid-indexing-service-0.12.0-iap10.jar:0.12.0-iap10]

at io.druid.indexing.worker.WorkerTaskManager$2.onSuccess(WorkerTaskManager.java:243) ~[druid-indexing-service-0.12.0-iap10.jar:0.12.0-iap10]

at io.druid.indexing.worker.WorkerTaskManager$2.onSuccess(WorkerTaskManager.java:239) ~[druid-indexing-service-0.12.0-iap10.jar:0.12.0-iap10]

at com.google.common.util.concurrent.Futures$4.run(Futures.java:1181) ~[guava-16.0.1.jar:?]

at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) ~[guava-16.0.1.jar:?]

at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) [guava-16.0.1.jar:?]

at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) [guava-16.0.1.jar:?]

at com.google.common.util.concurrent.ListenableFutureTask.done(ListenableFutureTask.java:91) [guava-16.0.1.jar:?]

at java.util.concurrent.FutureTask.finishCompletion(FutureTask.java:384) [?:1.8.0_171]

at java.util.concurrent.FutureTask.set(FutureTask.java:233) [?:1.8.0_171]

at java.util.concurrent.FutureTask.run(FutureTask.java:274) [?:1.8.0_171]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]

2018-06-01T12:27:23,772 INFO [Thread-52] io.druid.indexing.overlord.ForkingTaskRunner - Finished stopping in 1,201ms.

2018-06-01T12:27:23,772 INFO [Thread-52] io.druid.indexing.worker.WorkerTaskManager - Stopped WorkerTaskManager.

2018-06-01T12:27:23,772 INFO [Thread-52] io.druid.indexing.worker.WorkerTaskMonitor - Stopped WorkerTaskMonitor.

2018-06-01T12:27:23,773 INFO [Thread-52] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.discovery.DruidLeaderClient.stop()] on object[io.druid.discovery.DruidLeaderClient@3fa7df1].

2018-06-01T12:27:23,773 INFO [Thread-52] io.druid.discovery.DruidLeaderClient - Stopped.

2018-06-01T12:27:23,773 INFO [Thread-52] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.curator.discovery.ServerDiscoverySelector.stop() throws java.io.IOException] on object[io.druid.curator.discovery.ServerDiscoverySelector@3596b249].

2018-06-01T12:27:23,776 INFO [Thread-52] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider.stop()] on object[io.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider@342e690b].

2018-06-01T12:27:23,776 INFO [Thread-52] io.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider - stopping

2018-06-01T12:27:23,777 INFO [Thread-52] io.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider - stopped

2018-06-01T12:27:23,777 INFO [Thread-52] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.java.util.http.client.NettyHttpClient.stop()] on object[io.druid.java.util.http.client.NettyHttpClient@de81be1].

2018-06-01T12:27:23,787 INFO [Thread-52] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.indexing.worker.WorkerCuratorCoordinator.stop() throws java.lang.Exception] on object[io.druid.indexing.worker.WorkerCuratorCoordinator@483b0690].

2018-06-01T12:27:23,787 INFO [Thread-52] io.druid.indexing.worker.WorkerCuratorCoordinator - Stopping WorkerCuratorCoordinator for worker[iZuf66td8os35tyt69buz9Z:8091]

2018-06-01T12:27:23,787 INFO [Thread-52] io.druid.curator.announcement.Announcer - Stopping announcer

2018-06-01T12:27:23,788 INFO [Thread-52] io.druid.curator.announcement.Announcer - unannouncing [/druid/indexer/announcements/iZuf66td8os35tyt69buz9Z:8091]

2018-06-01T12:27:23,791 INFO [Thread-52] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.storage.hdfs.HdfsStorageAuthentication.stop()] on object[io.druid.storage.hdfs.HdfsStorageAuthentication@1f2f0109].

2018-06-01T12:27:23,791 INFO [Thread-52] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.java.util.metrics.MonitorScheduler.stop()] on object[io.druid.java.util.metrics.MonitorScheduler@16736040].

2018-06-01T12:27:23,791 INFO [Thread-52] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.java.util.emitter.service.ServiceEmitter.close() throws java.io.IOException] on object[ServiceEmitter{serviceDimensions={service=druid/middlemanager, host=iZuf66td8os35tyt69buz9Z:8091, version=0.12.0-iap10}, emitter=LoggingEmitter{log=Logger{name=[io.druid.java.util.emitter.core.LoggingEmitter], class[class org.apache.logging.slf4j.Log4jLogger]}, level=INFO}}].

2018-06-01T12:27:23,791 INFO [Thread-52] io.druid.java.util.emitter.core.LoggingEmitter - Close: started [false]

2018-06-01T12:27:23,791 INFO [Thread-52] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.initialization.Log4jShutterDownerModule$Log4jShutterDowner.stop()] on object[io.druid.initialization.Log4jShutterDownerModule$Log4jShutterDowner@4b0bd001].

Heap

par new generation total 19648K, used 5907K [0x00000000fc000000, 0x00000000fd550000, 0x00000000fd550000)

eden space 17472K, 33% used [0x00000000fc000000, 0x00000000fc5c4fc8, 0x00000000fd110000)

from space 2176K, 0% used [0x00000000fd330000, 0x00000000fd330000, 0x00000000fd550000)

to space 2176K, 0% used [0x00000000fd110000, 0x00000000fd110000, 0x00000000fd330000)

concurrent mark-sweep generation total 43712K, used 29650K [0x00000000fd550000, 0x0000000100000000, 0x0000000100000000)

Metaspace used 56345K, capacity 57428K, committed 57752K, reserved 1099776K

class space used 6984K, capacity 7248K, committed 7344K, reserved 1048576K

Here is the middelmanager config:

jvm.config

-server

-Xms64m

-Xmx64m

-XX:+UseConcMarkSweepGC

-XX:+PrintGCDetails

-XX:+PrintGCTimeStamps

-Duser.timezone=UTC

-Dfile.encoding=UTF-8

-Djava.io.tmpdir=var/tmp

-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager

runtime.config

druid.service=druid/middlemanager

druid.port=8091

Number of tasks per middleManager

druid.worker.capacity=20

Task launch parameters

druid.indexer.runner.javaOpts=-server -Xms512m -Xmx512m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager

druid.indexer.task.baseTaskDir=var/druid/task

druid.indexer.task.restoreTasksOnRestart=true

HTTP server threads

druid.server.http.numThreads=40

Processing threads and buffers

druid.processing.buffer.sizeBytes=100000000

druid.processing.numMergeBuffers=2

druid.processing.numThreads=2

druid.processing.tmpDir=var/druid/processing

Hadoop indexing

druid.indexer.task.hadoopWorkingPath=var/druid/hadoop-tmp

druid.indexer.task.defaultHadoopCoordinates=[“org.apache.hadoop:hadoop-client:2.7.3”]