No log was found for this task. The task may not exist, or it may not have begun running yet

Any thoughts on how to start debugging this issue ?. Logs on coordinator console display “No log was found for this task. The task may not exist, or it may not have begun running yet.”.

**Middle manager logs : **

16:15:13 druid[55068]: 2018-11-09T16:15:13,802 INFO [WorkerTaskMonitor] io.druid.indexing.worker.WorkerTaskMonitor - Submitting runnable for task[merge_test-requests_b240404cb87365dbd7dd183ba3b0d8c0d46f1032_2018-11-09T16:15:06.920Z]

16:15:13 druid[55068]: 2018-11-09T16:15:13,804 INFO [WorkerTaskMonitor] io.druid.indexing.worker.WorkerTaskMonitor - Affirmative. Running task [merge_test-requests_b240404cb87365dbd7dd183ba3b0d8c0d46f1032_2018-11-09T16:15:06.920Z]

16:15:13 druid[55068]: 2018-11-09T16:15:13,807 INFO [forking-task-runner-2] io.druid.indexing.overlord.ForkingTaskRunner - Running command: /test/java/jdk8/bin/java -cp /test/druid/conf/_common:/bms/tools/druid/conf/middleManager:/bms/tools/druid/current/lib/druid-indexing-hadoop-0.10.1.jar:/test/druid/current/lib/commons-io-2.5.jar:/test/druid/current/lib/disruptor-3.3.6.jar:/test/druid/current/lib/compress-lzf-1.0.3.jar:/test/druid/current/lib/jackson-core-asl-1.9.13.jar:/test/druid/current/lib/aopalliance-1.0.jar:/test/druid/current/lib/log4j-jul-2.5.jar:/test/druid/current/lib/jersey-core-1.19.3.jar:/test/druid/current/lib/maven-model-builder-3.1.1.jar:/test/druid/current/lib/spymemcached-2.12.3.jar:/test/druid/current/lib/avatica-server-1.9.0.jar:/test/druid/current/lib/curator-recipes-2.11.0.jar:/test/druid/current/lib/plexus-utils-3.0.15.jar:/test/druid/current/lib/aether-impl-0.9.0.M2.jar:/test/druid/current/lib/calcite-linq4j-1.11.0.jar:/test/druid/current/lib/javax.inject-1.jar:/test/druid/current/lib/jetty-io-9.3.19.v20170502.jar:/test/druid/current/lib/antlr4-runtime-4.5.1.jar:/test/druid/current/lib/druid-api-0.10.1.jar:/test/druid/current/lib/jackson-mapper-asl-1.9.13.jar:/test/druid/current/lib/derbynet-10.11.1.1.jar:/test/druid/current/lib/joda-time-2.9.9.jar:/test/druid/current/lib/jcl-over-slf4j-1.7.12.jar:/test/druid/current/lib/jersey-guice-1.19.3.jar:/test/druid/current/lib/irc-api-1.0-0014.jar:/test/druid/current/lib/bytebuffer-collections-0.10.1.jar:/test/druid/current/lib/lz4-1.3.0.jar:/test/druid/current/lib/druid-processing-0.10.1.jar:/test/druid/current/lib/asm-5.2.jar:/test/druid/current/lib/httpclient-4.5.1.jar:/test/druid/current/lib/emitter-0.4.5.jar:/test/druid/current/lib/javax.servlet-api-3.1.0.jar:/test/druid/current/lib/commons-lang3-3.2.jar:/test/druid/current/lib/icu4j-54.1.1.jar:/test/druid/current/lib/janino-2.7.6.jar:/

16:15:13 druid[55068]: .logging.log4j.jul.LogManager -Ddruid.indexer.task.baseTaskDir=/data/druid/tasks/work -Ddruid.host= -Ddruid.processing.buffer.sizeBytes=1610612736 -Ddruid.metadata.storage.connector.password=**** -Ddruid.emitter=noop -Duser.timezone=UTC -Ddruid.cache.sizeInBytes=10000000 -Dfile.encoding.pkg=sun.io -Ddruid.storage.storageDirectory=/auto/test/druid/data -Ddruid.selectors.coordinator.serviceName=coordinator -Ddruid.selectors.indexing.serviceName=overlord -Ddruid.port=8080 -Ddruid.server.http.numThreads=25 -Ddruid.worker.capacity=3 -Ddruid.cache.type=local -Ddruid.indexer.runner.javaCommand=/bms/tools/java/jdk8/bin/java -Ddruid.service=middleManager -Ddruid.metadata.storage.connector.user=druid -Ddruid.metadata.storage.type=mysql -Ddruid.metadata.storage.connector.connectURI=jdbc:mysql://localhost:3306/druid -Djava.io.tmpdir=/data/druid/tmp -Ddruid.extensions.loadList=[“mysql-metadata-storage”, “druid-kafka-indexing-service”, “druid-kafka-eight”] -Ddruid.processing.numThreads=2 -Ddruid.zk.service.host=localhost:2181 -Ddruid.indexer.logs.directory=/data/druid/tasks/logs -Dfile.encoding=UTF-8 -Ddruid.storage.type=local -Ddruid.indexer.logs.type=file -Ddruid.metrics.emitter.dimension.dataSource=test-requests -Ddruid.metrics.emitter.dimension.taskId=merge_test-requests_b240404cb87365dbd7dd183ba3b0d8c0d46f1032_2018-11-09T16:15:06.920Z -Ddruid.host= -Ddruid.port=8101 io.druid.cli.Main internal peon /data/druid/tasks/work/merge_test-requests_b240404cb87365dbd7dd183ba3b0d8c0d46f1032_2018-11-09T16:15:06.920Z/task.json /data/druid/tasks/work/merge_test-requests_b240404cb87365dbd7dd183ba3b0d8c0d46f1032_2018-11-09T16:15:06.920Z/7456a6f8-5942-47df-b45e-5858518f9a1d/status.json

16:15:13 druid[55068]: 2018-11-09T16:15:13,808 INFO [forking-task-runner-2] io.druid.indexing.overlord.TaskRunnerUtils - Task [merge_test-requests_b240404cb87365dbd7dd183ba3b0d8c0d46f1032_2018-11-09T16:15:06.920Z] location changed to [TaskLocation{host=’’, port=8101}].

16:15:13 druid[55068]: 2018-11-09T16:15:13,808 INFO [forking-task-runner-2] io.druid.indexing.overlord.TaskRunnerUtils - Task [merge_test-requests_b240404cb87365dbd7dd183ba3b0d8c0d46f1032_2018-11-09T16:15:06.920Z] status changed to [RUNNING].

16:15:13 druid[55068]: 2018-11-09T16:15:13,808 INFO [WorkerTaskMonitor] io.druid.indexing.worker.WorkerTaskMonitor - Updating task [merge_test-requests_b240404cb87365dbd7dd183ba3b0d8c0d46f1032_2018-11-09T16:15:06.920Z] announcement with location [TaskLocation{host=’’, port=8101}]

16:15:13 druid[55068]: 2018-11-09T16:15:13,808 INFO [forking-task-runner-2] io.druid.indexing.overlord.ForkingTaskRunner - Logging task merge_test-requests_b240404cb87365dbd7dd183ba3b0d8c0d46f1032_2018-11-09T16:15:06.920Z output to: /data/druid/tasks/work/merge_test-requests_b240404cb87365dbd7dd183ba3b0d8c0d46f1032_2018-11-09T16:15:06.920Z/log

16:15:19 druid[55068]: 2018-11-09T16:15:19,319 INFO [forking-task-runner-2-[merge_test-requests_b240404cb87365dbd7dd183ba3b0d8c0d46f1032_2018-11-09T16:15:06.920Z]] io.druid.indexing.overlord.ForkingTaskRunner - Process exited with status[0] for task: merge_test-requests_b240404cb87365dbd7dd183ba3b0d8c0d46f1032_2018-11-09T16:15:06.920Z

16:15:19 druid[55068]: 2018-11-09T16:15:19,319 INFO [forking-task-runner-2] io.druid.indexing.common.tasklogs.FileTaskLogs - Wrote task log to: /data/druid/tasks/logs/merge_test-requests_b240404cb87365dbd7dd183ba3b0d8c0d46f1032_2018-11-09T16:15:06.920Z.log

16:15:19 druid[55068]: 2018-11-09T16:15:19,319 INFO [forking-task-runner-2] io.druid.indexing.overlord.TaskRunnerUtils - Task [merge_test-requests_b240404cb87365dbd7dd183ba3b0d8c0d46f1032_2018-11-09T16:15:06.920Z] status changed to [FAILED].

16:15:19 druid[55068]: 2018-11-09T16:15:19,320 INFO [forking-task-runner-2] io.druid.indexing.overlord.ForkingTaskRunner - Removing task directory: /data/druid/tasks/work/merge_test-requests_b240404cb87365dbd7dd183ba3b0d8c0d46f1032_2018-11-09T16:15:06.920Z

16:15:19 druid[55068]: 2018-11-09T16:15:19,322 INFO [WorkerTaskMonitor] io.druid.indexing.worker.WorkerTaskMonitor - Job’s finished. Completed [merge_test-requests_b240404cb87365dbd7dd183ba3b0d8c0d46f1032_2018-11-09T16:15:06.920Z] with status [FAILED]

Thanks !

You may have the logging directory on a NFS or most likely there are some permission issues in the log directory so Druid process can’t read from it…