Cannot see the logs of FAILED Task

During data ingestion from kafka, some of the tasks are in FAILED status, but when I try to see the log of why they failed to debug I get the following error: Request failed with status code 404. I have accessed inside the container to see the logs of that task, but I don’t see any folder for that ID the tasks that have failed, which happens with the tasks that have been successful.

Affected Version

apache/druid:0.21.1

Description

Please include as much detailed information about the problem as possible.
Docker implementation using containers

  • 3 zookeeper nodes using BITNAMI image
  • 2 coordinators nodes using micro HW profile
  • 2 broker nodes using micro HW profile
  • 2 historical nodes using micro HW profile
  • 3 middlemanager nodes using small HW profile
  • Steps to reproduce the problem: Ingest data from kafka into druid and see the logs in FAILED task using docker
  • The error message or stack traces encountered: Request failed with status code 404
  • Any debugging that you have already done: Acess the OS system and analyse the folders of the task. There is no folder for failed tasks (it can be found for SUCESS task).

First check is from someone who knows more than me (!)

You might want to have a look in your deep storage to see if the druid.indexer.logs.directory is configured correctly and accessible to all hosts

A second one is to note that the logs actually appear by calling an API on the task.

The API that gets called to get a log on the console is:

/druid/indexer/v1/task/{taskId}/log

So 404 is possibly indicating that the task itself never got spun up (literally “I couldn’t find the task”).

Maybe check in the Overlord logs?

I have not detected any failed tasks in the last two days to be able to check it properly. I will keep you informed, and thank you very much for your quick response.

Also, what seems strange to me is that it does detect the logs of the tasks that have gone correctly and not of the tasks that fail.Screenshot_27

One of my docs PRs just got merged into 0.22 (yay!) — this may help?

Hi!. Sorry for the delay in replying, but during this week I have had not enough time to migrate the data platform to your version and do all the necessary tests. I hope to do it next week, I will keep you informed! Thank you very much for the progress