Issue with Realtime

Hello Druid

We are facing following issue randomly:

Overlord console is showing that middle Manager peon capacity as 4 ( configured value) but when i did check randomly: Many middle managers are having 2 realtime nodes. Rest of the 2 are disappeared. Again until the next hour, system is going to loose data. Only after hour change again all 4 peons are going to get created. Until next hour, all the workers are trying to push data to missing realtime nodes and they loose data. Is it a known bug? Or any other solution to handle this?

Thanks

Bhaskar

Hi Bhaskar,
seems like some of your tasks might be occasionally failing.

do you see any tasks with status FAILED in overlord console ?

checking the task logs for the failed tasks will provide more details about the root cause of failure.

Also, make sure that you have enough memory to start a peon on your machine. We had this problem before and by looking at the log. It showed that the task couldn’t be started because of not enough memory. So, the task was started and failed.