Hi,
I am using the Kafka Indexing Service and I am noticing abnormal times when the tasks transition to persist stage.
taskDuration for reading is set to 30 Minutes and on an average the total task duration is ~ 50 minutes.
If anyone has ideas on why the persists take such a long time. What metrics should I be looking to understand this?
I am not using any SSDs and all druid services(0.9.2.1) are on the same box, 24 Core, 128 GB RAM
Also, during certain phases, realtime queries take a really long time or completely timeout. Query older data seems to work well, but the real time seems to be unavailable for a while.
Could this be caused by the long persist periods I mentioned earlier?
Any help is much appreciated.
Thanks,
Ankush