Context: Druid 0.13.0, Overlord storage.
Problem: We configured druid.indexer.storage.recentlyFinishedThreshold to 4 hours, but Overlord doesn’t delete finished tasks from storage. Therefore we have many completed tasks which cause Overlord UI to be unresponsive.
Please ask if you need more info.
Post the change, how many competed tasks do you see in the Overlord UI? Are they from the last 4 hours?
I hope the services are restarted after changing configurations across.
You may want to check the overlord system performance using vmstat/iostat.
Thanks & Rgds
I currently have 69.7K completed tasks in overlord UI. the “oldest” one is from 5 days ago.
We didn’t change this setting ever, it worked well for when we ran on druid 0.10 (the cluster wasn’t upgraded though, we created a new 0.13 cluster).
I did try to restart the overlord, it didn’t fix the issue.
Thanks for your help and time,
You could backup and then clean up druid_tasks table in the metadb once.
Thanks & Rgds
We use local storage type, not sure if it is relevant.
Any help on this one, please?
Still struggling with this issue, any help will be appreciated.
Any kind of help will be appreciated.
druid.indexer.storage.recentlyFinishedThreshold is just something that affects how the table is queried, it doesn’t actually make Druid delete older stuff. You could get away with manually deleting any record with a non-RUNNING status and an old enough created_date. Nothing bad should happen.
Thanks for your answer!
How can I manually delete records?
Ah got it. I’m not sure if you can. Restarting the overlord might do it, I think “local” is actually just in-memory. I would recommend using metadata store for this anyway instead of local.
You are right, local is in-memory.
I was hoping to find a way to delete it even though it’s in-memory.