Overlord doesn't delete finished tasks from storage

Hi everyone,

Context: Druid 0.13.0, Overlord storage.

Problem: We configured druid.indexer.storage.recentlyFinishedThreshold to 4 hours, but Overlord doesn’t delete finished tasks from storage. Therefore we have many completed tasks which cause Overlord UI to be unresponsive.

Overlord configuration:

druid.service=druid/production/overlord

druid.port=8090

druid.indexer.queue.startDelay=PT30S

druid.indexer.runner.type=remote

druid.indexer.storage.type=local

druid.server.http.numThreads=128

druid.indexer.storage.recentlyFinishedThreshold=PT4H

druid.monitoring.monitors=[“io.druid.java.util.metrics.JvmMonitor”]

``

Please ask if you need more info.

Thanks,

Shachar

Hi Shachar

Post the change, how many competed tasks do you see in the Overlord UI? Are they from the last 4 hours?

I hope the services are restarted after changing configurations across.

You may want to check the overlord system performance using vmstat/iostat.

Thanks & Rgds

Venkat

Hi Venkat,

I currently have 69.7K completed tasks in overlord UI. the “oldest” one is from 5 days ago.
We didn’t change this setting ever, it worked well for when we ran on druid 0.10 (the cluster wasn’t upgraded though, we created a new 0.13 cluster).

I did try to restart the overlord, it didn’t fix the issue.

Thanks for your help and time,

Shachar

Sachar,

You could backup and then clean up druid_tasks table in the metadb once.

Refer :
https://github.com/apache/incubator-druid/issues/5994

Thanks & Rgds

Venkat

We use local storage type, not sure if it is relevant.

Any help on this one, please?

Still struggling with this issue, any help will be appreciated.

Any kind of help will be appreciated.

Hi Shachar,

druid.indexer.storage.recentlyFinishedThreshold is just something that affects how the table is queried, it doesn’t actually make Druid delete older stuff. You could get away with manually deleting any record with a non-RUNNING status and an old enough created_date. Nothing bad should happen.

Hi Gian,

Thanks for your answer!

I use:

Druid.indexer.storage.type=local

How can I manually delete records?

Thanks,

Shachar

Ah got it. I’m not sure if you can. Restarting the overlord might do it, I think “local” is actually just in-memory. I would recommend using metadata store for this anyway instead of local.

You are right, local is in-memory.

I was hoping to find a way to delete it even though it’s in-memory.

Thanks,

Shachar