Is there a way to add callback hooks to Druid middleManager tasks

I want to know is there a way Druid can call back an endpoint I specify after ingestion is done.

We have a spark based etl pipeline at the end of which, we upscale middleManager nodes (our whole infra is on AWS so we have a lambda to set different ASG parameters). I want to downscale after the ingestion is done.

I already have a polling based solution. I wanted to know if there is callback way to avoid polling. Please let me know if there are any ideas around it.

Hi Karthik - you can use AWS auto scaling with your cluster in Druid. I will look for some sample config and send soon.

Karthik,

Search for autoscaling in the below configuration reference. This can be enabled for your middle managers.

https://druid.apache.org/docs/latest/configuration/index.html

Eric

Eric Graham

Solutions Engineer -** **Imply

**cell: **303-589-4581

email: eric.graham@imply.io

www.imply.io