Is there a way to specify the middle managers that a specific datasource ingestion will use?

We are trying to add a new datasource to our existing druid cluster, but we do not want the new datasource ingestion to cannibalize the resources for the existing datasource ingestion.

Is there a way to limit the middle-managers can be used for a specific datasource ingestion? The ingestion will be using Kinesis.

Hey James,
By specifying workerCategorySpec , you can assign tasks to run on different categories of MiddleManagers based on the tasks’ taskType and dataSource name

2 Likes

Thanks a lot vsevall, I was able to get this to work by using selectStrategy - affinityConfig. For some reason the workerCategorySpec wasn’t working, but I think that affinitiyConfig might be a better option in any case. For anyone wondering, you can submit the selectStrategy by hitting the gear at the top right of the ui and selecting " Overlord dynamic config"