Dynamically set JVM settings for indexer tasks


I couldn’t find documentation for this so I am not sure if it’s a supported feature. Is there a way to dynamically set the JVM heap and direct memory allocated for Kafka indexing processes using the supervisor spec? We have indexing processes that need different memory settings but right now they all use the same JVM settings that were configured in the properties file.



I think there is an undocumented way to set JVM settings directly on a per-task basis: https://github.com/apache/incubator-druid/issues/2320#issuecomment-173858106

I haven’t tried this myself but it looks like you could set the context in the supervisor spec and it would be inherited by the created tasks.

If your scaling needs can be categorized by datasource, you might be able to use worker affinity configs to tie tasks for certain datasources to specific MMs: http://druid.io/docs/latest/configuration/index.html#affinity