I recently migrated our historical nodes to a new aws instance type, with more processing cores. I did this by bringing up the new instances, adding a rule to replicate data across all nodes, then stopped the older nodes.
This migration went smoothly. However afterwards, I changed the configuration of my historical nodes to match the new node types. One of the configuration properties, specifically druid.processing.numThreads, seems to cause the historical nodes to not be available for the coordinator nodes.
It had previously been set to 7, (old nodes had 8 cores), and the new configuration is set to 31 (new nodes have 31 cores). After this change goes in, the coordinator console view shows all datasources as unavailable. I also tried waiting 5 minutes and restarting the coordinator node but to no avail.
The interesting part is that if I change the value to something lower (e.g. 6) the coordinator node is fine and can still view the historical nodes. As soon as I move it above 7 (I tried 8), the historical nodes are foobar’d.
Is there any other configuration or setting that is tied to this that I am missing/forgetting?