Adding new historical node does not rebalances existing data distribution

When we added a new historical node to our cluster it did not basically trigger any balancing for data present in existing across node.

Are there any manual steps for invoking this rebalance?

However new data ingestion does gets loaded into the new historical node.

We need automatic rebalance to address an unplanned spike in number of queries getting served and scaling horizontally by adding new historical nodes.

Regards,

Pushkar

Hey Pushkar,

Rebalancing should proceed automatically, albeit slowly. You can make it go faster by adjusting the max segments to move and replication throttle params in the coordinator console. By default they’re pretty low, to prevent flooding the cluster with segment moves whenever you add new nodes.

Thanks Gian.I do see it picking up slowly.Here are the parameter that i see in my coordinator console
maxSegmentsToMove

replicantLifetime

replicationThrottleLimit
10

We have typically ~14k segments in each of our historical with replication factor of two.

does changing these parameters to:

maxSegmentsToMove 100

replicantLifeTime 15

replicationThrottleLimit 200

Makes sense?or is there a correlation between these 3 parameters.

our coordinator is configured to run every minute through druid.coordinator.period=PT60S

Regards,

Pushkar

Yes, those parameters make sense. Although separately from that: unless you have TBs of data per historical, 14k segments is a bit on the high side, so you may want to look into reindexing or compaction.