We have a big problem with the segments load balancing between historical nodes.
When we query on data, the first run has a good time but then the segments load balancing begin et the historical nodes spend so much time in “scanning segments”.
We can’t deal with this performance issue.
For example, a TopN answer in 1500ms in the first run and then the time is random between 2000 and 11000ms …
We use r3.2xlarge for historicals (8 cores for the hardware and 7 workers on the historical).
Here are the jvm.config for historical nodes:
8 vCPU - 61 Go Memory - 160 Go SSD
Any idea or help for this ?