How to avoid historical always trying to load data

Hi Druid Guru,

I setup a druid cluster internally. It contains two historical nodes – r4.8x. It loaded two month data, around 800G. The reason why I want to use this kind of ec2 instance is I want the data always in Memory.

However, I found historical node always want to load some segments from s3. As each node has more than 1T ssd. Why this happened?

When the historical node try to load data from s3, it cannot serve queries especially those heavy queries.

Does any guru know how to avoid this? Thanks in advance.

2017-06-27T07:54:57,589 INFO [ZkCoordinator-0] io.druid.server.coordination.ZkCoordinator - Completed request [LOAD: xxxxxx_2017-05-16T17:00:00.000Z_2017-05-16T18:00:00.000Z_2017-06-13T05:55:01.305Z]

Hi Gabriel,
With r4.8xlarge instances the impact of segment loading should be minimal,

I wonder if you have some other config issue causing the slowness?

Can you share your node configs as well ?

Also, the continuous segment balancing is a known issue - https://github.com/druid-io/druid/issues/3354

One workaround is to set maxSegmentsToMove in coordinator dynamic properties to a very low value to throttle segment balancing across nodes.

Thank you for your reply. I list the configuration files below. Could you please help me to find whether there is anything wrong? Thanks in advance.

The runtime.properties for coordinator

druid.service=druid/coordinator

druid.port=8081

druid.coordinator.startDelay=PT30S

druid.coordinator.period=PT30S

jvm.config

-server

-Xms10g

-Xmx10g

-Duser.timezone=UTC

-Dfile.encoding=UTF-8

-Djava.io.tmpdir=var/tmp

-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager

-Dderby.stream.error.file=var/druid/derby.log

runtime.properties of historical node

druid.service=druid/historical

druid.port=8083

druid.server.http.numThreads=100

druid.processing.buffer.sizeBytes=268435456

druid.processing.numThreads=31

druid.segmentCache.locations=[{“path”:“var/druid/segment-cache”,“maxSize”:3830000000000}]

druid.server.maxSize=3830000000000

druid.historical.cache.useCache=true

druid.historical.cache.populateCache=true

druid.segmentCache.dropSegmentDelayMillis=300000

druid.cache.type=local

druid.server.tier=traffic_data_hot

druid.cache.sizeInBytes=16000000000

jvm.config of historical node

-server

-Xms80g

-Xmx80g

-XX:NewSize=10g

-XX:MaxNewSize=10g

-XX:+UseConcMarkSweepGC

-XX:+PrintGCDetails

-XX:+PrintGCTimeStamps

-XX:+HeapDumpOnOutOfMemoryError

-Dcom.sun.management.jmxremote.port=17071

-Dcom.sun.management.jmxremote.authenticate=false

-Dcom.sun.management.jmxremote.ssl=false

-XX:MaxDirectMemorySize=16384m

-Duser.timezone=UTC

-Dfile.encoding=UTF-8

-Djava.io.tmpdir=var/tmp

-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager