How to told MiddleManager to use whole server while Hadoop indexing, not just 1 core?


I have r3.8xlarge instance for Druid MiddleManager, but while indexing worker using just 1-2 cores and few of memory:

reserved: 47,705g, actual: 2,722g cpu load: 114,6%

load average: 0,61, 0,91, 0,63

Tasks: 527 total, 1 running, 526 sleeping, 0 stopped, 0 zombie

%Cpu(s): 5,0 us, 0,2 sy, 0,0 ni, 94,8 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st

How to get to use all server resources for map reducers? I have 32 cores and 230Gb so it would be nice to use maximum of server power.

Resources for peons

druid.indexer.runner.javaOpts=-server -Xmx32g -XX:+UseG1GC -XX:MaxGCPauseMillis=100 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps

Peon properties[“com.metamx.metrics.JvmMonitor”][{“path”: “var/druid/segment_cache”, “maxSize”: 0}]



You can use: (per task num threads)
druid.worker.capacity=9 (number of tasks)

Fangjin, I also posted a very similar question in:!topic/druid-user/lQA-rpMy9p4

In Max’s config, he already has, I’ve also tried this config, does not seem MM runing peons in parallel, still only one Peon and same runtime overall = 20