Questions about production config of middel manager node

Hi,
I have some questions about the production config of middel manager node.

As it described.(http://druid.io/docs/0.8.2/configuration/production-cluster.html)

druid.indexer.runner.javaOpts=-server -Xmx3g -XX:+UseG1GC -XX:MaxGCPauseMillis=100 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps

druid.indexer.fork.property.druid.processing.buffer.sizeBytes=536870912

druid.indexer.fork.property.druid.processing.numThreads=2

druid.worker.capacity=9

We chose the r3.8xlarge type of ec whose memory is 244GB and cores is 32.

The setting capacity is 9 and num threads is 2, the total of used cores is 9 * 2 = 18. What are the remaining cores(14) used for? Is them used for computing indexing for realtime ingestion?

And there are remaining memory(244GB - 9 * 2 * 512m = 235GB), Are these remaining memory used for loading segments which haven’t been persisted to historical node?

Sorry, the remaining memory is (244GB - 9 * 2 * 512m - 9 * 3g = 208GB)

在 2015年12月1日星期二 UTC+8下午12:10:14,luo…@conew.com写道:

The short answer is they are not used and we choose an example server that is overkill to keep things simple. With an r3.8xlarge, you can use more threads per task and have more tasks on the server.

Thanks Fangjin. Is my comprehension right? about remaining cores and memory?

在 2015年12月4日星期五 UTC+8下午3:30:21,Fangjin Yang写道:

Hi,

general answer to your question is hard, but in most cases middle managers will be running realtime indexing tasks. with that in mind , there would be

10 druid jvm processes (1 middle manager itself and 9 worker peons created as needed)

middle manager will have jetty threads and some more for background book keeping

peon process for realtime task would use
2 configured threads for query processing + some more (~3-5 for other indexing related work)
heap memory would be used for indexing and to keep track of all the served segments [metadata] so that tasks know when a segment has been handed off completely and safe to be deleted
off heap memory for loading intermediate persisted segments, segments to be handed off to historical and for the “processing buffer = 2*512M”

tuning precisely would require some fiddling around.

– Himanshu

Thanks Himanshu very much

在 2015年12月6日星期日 UTC+8下午2:05:47,Himanshu Gupta写道: