Very high query wait time in historical node

Hi,

I’m getting very poor performance from my historical node.

system configuration:

192 Gig memory with 2TB of SSD, 40 core machine

runtime configuration:

druid.server.maxSize=1500000000000

druid.server.http.numThreads=50

druid.processing.buffer.sizeBytes=1073741824

druid.segmentCache.locations=[{“path”:"/user/druid/segment-cache",“maxSize”:1500000000000}]

druid.port=8083

druid.server.tier=slow

druid.indexer.runner.allowedPrefixes=[“com.metamx”, “druid”, “io.druid”, “user.timezone”, “file.encoding”, “hadoop”]

druid.processing.numThreads=30

druid.service=druid/historical

jvm config:

-server -Xms4g -Xmx4g -XX:MaxDirectMemorySize=34g -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djava.io.tmpdir=var/tmp -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager

log:

2017-10-08T04:51:16,059 INFO [topN_summary_daily_[2016-12-21T00:00:00.000Z/2016-12-22T00:00:00.000Z]] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2017-10-08T04:51:16.059Z”,“service”:“druid/historical”,“host”:“localhost:8083”,“metric”:“query/wait/time”,“value”:44890,“dataSource”:“summary_daily”,“dimension”:“user_type”,“duration”:“PT86400S”,“hasFilters”:“false”,“id”:“361e2078-188e-4a73-9353-99bf961c622c”,“interval”:[“2016-12-21T00:00:00.000Z/2016-12-22T00:00:00.000Z”],“numComplexMetrics”:“0”,“numMetrics”:“1”,“segment”:“summary_daily_2016-12-21T00:00:00.000Z_2016-12-22T00:00:00.000Z_2017-10-04T04:47:47.374Z_2”,“threshold”:“1000”,“type”:“topN”,“version”:“0.9.2”}]

Any pointers or help to improve this?

Were you able to troubleshoot the problem.