Best practise for setting cached in Historical Node

Hi everyone,

I am a newbie to Druid just using it for a month. Right now I have pretty large data about 1 billion records.

I have 4 historical nodes each one

-Xms24g

-Xmx24g
-XX:NewSize=8g
-XX:MaxNewSize=8g
-XX:MaxDirectMemorySize=12g

10 threads each.

``

I have config cached to I choose hybrid

druid.historical.cache.useCache=true
druid.historical.cache.populateCache=true

Cache config hybrid

druid.cache.type=hybrid
druid.cache.l1.type=local
druid.cache.l2.type=memcached

l1

druid.cache.l1.sizeInBytes=4000000000
druid.cache.l1.initialSize=50000000

l2

druid.cache.l2.hosts=localhost:11211
druid.cache.l2.maxObjectSize=104857600
druid.cache.l2.timeout=1000

``

My datasource segment by hour

"segmentGranularity" : "HOUR",
"queryGranularity" : "HOUR",

The query I tested is on 1 day range.

But the response time is not good.

I wandering If I did a right config for a cached or not?

Is hybrid cache is the idea solution or just memcached or local one?

What is the next step to debug?

Thanks.

2016-09-15T05:19:45,938 INFO [qtp1655939945-35] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2016-09-15T05:19:45.938Z”,“service”:“druid/historical”,“host”:“adx-pro-vdchcm-ants19-ip-10-199-0-19:8083”,“metric”:“query/cpu/time”,“value”:187646,“dataSource”:“statistic_hourly”,“dimension”:“device_id”,“duration”:“PT43260S”,“hasFilters”:“true”,“id”:“a708b1b0-0b5e-434e-9cd2-b5b4025fe110”,“interval”:[“2016-09-14T10:00:00.000Z/2016-09-14T11:00:00.000Z”,“2016-09-14T12:00:00.000Z/2016-09-14T13:00:00.000Z”,“2016-09-14T14:00:00.000Z/2016-09-14T15:00:00.000Z”,“2016-09-14T16:00:00.000Z/2016-09-14T20:00:00.000Z”,“2016-09-14T21:00:00.000Z/2016-09-14T22:00:00.000Z”,“2016-09-14T23:00:00.000Z/2016-09-15T00:00:00.000Z”,“2016-09-15T01:00:00.000Z/2016-09-15T03:00:00.000Z”,“2016-09-15T06:00:00.000Z/2016-09-15T07:00:00.000Z”,“2016-09-15T08:00:00.000Z/2016-09-15T08:01:00.000Z”],“numComplexMetrics”:“0”,“numMetrics”:“2”,“threshold”:“1000”,“type”:“topN”,“version”:“0.9.1.1”}]

2016-09-15T05:19:45,961 INFO [qtp1655939945-27] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2016-09-15T05:19:45.961Z”,“service”:“druid/historical”,“host”:“adx-pro-vdchcm-ants19-ip-10-199-0-19:8083”,“metric”:“query/cpu/time”,“value”:227555,“dataSource”:“statistic_hourly”,“dimension”:“device_id”,“duration”:“PT32460S”,“hasFilters”:“true”,“id”:“7f093fdd-3890-4016-b51a-1a8262b0d380”,“interval”:[“2016-09-14T14:00:00.000Z/2016-09-14T15:00:00.000Z”,“2016-09-14T16:00:00.000Z/2016-09-14T20:00:00.000Z”,“2016-09-14T21:00:00.000Z/2016-09-14T22:00:00.000Z”,“2016-09-15T01:00:00.000Z/2016-09-15T03:00:00.000Z”,“2016-09-15T06:00:00.000Z/2016-09-15T07:00:00.000Z”,“2016-09-15T08:00:00.000Z/2016-09-15T08:01:00.000Z”],“numComplexMetrics”:“0”,“numMetrics”:“2”,“threshold”:“1000”,“type”:“topN”,“version”:“0.9.1.1”}]

2016-09-15T05:19:46,015 INFO [qtp1655939945-31] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2016-09-15T05:19:46.015Z”,“service”:“druid/historical”,“host”:“adx-pro-vdchcm-ants19-ip-10-199-0-19:8083”,“metric”:“query/cpu/time”,“value”:67521,“dataSource”:“statistic_hourly”,“dimension”:“device_id”,“duration”:“PT28860S”,“hasFilters”:“true”,“id”:“88eabf49-8a1c-4ff6-a7be-11f152f2fb8f”,“interval”:[“2016-09-14T10:00:00.000Z/2016-09-14T11:00:00.000Z”,“2016-09-14T12:00:00.000Z/2016-09-14T13:00:00.000Z”,“2016-09-14T16:00:00.000Z/2016-09-14T19:00:00.000Z”,“2016-09-14T23:00:00.000Z/2016-09-15T00:00:00.000Z”,“2016-09-15T01:00:00.000Z/2016-09-15T02:00:00.000Z”,“2016-09-15T06:00:00.000Z/2016-09-15T07:00:00.000Z”,“2016-09-15T08:00:00.000Z/2016-09-15T08:01:00.000Z”],“numComplexMetrics”:“0”,“numMetrics”:“2”,“threshold”:“1000”,“type”:“topN”,“version”:“0.9.1.1”}]

2016-09-15T05:19:46,069 INFO [qtp1655939945-39] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2016-09-15T05:19:46.069Z”,“service”:“druid/historical”,“host”:“adx-pro-vdchcm-ants19-ip-10-199-0-19:8083”,“metric”:“query/cpu/time”,“value”:312586,“dataSource”:“statistic_hourly”,“dimension”:“device_id”,“duration”:“PT14400S”,“hasFilters”:“true”,“id”:“a3ea5166-079d-4784-b9aa-5f9769e20977”,“interval”:[“2016-09-14T14:00:00.000Z/2016-09-14T15:00:00.000Z”,“2016-09-14T19:00:00.000Z/2016-09-14T20:00:00.000Z”,“2016-09-14T21:00:00.000Z/2016-09-14T22:00:00.000Z”,“2016-09-15T02:00:00.000Z/2016-09-15T03:00:00.000Z”],“numComplexMetrics”:“0”,“numMetrics”:“2”,“threshold”:“1000”,“type”:“topN”,“version”:“0.9.1.1”}]

2016-09-15T05:19:46,093 INFO [qtp1655939945-23] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2016-09-15T05:19:46.093Z”,“service”:“druid/historical”,“host”:“adx-pro-vdchcm-ants19-ip-10-199-0-19:8083”,“metric”:“query/cpu/time”,“value”:87316,“dataSource”:“statistic_hourly”,“dimension”:“device_id”,“duration”:“PT28860S”,“hasFilters”:“true”,“id”:“1f546b56-a3ef-48e4-9a10-52a5ef664e56”,“interval”:[“2016-09-14T10:00:00.000Z/2016-09-14T11:00:00.000Z”,“2016-09-14T12:00:00.000Z/2016-09-14T13:00:00.000Z”,“2016-09-14T16:00:00.000Z/2016-09-14T19:00:00.000Z”,“2016-09-14T23:00:00.000Z/2016-09-15T00:00:00.000Z”,“2016-09-15T01:00:00.000Z/2016-09-15T02:00:00.000Z”,“2016-09-15T06:00:00.000Z/2016-09-15T07:00:00.000Z”,“2016-09-15T08:00:00.000Z/2016-09-15T08:01:00.000Z”],“numComplexMetrics”:“0”,“numMetrics”:“2”,“threshold”:“1000”,“type”:“topN”,“version”:“0.9.1.1”}]

2016-09-15T05:19:46,130 INFO [qtp1655939945-28] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2016-09-15T05:19:46.130Z”,“service”:“druid/historical”,“host”:“adx-pro-vdchcm-ants19-ip-10-199-0-19:8083”,“metric”:“query/cpu/time”,“value”:20222,“dataSource”:“statistic_hourly”,“dimension”:“device_id”,“duration”:“PT18060S”,“hasFilters”:“true”,“id”:“db20b931-e4a6-4b06-a0a2-cd76cd173db7”,“interval”:[“2016-09-14T16:00:00.000Z/2016-09-14T19:00:00.000Z”,“2016-09-15T01:00:00.000Z/2016-09-15T02:00:00.000Z”,“2016-09-15T06:00:00.000Z/2016-09-15T07:00:00.000Z”,“2016-09-15T08:00:00.000Z/2016-09-15T08:01:00.000Z”],“numComplexMetrics”:“0”,“numMetrics”:“2”,“threshold”:“1000”,“type”:“topN”,“version”:“0.9.1.1”}]

2016-09-15T05:19:46,153 INFO [qtp1655939945-40] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2016-09-15T05:19:46.153Z”,“service”:“druid/historical”,“host”:“adx-pro-vdchcm-ants19-ip-10-199-0-19:8083”,“metric”:“query/cpu/time”,“value”:13938,“dataSource”:“statistic_hourly”,“dimension”:“device_id”,“duration”:“PT18060S”,“hasFilters”:“true”,“id”:“878264db-9614-4925-ab3c-29a6e244399e”,“interval”:[“2016-09-14T16:00:00.000Z/2016-09-14T19:00:00.000Z”,“2016-09-15T01:00:00.000Z/2016-09-15T02:00:00.000Z”,“2016-09-15T06:00:00.000Z/2016-09-15T07:00:00.000Z”,“2016-09-15T08:00:00.000Z/2016-09-15T08:01:00.000Z”],“numComplexMetrics”:“0”,“numMetrics”:“2”,“threshold”:“1000”,“type”:“topN”,“version”:“0.9.1.1”}]

2016-09-15T05:19:46,161 INFO [qtp1655939945-37] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2016-09-15T05:19:46.161Z”,“service”:“druid/historical”,“host”:“adx-pro-vdchcm-ants19-ip-10-199-0-19:8083”,“metric”:“query/cpu/time”,“value”:8127,“dataSource”:“statistic_hourly”,“dimension”:“device_id”,“duration”:“PT18060S”,“hasFilters”:“true”,“id”:“939aa7ba-f22d-4372-94ae-74b9e3167dec”,“interval”:[“2016-09-14T16:00:00.000Z/2016-09-14T19:00:00.000Z”,“2016-09-15T01:00:00.000Z/2016-09-15T02:00:00.000Z”,“2016-09-15T06:00:00.000Z/2016-09-15T07:00:00.000Z”,“2016-09-15T08:00:00.000Z/2016-09-15T08:01:00.000Z”],“numComplexMetrics”:“0”,“numMetrics”:“2”,“threshold”:“1000”,“type”:“topN”,“version”:“0.9.1.1”}]

``

2016-09-15T05:20:32,339 INFO [segmentMetadata_statistic_hourly_24h_[2016-09-15T09:00:00.000Z/2016-09-15T10:00:00.000Z]] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2016-09-15T05:20:32.339Z”,“service”:“druid/historical”,“host”:“adx-pro-vdchcm-ants19-ip-10-199-0-19:8083”,“metric”:“query/segmentAndCache/time”,“value”:0,“dataSource”:“statistic_hourly_24h”,“duration”:“PT3600S”,“hasFilters”:“false”,“id”:“9d6e9cff-744b-4ee1-936e-9229ee52c0cc”,“interval”:[“2016-09-15T09:00:00.000Z/2016-09-15T10:00:00.000Z”],“segment”:“statistic_hourly_24h_2016-09-15T09:00:00.000Z_2016-09-15T10:00:00.000Z_2016-09-15T03:15:01.472Z”,“type”:“segmentMetadata”,“version”:“0.9.1.1”}]

2016-09-15T05:20:32,341 INFO [segmentMetadata_statistic_hourly_24h_[2016-09-15T11:00:00.000Z/2016-09-15T12:00:00.000Z]] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2016-09-15T05:20:32.341Z”,“service”:“druid/historical”,“host”:“adx-pro-vdchcm-ants19-ip-10-199-0-19:8083”,“metric”:“query/segmentAndCache/time”,“value”:1,“dataSource”:“statistic_hourly_24h”,“duration”:“PT3600S”,“hasFilters”:“false”,“id”:“9d6e9cff-744b-4ee1-936e-9229ee52c0cc”,“interval”:[“2016-09-15T11:00:00.000Z/2016-09-15T12:00:00.000Z”],“segment”:“statistic_hourly_24h_2016-09-15T11:00:00.000Z_2016-09-15T12:00:00.000Z_2016-09-15T05:15:01.569Z”,“type”:“segmentMetadata”,“version”:“0.9.1.1”}]

``

2016-09-15T05:22:32,303 INFO [segmentMetadata_statistic_hourly_segment_by_day_[2016-08-23T00:00:00.000Z/2016-08-24T00:00:00.000Z]] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2016-09-15T05:22:32.303Z”,“service”:“druid/historical”,“host”:“adx-pro-vdchcm-ants19-ip-10-199-0-19:8083”,“metric”:“query/wait/time”,“value”:0,“dataSource”:“statistic_hourly_segment_by_day”,“duration”:“PT86400S”,“hasFilters”:“false”,“id”:“d4f79b96-a129-4c5d-8e8d-82160854ae4d”,“interval”:[“2016-08-23T00:00:00.000Z/2016-08-24T00:00:00.000Z”],“segment”:“statistic_hourly_segment_by_day_2016-08-23T00:00:00.000Z_2016-08-24T00:00:00.000Z_2016-08-25T17:02:56.469Z_1”,“type”:“segmentMetadata”,“version”:“0.9.1.1”}]

2016-09-15T05:22:32,304 INFO [segmentMetadata_statistic_hourly_segment_by_day_[2016-08-25T00:00:00.000Z/2016-08-26T00:00:00.000Z]] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2016-09-15T05:22:32.303Z”,“service”:“druid/historical”,“host”:“adx-pro-vdchcm-ants19-ip-10-199-0-19:8083”,“metric”:“query/wait/time”,“value”:0,“dataSource”:“statistic_hourly_segment_by_day”,“duration”:“PT86400S”,“hasFilters”:“false”,“id”:“d4f79b96-a129-4c5d-8e8d-82160854ae4d”,“interval”:[“2016-08-25T00:00:00.000Z/2016-08-26T00:00:00.000Z”],“segment”:“statistic_hourly_segment_by_day_2016-08-25T00:00:00.000Z_2016-08-26T00:00:00.000Z_2016-08-25T17:02:56.469Z”,“type”:“segmentMetadata”,“version”:“0.9.1.1”}]

2016-09-15T05:22:32,304 INFO [segmentMetadata_statistic_hourly_segment_by_day_[2016-08-25T00:00:00.000Z/2016-08-26T00:00:00.000Z]] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2016-09-15T05:22:32.304Z”,“service”:“druid/historical”,“host”:“adx-pro-vdchcm-ants19-ip-10-199-0-19:8083”,“metric”:“query/wait/time”,“value”:0,“dataSource”:“statistic_hourly_segment_by_day”,“duration”:“PT86400S”,“hasFilters”:“false”,“id”:“d4f79b96-a129-4c5d-8e8d-82160854ae4d”,“interval”:[“2016-08-25T00:00:00.000Z/2016-08-26T00:00:00.000Z”],“segment”:“statistic_hourly_segment_by_day_2016-08-25T00:00:00.000Z_2016-08-26T00:00:00.000Z_2016-08-25T17:02:56.469Z_1”,“type”:“segmentMetadata”,“version”:“0.9.1.1”}]

``

2016-09-15T05:22:13,743 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2016-09-15T05:22:13.743Z”,“service”:“druid/historical”,“host”:“adx-pro-vdchcm-ants19-ip-10-199-0-19:8083”,“metric”:“segment/scan/pending”,“value”:0}]

2016-09-15T05:23:13,743 INFO [MonitorScheduler-0] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2016-09-15T05:23:13.743Z”,“service”:“druid/historical”,“host”:“adx-pro-vdchcm-ants19-ip-10-199-0-19:8083”,“metric”:“segment/scan/pending”,“value”:0}]

``

Hello,
I put some metrics in a historical node.
As I mention I saw that **“query/cpu/time”,“value”:312586 **is high that why the query is slow but the cached seems good I think because “query/segmentAndCache/time”,“value”:0 for almost query .

Any reasons for that?

Hi query/cpu/time can be more than the actual query latency as the query processing is done parallely by multiple processing threads.
what is the value of ‘query/time’ metric for your query ?

The time may not be consistent but here is some of my logs.

2016-09-19T14:44:01,592 INFO [qtp373437306-35] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2016-09-19T14:44:01.592Z”,“service”:“druid/historical”,“host”:“adx-pro-vdchcm-ants19-ip-10-199-0-19:8083”,“metric”:“query/time”,“value”:76,“context”:"{“finalize”:false,“queryId”:“1216c8fb-be3a-4740-9ffe-6af954b609a9”,“timeout”:40000}",“dataSource”:“statistic_hourly”,“duration”:“PT50400S”,“hasFilters”:“true”,“id”:“1216c8fb-be3a-4740-9ffe-6af954b609a9”,“interval”:[“2016-09-13T18:00:00.000Z/2016-09-13T19:00:00.000Z”,“2016-09-13T20:00:00.000Z/2016-09-13T21:00:00.000Z”,“2016-09-13T22:00:00.000Z/2016-09-14T00:00:00.000Z”,“2016-09-14T10:00:00.000Z/2016-09-14T11:00:00.000Z”,“2016-09-14T12:00:00.000Z/2016-09-14T13:00:00.000Z”,“2016-09-14T16:00:00.000Z/2016-09-14T17:00:00.000Z”,“2016-09-14T18:00:00.000Z/2016-09-14T19:00:00.000Z”,“2016-09-15T02:00:00.000Z/2016-09-15T03:00:00.000Z”,“2016-09-15T06:00:00.000Z/2016-09-15T07:00:00.000Z”,“2016-09-15T08:00:00.000Z/2016-09-15T09:00:00.000Z”,“2016-09-19T12:00:00.000Z/2016-09-19T13:00:00.000Z”,“2016-09-19T14:00:00.000Z/2016-09-19T15:00:00.000Z”,“2016-09-19T18:00:00.000Z/2016-09-19T19:00:00.000Z”],“remoteAddress”:“10.199.0.19”,“success”:“true”,“type”:“topN”,“version”:“0.9.1.1”}]

2016-09-19T14:44:01,606 INFO [qtp373437306-39] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2016-09-19T14:44:01.606Z”,“service”:“druid/historical”,“host”:“adx-pro-vdchcm-ants19-ip-10-199-0-19:8083”,“metric”:“query/time”,“value”:96,“context”:"{“finalize”:false,“queryId”:“1700223c-5a11-4af4-b8c0-f56593810829”,“timeout”:40000}",“dataSource”:“statistic_hourly”,“duration”:“PT50400S”,“hasFilters”:“true”,“id”:“1700223c-5a11-4af4-b8c0-f56593810829”,“interval”:[“2016-09-13T18:00:00.000Z/2016-09-13T19:00:00.000Z”,“2016-09-13T20:00:00.000Z/2016-09-13T21:00:00.000Z”,“2016-09-13T22:00:00.000Z/2016-09-14T00:00:00.000Z”,“2016-09-14T10:00:00.000Z/2016-09-14T11:00:00.000Z”,“2016-09-14T12:00:00.000Z/2016-09-14T13:00:00.000Z”,“2016-09-14T16:00:00.000Z/2016-09-14T17:00:00.000Z”,“2016-09-14T18:00:00.000Z/2016-09-14T19:00:00.000Z”,“2016-09-15T02:00:00.000Z/2016-09-15T03:00:00.000Z”,“2016-09-15T06:00:00.000Z/2016-09-15T07:00:00.000Z”,“2016-09-15T08:00:00.000Z/2016-09-15T09:00:00.000Z”,“2016-09-19T12:00:00.000Z/2016-09-19T13:00:00.000Z”,“2016-09-19T14:00:00.000Z/2016-09-19T15:00:00.000Z”,“2016-09-19T18:00:00.000Z/2016-09-19T19:00:00.000Z”],“remoteAddress”:“10.199.0.19”,“success”:“true”,“type”:“topN”,“version”:“0.9.1.1”}]

2016-09-19T14:44:01,710 INFO [qtp373437306-52] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2016-09-19T14:44:01.710Z”,“service”:“druid/historical”,“host”:“adx-pro-vdchcm-ants19-ip-10-199-0-19:8083”,“metric”:“query/time”,“value”:43,“context”:"{“finalize”:false,“queryId”:“c8d4d798-4db7-40a3-973c-9f40fb247896”,“timeout”:40000}",“dataSource”:“statistic_hourly”,“duration”:“PT14400S”,“hasFilters”:“true”,“id”:“c8d4d798-4db7-40a3-973c-9f40fb247896”,“interval”:[“2016-09-13T18:00:00.000Z/2016-09-13T19:00:00.000Z”,“2016-09-14T10:00:00.000Z/2016-09-14T11:00:00.000Z”,“2016-09-14T12:00:00.000Z/2016-09-14T13:00:00.000Z”,“2016-09-15T06:00:00.000Z/2016-09-15T07:00:00.000Z”],“remoteAddress”:“10.199.0.19”,“success”:“true”,“type”:“topN”,“version”:“0.9.1.1”}]

2016-09-19T14:44:01,710 INFO [qtp373437306-46] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2016-09-19T14:44:01.710Z”,“service”:“druid/historical”,“host”:“adx-pro-vdchcm-ants19-ip-10-199-0-19:8083”,“metric”:“query/time”,“value”:76,“context”:"{“finalize”:false,“queryId”:“0d13aff3-a609-41c3-9dd2-2912b948ac28”,“timeout”:40000}",“dataSource”:“statistic_hourly”,“duration”:“PT39600S”,“hasFilters”:“true”,“id”:“0d13aff3-a609-41c3-9dd2-2912b948ac28”,“interval”:[“2016-09-13T18:00:00.000Z/2016-09-13T19:00:00.000Z”,“2016-09-13T20:00:00.000Z/2016-09-13T21:00:00.000Z”,“2016-09-13T23:00:00.000Z/2016-09-14T00:00:00.000Z”,“2016-09-14T10:00:00.000Z/2016-09-14T11:00:00.000Z”,“2016-09-14T12:00:00.000Z/2016-09-14T13:00:00.000Z”,“2016-09-14T16:00:00.000Z/2016-09-14T17:00:00.000Z”,“2016-09-14T18:00:00.000Z/2016-09-14T19:00:00.000Z”,“2016-09-15T06:00:00.000Z/2016-09-15T07:00:00.000Z”,“2016-09-15T08:00:00.000Z/2016-09-15T09:00:00.000Z”,“2016-09-19T14:00:00.000Z/2016-09-19T15:00:00.000Z”,“2016-09-19T18:00:00.000Z/2016-09-19T19:00:00.000Z”],“remoteAddress”:“10.199.0.19”,“success”:“true”,“type”:“topN”,“version”:“0.9.1.1”}]

2016-09-19T14:44:01,763 INFO [qtp373437306-47] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2016-09-19T14:44:01.763Z”,“service”:“druid/historical”,“host”:“adx-pro-vdchcm-ants19-ip-10-199-0-19:8083”,“metric”:“query/time”,“value”:19,“context”:"{“finalize”:false,“queryId”:“037cc6d6-692f-4046-abe2-b2b7d740deed”,“timeout”:40000}",“dataSource”:“statistic_hourly”,“duration”:“PT10800S”,“hasFilters”:“true”,“id”:“037cc6d6-692f-4046-abe2-b2b7d740deed”,“interval”:[“2016-09-13T18:00:00.000Z/2016-09-13T19:00:00.000Z”,“2016-09-14T12:00:00.000Z/2016-09-14T13:00:00.000Z”,“2016-09-15T06:00:00.000Z/2016-09-15T07:00:00.000Z”],“remoteAddress”:“10.199.0.19”,“success”:“true”,“type”:“topN”,“version”:“0.9.1.1”}]

2016-09-19T14:44:01,768 INFO [qtp373437306-45] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2016-09-19T14:44:01.768Z”,“service”:“druid/historical”,“host”:“adx-pro-vdchcm-ants19-ip-10-199-0-19:8083”,“metric”:“query/time”,“value”:47,“context”:"{“finalize”:false,“queryId”:“1738ad3f-01fb-4c14-8cb7-3786caf6bfc9”,“timeout”:40000}",“dataSource”:“statistic_hourly”,“duration”:“PT39600S”,“hasFilters”:“true”,“id”:“1738ad3f-01fb-4c14-8cb7-3786caf6bfc9”,“interval”:[“2016-09-13T18:00:00.000Z/2016-09-13T19:00:00.000Z”,“2016-09-13T20:00:00.000Z/2016-09-13T21:00:00.000Z”,“2016-09-13T23:00:00.000Z/2016-09-14T00:00:00.000Z”,“2016-09-14T10:00:00.000Z/2016-09-14T11:00:00.000Z”,“2016-09-14T12:00:00.000Z/2016-09-14T13:00:00.000Z”,“2016-09-14T16:00:00.000Z/2016-09-14T17:00:00.000Z”,“2016-09-14T18:00:00.000Z/2016-09-14T19:00:00.000Z”,“2016-09-15T06:00:00.000Z/2016-09-15T07:00:00.000Z”,“2016-09-15T08:00:00.000Z/2016-09-15T09:00:00.000Z”,“2016-09-19T14:00:00.000Z/2016-09-19T15:00:00.000Z”,“2016-09-19T18:00:00.000Z/2016-09-19T19:00:00.000Z”],“remoteAddress”:“10.199.0.19”,“success”:“true”,“type”:“topN”,“version”:“0.9.1.1”}]

2016-09-19T14:44:01,825 INFO [qtp373437306-43] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2016-09-19T14:44:01.825Z”,“service”:“druid/historical”,“host”:“adx-pro-vdchcm-ants19-ip-10-199-0-19:8083”,“metric”:“query/time”,“value”:112,“context”:"{“finalize”:false,“queryId”:“472b2bc3-7590-4ff7-9f0a-f263901a1280”,“timeout”:40000}",“dataSource”:“statistic_hourly”,“duration”:“PT50400S”,“hasFilters”:“true”,“id”:“472b2bc3-7590-4ff7-9f0a-f263901a1280”,“interval”:[“2016-09-13T18:00:00.000Z/2016-09-13T19:00:00.000Z”,“2016-09-13T20:00:00.000Z/2016-09-13T21:00:00.000Z”,“2016-09-13T22:00:00.000Z/2016-09-14T00:00:00.000Z”,“2016-09-14T10:00:00.000Z/2016-09-14T11:00:00.000Z”,“2016-09-14T12:00:00.000Z/2016-09-14T13:00:00.000Z”,“2016-09-14T16:00:00.000Z/2016-09-14T17:00:00.000Z”,“2016-09-14T18:00:00.000Z/2016-09-14T19:00:00.000Z”,“2016-09-15T02:00:00.000Z/2016-09-15T03:00:00.000Z”,“2016-09-15T06:00:00.000Z/2016-09-15T07:00:00.000Z”,“2016-09-15T08:00:00.000Z/2016-09-15T09:00:00.000Z”,“2016-09-19T12:00:00.000Z/2016-09-19T13:00:00.000Z”,“2016-09-19T14:00:00.000Z/2016-09-19T15:00:00.000Z”,“2016-09-19T18:00:00.000Z/2016-09-19T19:00:00.000Z”],“remoteAddress”:“10.199.0.19”,“success”:“true”,“type”:“topN”,“version”:“0.9.1.1”}]

2016-09-19T14:44:01,922 INFO [qtp373437306-44] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2016-09-19T14:44:01.922Z”,“service”:“druid/historical”,“host”:“adx-pro-vdchcm-ants19-ip-10-199-0-19:8083”,“metric”:“query/time”,“value”:34,“context”:"{“finalize”:false,“queryId”:“74da0059-f5ff-45d5-a70b-9d95fbeeb6fc”,“timeout”:40000}",“dataSource”:“statistic_hourly”,“duration”:“PT7200S”,“hasFilters”:“true”,“id”:“74da0059-f5ff-45d5-a70b-9d95fbeeb6fc”,“interval”:[“2016-09-14T12:00:00.000Z/2016-09-14T13:00:00.000Z”,“2016-09-15T06:00:00.000Z/2016-09-15T07:00:00.000Z”],“remoteAddress”:“10.199.0.19”,“success”:“true”,“type”:“topN”,“version”:“0.9.1.1”}]

2016-09-19T14:44:01,923 INFO [qtp373437306-33] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2016-09-19T14:44:01.923Z”,“service”:“druid/historical”,“host”:“adx-pro-vdchcm-ants19-ip-10-199-0-19:8083”,“metric”:“query/time”,“value”:48,“context”:"{“finalize”:false,“queryId”:“31fae327-ce3a-402a-af02-98400d12bd8f”,“timeout”:40000}",“dataSource”:“statistic_hourly”,“duration”:“PT14400S”,“hasFilters”:“true”,“id”:“31fae327-ce3a-402a-af02-98400d12bd8f”,“interval”:[“2016-09-13T18:00:00.000Z/2016-09-13T19:00:00.000Z”,“2016-09-14T10:00:00.000Z/2016-09-14T11:00:00.000Z”,“2016-09-14T12:00:00.000Z/2016-09-14T13:00:00.000Z”,“2016-09-15T06:00:00.000Z/2016-09-15T07:00:00.000Z”],“remoteAddress”:“10.199.0.19”,“success”:“true”,“type”:“topN”,“version”:“0.9.1.1”}]

2016-09-19T14:44:01,934 INFO [qtp373437306-48] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2016-09-19T14:44:01.934Z”,“service”:“druid/historical”,“host”:“adx-pro-vdchcm-ants19-ip-10-199-0-19:8083”,“metric”:“query/time”,“value”:71,“context”:"{“finalize”:false,“queryId”:“a2280413-3a9e-4ff9-85cf-26cbe5bc4e5a”,“timeout”:40000}",“dataSource”:“statistic_hourly”,“duration”:“PT14400S”,“hasFilters”:“true”,“id”:“a2280413-3a9e-4ff9-85cf-26cbe5bc4e5a”,“interval”:[“2016-09-13T18:00:00.000Z/2016-09-13T19:00:00.000Z”,“2016-09-14T10:00:00.000Z/2016-09-14T11:00:00.000Z”,“2016-09-14T12:00:00.000Z/2016-09-14T13:00:00.000Z”,“2016-09-15T06:00:00.000Z/2016-09-15T07:00:00.000Z”],“remoteAddress”:“10.199.0.19”,“success”:“true”,“type”:“topN”,“version”:“0.9.1.1”}]

2016-09-19T14:44:01,935 INFO [qtp373437306-51] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2016-09-19T14:44:01.935Z”,“service”:“druid/historical”,“host”:“adx-pro-vdchcm-ants19-ip-10-199-0-19:8083”,“metric”:“query/time”,“value”:85,“context”:"{“finalize”:false,“queryId”:“c3dcbcfd-ad4b-4d76-a1d2-a89f5d298529”,“timeout”:40000}",“dataSource”:“statistic_hourly”,“duration”:“PT39600S”,“hasFilters”:“true”,“id”:“c3dcbcfd-ad4b-4d76-a1d2-a89f5d298529”,“interval”:[“2016-09-13T18:00:00.000Z/2016-09-13T19:00:00.000Z”,“2016-09-13T20:00:00.000Z/2016-09-13T21:00:00.000Z”,“2016-09-13T23:00:00.000Z/2016-09-14T00:00:00.000Z”,“2016-09-14T10:00:00.000Z/2016-09-14T11:00:00.000Z”,“2016-09-14T12:00:00.000Z/2016-09-14T13:00:00.000Z”,“2016-09-14T16:00:00.000Z/2016-09-14T17:00:00.000Z”,“2016-09-14T18:00:00.000Z/2016-09-14T19:00:00.000Z”,“2016-09-15T06:00:00.000Z/2016-09-15T07:00:00.000Z”,“2016-09-15T08:00:00.000Z/2016-09-15T09:00:00.000Z”,“2016-09-19T14:00:00.000Z/2016-09-19T15:00:00.000Z”,“2016-09-19T18:00:00.000Z/2016-09-19T19:00:00.000Z”],“remoteAddress”:“10.199.0.19”,“success”:“true”,“type”:“topN”,“version”:“0.9.1.1”}]

2016-09-19T14:44:02,005 INFO [qtp373437306-36] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2016-09-19T14:44:02.005Z”,“service”:“druid/historical”,“host”:“adx-pro-vdchcm-ants19-ip-10-199-0-19:8083”,“metric”:“query/time”,“value”:31,“context”:"{“finalize”:false,“queryId”:“0a51a278-8bf4-4f9e-a4c7-47bfd7b50083”,“timeout”:40000}",“dataSource”:“statistic_hourly”,“duration”:“PT39600S”,“hasFilters”:“true”,“id”:“0a51a278-8bf4-4f9e-a4c7-47bfd7b50083”,“interval”:[“2016-09-13T18:00:00.000Z/2016-09-13T19:00:00.000Z”,“2016-09-13T20:00:00.000Z/2016-09-13T21:00:00.000Z”,“2016-09-13T23:00:00.000Z/2016-09-14T00:00:00.000Z”,“2016-09-14T10:00:00.000Z/2016-09-14T11:00:00.000Z”,“2016-09-14T12:00:00.000Z/2016-09-14T13:00:00.000Z”,“2016-09-14T16:00:00.000Z/2016-09-14T17:00:00.000Z”,“2016-09-14T18:00:00.000Z/2016-09-14T19:00:00.000Z”,“2016-09-15T06:00:00.000Z/2016-09-15T07:00:00.000Z”,“2016-09-15T08:00:00.000Z/2016-09-15T09:00:00.000Z”,“2016-09-19T14:00:00.000Z/2016-09-19T15:00:00.000Z”,“2016-09-19T18:00:00.000Z/2016-09-19T19:00:00.000Z”],“remoteAddress”:“10.199.0.19”,“success”:“true”,“type”:“topN”,“version”:“0.9.1.1”}]

2016-09-19T14:44:02,123 INFO [qtp373437306-38] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2016-09-19T14:44:02.123Z”,“service”:“druid/historical”,“host”:“adx-pro-vdchcm-ants19-ip-10-199-0-19:8083”,“metric”:“query/time”,“value”:34,“context”:"{“finalize”:false,“queryId”:“0780e55f-6b2e-490e-b82d-27d16fc80f04”,“timeout”:40000}",“dataSource”:“statistic_hourly”,“duration”:“PT7200S”,“hasFilters”:“true”,“id”:“0780e55f-6b2e-490e-b82d-27d16fc80f04”,“interval”:[“2016-09-14T12:00:00.000Z/2016-09-14T13:00:00.000Z”,“2016-09-15T06:00:00.000Z/2016-09-15T07:00:00.000Z”],“remoteAddress”:“10.199.0.19”,“success”:“true”,“type”:“topN”,“version”:“0.9.1.1”}]

2016-09-19T14:44:02,131 INFO [qtp373437306-32] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2016-09-19T14:44:02.131Z”,“service”:“druid/historical”,“host”:“adx-pro-vdchcm-ants19-ip-10-199-0-19:8083”,“metric”:“query/time”,“value”:71,“context”:"{“finalize”:false,“queryId”:“7f272dc0-739e-4fc3-8787-199e65cbd14e”,“timeout”:40000}",“dataSource”:“statistic_hourly”,“duration”:“PT14400S”,“hasFilters”:“true”,“id”:“7f272dc0-739e-4fc3-8787-199e65cbd14e”,“interval”:[“2016-09-13T18:00:00.000Z/2016-09-13T19:00:00.000Z”,“2016-09-14T10:00:00.000Z/2016-09-14T11:00:00.000Z”,“2016-09-14T12:00:00.000Z/2016-09-14T13:00:00.000Z”,“2016-09-15T06:00:00.000Z/2016-09-15T07:00:00.000Z”],“remoteAddress”:“10.199.0.19”,“success”:“true”,“type”:“topN”,“version”:“0.9.1.1”}]

2016-09-19T14:44:02,132 INFO [qtp373437306-49] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2016-09-19T14:44:02.132Z”,“service”:“druid/historical”,“host”:“adx-pro-vdchcm-ants19-ip-10-199-0-19:8083”,“metric”:“query/time”,“value”:73,“context”:"{“finalize”:false,“queryId”:“0d7e8656-1cf5-48eb-965e-3148cd46d4a2”,“timeout”:40000}",“dataSource”:“statistic_hourly”,“duration”:“PT39600S”,“hasFilters”:“true”,“id”:“0d7e8656-1cf5-48eb-965e-3148cd46d4a2”,“interval”:[“2016-09-13T18:00:00.000Z/2016-09-13T19:00:00.000Z”,“2016-09-13T20:00:00.000Z/2016-09-13T21:00:00.000Z”,“2016-09-13T23:00:00.000Z/2016-09-14T00:00:00.000Z”,“2016-09-14T10:00:00.000Z/2016-09-14T11:00:00.000Z”,“2016-09-14T12:00:00.000Z/2016-09-14T13:00:00.000Z”,“2016-09-14T16:00:00.000Z/2016-09-14T17:00:00.000Z”,“2016-09-14T18:00:00.000Z/2016-09-14T19:00:00.000Z”,“2016-09-15T06:00:00.000Z/2016-09-15T07:00:00.000Z”,“2016-09-15T08:00:00.000Z/2016-09-15T09:00:00.000Z”,“2016-09-19T14:00:00.000Z/2016-09-19T15:00:00.000Z”,“2016-09-19T18:00:00.000Z/2016-09-19T19:00:00.000Z”],“remoteAddress”:“10.199.0.19”,“success”:“true”,“type”:“topN”,“version”:“0.9.1.1”}]

2016-09-19T14:44:02,137 INFO [qtp373437306-34] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2016-09-19T14:44:02.137Z”,“service”:“druid/historical”,“host”:“adx-pro-vdchcm-ants19-ip-10-199-0-19:8083”,“metric”:“query/time”,“value”:54,“context”:"{“finalize”:false,“queryId”:“d098a9eb-2937-4001-ae49-07a35457f093”,“timeout”:40000}",“dataSource”:“statistic_hourly”,“duration”:“PT14400S”,“hasFilters”:“true”,“id”:“d098a9eb-2937-4001-ae49-07a35457f093”,“interval”:[“2016-09-13T18:00:00.000Z/2016-09-13T19:00:00.000Z”,“2016-09-14T10:00:00.000Z/2016-09-14T11:00:00.000Z”,“2016-09-14T12:00:00.000Z/2016-09-14T13:00:00.000Z”,“2016-09-15T06:00:00.000Z/2016-09-15T07:00:00.000Z”],“remoteAddress”:“10.199.0.19”,“success”:“true”,“type”:“topN”,“version”:“0.9.1.1”}]

2016-09-19T14:44:02,256 INFO [qtp373437306-50] com.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“metrics”,“timestamp”:“2016-09-19T14:44:02.256Z”,“service”:“druid/historical”,“host”:“adx-pro-vdchcm-ants19-ip-10-199-0-19:8083”,“metric”:“query/time”,“value”:89,“context”:"{“finalize”:false,“queryId”:“dd946f53-0e7e-47ae-9992-65da0e5902de”,“timeout”:40000}",“dataSource”:“statistic_hourly”,“duration”:“PT39600S”,“hasFilters”:“true”,“id”:“dd946f53-0e7e-47ae-9992-65da0e5902de”,“interval”:[“2016-09-13T18:00:00.000Z/2016-09-13T19:00:00.000Z”,“2016-09-13T20:00:00.000Z/2016-09-13T21:00:00.000Z”,“2016-09-13T23:00:00.000Z/2016-09-14T00:00:00.000Z”,“2016-09-14T10:00:00.000Z/2016-09-14T11:00:00.000Z”,“2016-09-14T12:00:00.000Z/2016-09-14T13:00:00.000Z”,“2016-09-14T16:00:00.000Z/2016-09-14T17:00:00.000Z”,“2016-09-14T18:00:00.000Z/2016-09-14T19:00:00.000Z”,“2016-09-15T06:00:00.000Z/2016-09-15T07:00:00.000Z”,“2016-09-15T08:00:00.000Z/2016-09-15T09:00:00.000Z”,“2016-09-19T14:00:00.000Z/2016-09-19T15:00:00.000Z”,“2016-09-19T18:00:00.000Z/2016-09-19T19:00:00.000Z”],“remoteAddress”:“10.199.0.19”,“success”:“true”,“type”:“topN”,“version”:“0.9.1.1”}]

The query/time is small but in a query for 3 dimensions and 4 metrics it takes over 10s to get the result.

From the metrics you sent, the historical seems to be doing good.
I guess the broker might be having sub optimal config or having GC issues.

Check your broker query metrics and gc logs.