Druid historical node CPU runtime 20 minutes

Hi,

Just want to check if this is normal for Druid. During peak time we get 10,000 request/minute some of them are intense queries. I see from the log that the historical node spends almost 20 minutes to fetch every segment. Is this normal? And that slows down the query a lot. If I put timeout on the query for 10 seconds would it kill the fetching historical part?


2016-03-03T15:36:00,103 INFO [qtp1137661827-40] LoggingEmitter - Event [{"feed":"metrics","timestamp":"2016-03-03T15:36:00.103Z","service":"druid/historical","host":"address:8083","metric":"query/cpu/time","value": 1240258,"dataSource":"firehose-web","dimension":"url","duration":"PT921600S","hasFilters":"true","id":"ec9beaf6-9a0f-41cb-b533-1e364a30e2bf","interval":["2016-02-09T22:00:00.000Z/2016-02-10T21:00:00.000Z","2016-02-10T22:00:00.000Z/2016-02-11T08:00:00.000Z","2016-02-11T09:00:00.000Z/2016-02-11T14:00:00.000Z","2016-02-11T15:00:00.000Z/2016-02-11T21:00:00.000Z","2016-02-11T22:00:00.000Z/2016-02-12T04:00:00.000Z","2016-02-12T05:00:00.000Z/2016-02-12T07:00:00.000Z","2016-02-12T09:00:00.000Z/2016-02-12T13:00:00.000Z","2016-02-12T14:00:00.000Z/2016-02-12T18:00:00.000Z","2016-02-12T20:00:00.000Z/2016-02-12T23:00:00.000Z","2016-02-13T00:00:00.000Z/2016-02-13T02:00:00.000Z","2016-02-13T04:00:00.000Z/2016-02-13T06:00:00.000Z","2016-02-13T08:00:00.000Z/2016-02-13T10:00:00.000Z","2016-02-13T13:00:00.000Z/2016-02-14T00:00:00.000Z","2016-02-14T01:00:00.000Z/2016-02-14T05:00:00.000Z","2016-02-14T06:00:00.000Z/2016-02-14T07:00:00.000Z","2016-02-14T09:00:00.000Z/2016-02-14T13:00:00.000Z","2016-02-14T14:00:00.000Z/2016-02-14T17:00:00.000Z","2016-02-14T18:00:00.000Z/2016-02-14T22:00:00.000Z","2016-02-14T23:00:00.000Z/2016-02-15T04:00:00.000Z","2016-02-15T05:00:00.000Z/2016-02-15T14:00:00.000Z","2016-02-15T15:00:00.000Z/2016-02-15T16:00:00.000Z","2016-02-15T17:00:00.000Z/2016-02-15T18:00:00.000Z","2016-02-15T19:00:00.000Z/2016-02-16T02:00:00.000Z","2016-02-16T03:00:00.000Z/2016-02-16T09:00:00.000Z","2016-02-16T10:00:00.000Z/2016-02-16T13:00:00.000Z","2016-02-16T14:00:00.000Z/2016-02-16T18:00:00.000Z","2016-02-16T22:00:00.000Z/2016-02-17T04:00:00.000Z","2016-02-17T06:00:00.000Z/2016-02-17T07:00:00.000Z","2016-02-17T08:00:00.000Z/2016-02-17T09:00:00.000Z","2016-02-17T12:00:00.000Z/2016-02-17T14:00:00.000Z","2016-02-17T16:00:00.000Z/2016-02-17T19:00:00.000Z","2016-02-17T20:00:00.000Z/2016-02-17T21:00:00.000Z","2016-02-17T22:00:00.000Z/2016-02-18T00:00:00.000Z","2016-02-18T01:00:00.000Z/2016-02-18T04:00:00.000Z","2016-02-18T05:00:00.000Z/2016-02-18T12:00:00.000Z","2016-02-18T14:00:00.000Z/2016-02-18T15:00:00.000Z","2016-02-18T17:00:00.000Z/2016-02-18T18:00:00.000Z","2016-02-23T00:00:00.000Z/2016-02-24T00:00:00.000Z","2016-02-26T00:00:00.000Z/2016-02-29T00:00:00.000Z"],"numComplexMetrics":"0","numMetrics":"1","threshold":"1000","type":"topN"}]

``

We have r3.4xlarge 6 boxes running.

Hi,
the metrics query/cpu/time refers to the microseconds of cpuTime taken by the query and not the actual time the query took to execute.

Check query/time metrics for the clock time taken to execute the query.

More details on metrics are given here -

http://druid.io/docs/latest/operations/metrics.html