Many "Maximum number of rows reached" errors in indexing service task

Hi,

My druid cluster version is 0.8.0, many exceptions “Maximum number of rows reached” found in the log of indexing service tasks .

In the meanwhile, the tranquility clients print lots of warnings :

[INFO] 2016-03-03 22:18:33,258 com.metamx.emitter.core.LoggingEmitter info - Event [{“feed”:“alerts”,“timestamp”:“2016-03-03T22:18:33.258+08:00”,“service”:“tranquility”,“host”:“localhost”,“severity”:“anomaly”,“description”:“Failed to propagate events: druid:prod:overlord/MiddletiersBillingProd”,“data”:{“exceptionType”:“com.twitter.finagle.GlobalRequestTimeoutException”,“exceptionStackTrace”:“com.twitter.finagle.GlobalRequestTimeoutException: exceeded 1.minutes+30.seconds to druid:firehose:MiddletiersBillingProd-14-0000-0000 while waiting for a response for the request, including retries (if applicable)\n\tat com.twitter.finagle.NoStacktrace(Unknown Source)\n”,“timestamp”:“2016-03-03T22:00:00.000+08:00”,“beams”:“HashPartitionBeam(DruidBeam(interval = 2016-03-03T22:00:00.000+08:00/2016-03-03T23:00:00.000+08:00, partition = 0, tasks = [index_realtime_MiddletiersBillingProd_2016-03-03T22:00:00.000+08:00_0_0/MiddletiersBillingProd-14-0000-0000]))”,“eventCount”:1000,“exceptionMessage”:“exceeded 1.minutes+30.seconds to druid:firehose:MiddletiersBillingProd-14-0000-0000 while waiting for a response for the request, including retries (if applicable)”}}]
[WARN] 2016-03-03 22:18:33,288 com.metamx.tranquility.beam.ClusteredBeam ? - Emitting alert: [anomaly] Failed to propagate events: druid:prod:overlord/MiddletiersBillingProd
{
“eventCount” : 1000,
“timestamp” : “2016-03-03T22:00:00.000+08:00”,
“beams” : “HashPartitionBeam(DruidBeam(interval = 2016-03-03T22:00:00.000+08:00/2016-03-03T23:00:00.000+08:00, partition = 0, tasks = [index_realtime_MiddletiersBillingProd_2016-03-03T22:00:00.000+08:00_0_0/MiddletiersBillingProd-14-0000-0000]))”
}
com.twitter.finagle.GlobalRequestTimeoutException: exceeded 1.minutes+30.seconds to druid:firehose:MiddletiersBillingProd-14-0000-0000 while waiting for a response for the request, including retries (if applicable)
at com.twitter.finagle.NoStacktrace(Unknown Source)

http://overlord.druid.data.srv/druid/indexer/v1/task/index_realtime_MiddletiersBillingProd_2016-03-04T07%3A00%3A00.000%2B08%3A00_0_0/log:

GroupBys have limits in them about how large results can get. You can override them with configs: http://druid.io/docs/latest/configuration/historical.html#general-query-configuration

Even better though, try to use plyQL (https://github.com/implydata/plyql) and see how it can optimize groupBy queries to multiple topN queries