Question/Issue regarding groupbyV2 performance

Hi,

I’m currently evaluating Druid 0.9.2 prior to a planned production rollup to make sure everything is setup and working correctly and while looking into the new groupbyV2 engine I noticed that queries take longer than expected.

The test setup is:

  • groupbyV2 query issued, 2 splits, no filter, 4 metrics
  • 61 segments to be scanned, 4-5 mil records per segment
  • 3 historicals with 7 processing threads each => 3 passes over the data are needed (61 segments / (3*7 processing threads))

The test results show:

  • the segment scan times are around 3 seconds
  • Druid metrics suggest
  • given that three sequential passes over the data are necessary, I would expect the historical query time to be around 9 seconds, but I observe 30 seconds instead.
  • it seems that no time is lost on the broker
  • I profiled one of the historical nodes using jvisualvm
  • Druid first spends 9 seconds scanning segments as expected (3 sequential passes taking 3 seconds each)
  • then it takes another 20 seconds until the query is done.
  • during this time there is moderate CPU utilization, increasing memory allocation as shown in the first attachment below
  • CPU time spent in Jetty’s DeflatedOutputStream.deflate() which delegates to java.util.zip.Deflater.deflate() as shown in screenshots 2 and 3
    I don’t know what this means and what I can do to remedy this issue.
    The query response size isn’t really big, so why should Jetty spend so much time deflating anything?


Any hints would be appreciated.
Thanks

p.s.
I found https://github.com/druid-io/druid/issues/3745
but we are already running stock Druid 0.9.2 on Oracle JDK8 r131

In 0.10.0 you can set druid.broker.http.compressionCodec=identity to disable compression of data from historicals to brokers, that might help. I’d also suggest double checking whether the “query response size isn’t really big” because assuming the profile is accurate, the amount of time gzipping suggests that the query response from historical -> broker is big. Maybe measure the traffic going between the historicals and the broker.

#3745 was more of a permanent stuck, not just being slow, so I don’t think it’s related.

Thanks a lot Gian,

good to know that its possible to switch off compression in 0.10.0.

Thanks also for mentioning that the response size between the historicals and the broker might actually be big. I’ll have a look into this.
I thought that because I used a limit spec, the resultset was small, but you made me realize that that might not actually be the case.

I know that for topN queries the ranking is approximate and that the limit is applied locally on the historicals.
I wonder how this works with the groupby engines. Do historicals always return the full resultset to the brokers? Is there a way to effectively limit the resultset size that historicals return to brokers, perhaps by instructing the engine to behave similar to the topN strategy and give back approximate rankings?

thanks a lot
Sascha

GroupBy limits are (currently, in 0.10.0) applied only on the broker, so data nodes first return the full unlimited result set back to the broker. In a future version of Druid, limits will be pushed down to data nodes in cases where it wouldn’t change the results (like when sorting and limiting on part of the grouping key). You’ll also be able to force limits to be pushed down in all cases through a config, which would make groupBy behave more like topN and potentially return approximate results.

If you have issues with big resultsets from historicals -> brokers then you should upgrade to 0.10.0 ASAP. It has two things that can help: the ability to disable http compression, and also this fix: https://github.com/druid-io/druid/pull/3748 which doubled transfer rate in our testing.