Why the historical node return "timeout" when querying

Hi,
Recently,there are many “timeout” in historical node log.

2015-10-29 14:12:23,116 INFO i.d.q.GroupByParallelQueryRunner [qtp980993229-24] Query timeout, cancelling pending results for query id [d660626b-c00d-4558-bd84-55a30f356845]

The historical node have 64G memory and have 24 cores

The total size of segments for each historical node served is about 16G

jvm setting:

-Xmx10g

-Xms10g

-XX:NewSize=6g

-XX:MaxNewSize=6g

-XX:MaxDirectMemorySize=24g

historical setting:

druid.processing.buffer.sizeBytes=1073741824

druid.processing.numThreads=23

druid.server.http.numThreads=50

druid.segmentCache.locations=[{“path”: “/data/druid/indexCache”, “maxSize”: 400000000000}, {“path”: “/data1/druid/indexCache”, “maxSize”: 640000000000}]

druid.server.maxSize=46707769344

Will the number of segments will effect the query performance?
In fact, I implement the hashed method to generate segments, the target size of rows in segment is about 5000.

The number of hourly data is about 80000, so there are about 160 segments in hour interval.

在 2015年10月30日星期五 UTC+8上午12:46:32,luo…@conew.com写道:

5000 row segments are quite small and not likely to give best performance. Generally something in the millions range is best- 5 million is a good start.

Hi, Gian:
When I do some query, there are some exception, I also confused about it.

2015-10-29 14:25:11,299 ERROR i.d.q.GroupByParallelQueryRunner [processing-0] Exception with one of the sequences!

com.metamx.common.ISE: Null storage adapter found. Probably trying to issue a query against a segment being memory unmapped.

    at io.druid.query.groupby.GroupByQueryEngine.process(GroupByQueryEngine.java:83) ~[druid-processing-0.8.1-rc2.jar:0.8.1-rc2]

    at io.druid.query.groupby.GroupByQueryRunnerFactory$GroupByQueryRunner.run(GroupByQueryRunnerFactory.java:212) ~[druid-processing-0.8.1-rc2.jar:0.8.1-rc2]

    at io.druid.query.ReferenceCountingSegmentQueryRunner.run(ReferenceCountingSegmentQueryRunner.java:49) ~[druid-processing-0.8.1-rc2.jar:0.8.1-rc2]

    at io.druid.query.MetricsEmittingQueryRunner$1.accumulate(MetricsEmittingQueryRunner.java:118) ~[druid-processing-0.8.1-rc2.jar:0.8.1-rc2]

    at io.druid.query.MetricsEmittingQueryRunner$1.accumulate(MetricsEmittingQueryRunner.java:118) ~[druid-processing-0.8.1-rc2.jar:0.8.1-rc2]

    at io.druid.query.spec.SpecificSegmentQueryRunner$2$1.call(SpecificSegmentQueryRunner.java:85) ~[druid-processing-0.8.1-rc2.jar:0.8.1-rc2]

    at io.druid.query.spec.SpecificSegmentQueryRunner.doNamed(SpecificSegmentQueryRunner.java:169) ~[druid-processing-0.8.1-rc2.jar:0.8.1-rc2]

    at io.druid.query.spec.SpecificSegmentQueryRunner.access$400(SpecificSegmentQueryRunner.java:39) ~[druid-processing-0.8.1-rc2.jar:0.8.1-rc2]

    at io.druid.query.spec.SpecificSegmentQueryRunner$2.doItNamed(SpecificSegmentQueryRunner.java:160) ~[druid-processing-0.8.1-rc2.jar:0.8.1-rc2]

    at io.druid.query.spec.SpecificSegmentQueryRunner$2.accumulate(SpecificSegmentQueryRunner.java:78) ~[druid-processing-0.8.1-rc2.jar:0.8.1-rc2]

    at io.druid.query.GroupByParallelQueryRunner$1$1.call(GroupByParallelQueryRunner.java:116) [druid-processing-0.8.1-rc2.jar:0.8.1-rc2]

    at io.druid.query.GroupByParallelQueryRunner$1$1.call(GroupByParallelQueryRunner.java:107) [druid-processing-0.8.1-rc2.jar:0.8.1-rc2]

    at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_79]

在 2015年10月30日星期五 UTC+8上午2:19:15,Gian Merlino写道:

What version of Druid is this? This exception should have been fixed in more recent versions.

Hi, Fangjin:

The version is “0.8.1-rc2”

在 2015年11月2日星期一 UTC+8上午2:49:41,Fangjin Yang写道:

Hmm interesting, how often does this exception occur? Regularly, semi-regularly, or every now and then. If if is one of the first two, if you set druid.segmentCache.dropSegmentDelayMillis=300000 on the historicals, does it help?

It doesn’t happen frequently, I will add this setting as your advice.
Thanks very much.

在 2015年11月5日星期四 UTC+8上午8:26:12,Fangjin Yang写道: