ReadTimeoutException in historical nodes

Hi Team,

Observed the read timeout exception in historical nodes. Could you please let me know, the reason for this issue. And fix that can be applied for this issue.

io.druid.java.util.common.RE: Failure getting results for query[null] url[http://servername:portno/druid/v2/] because of [org.jboss.netty.handler.timeout.ReadTimeoutException]

at io.druid.client.DirectDruidClient$JsonParserIterator.init(DirectDruidClient.java:631) ~[druid-server-0.10.1.jar:0.10.1]

at io.druid.client.DirectDruidClient$JsonParserIterator.hasNext(DirectDruidClient.java:563) ~[druid-server-0.10.1.jar:0.10.1]

at io.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:45) ~[java-util-0.10.1.jar:0.10.1]

at io.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43) ~[java-util-0.10.1.jar:0.10.1]

at io.druid.java.util.common.guava.MergeSequence.toYielder(MergeSequence.java:63) ~[java-util-0.10.1.jar:0.10.1]

at io.druid.java.util.common.guava.MergeSequence$2.accumulate(MergeSequence.java:70) ~[java-util-0.10.1.jar:0.10.1]

at io.druid.java.util.common.guava.MergeSequence$2.accumulate(MergeSequence.java:66) ~[java-util-0.10.1.jar:0.10.1]

at io.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:46) ~[java-util-0.10.1.jar:0.10.1]

at io.druid.java.util.common.guava.MergeSequence.toYielder(MergeSequence.java:63) ~[java-util-0.10.1.jar:0.10.1]

at io.druid.java.util.common.guava.LazySequence.toYielder(LazySequence.java:46) ~[java-util-0.10.1.jar:0.10.1]

at io.druid.query.RetryQueryRunner$1.toYielder(RetryQueryRunner.java:108) ~[druid-processing-0.10.1.jar:0.10.1]

at io.druid.common.guava.CombiningSequence.toYielder(CombiningSequence.java:80) ~[druid-common-0.10.1.jar:0.10.1]

at io.druid.java.util.common.guava.MappedSequence.toYielder(MappedSequence.java:49) ~[java-util-0.10.1.jar:0.10.1]

at io.druid.java.util.common.guava.MappedSequence.toYielder(MappedSequence.java:49) ~[java-util-0.10.1.jar:0.10.1]

at io.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:87) ~[java-util-0.10.1.jar:0.10.1]

at io.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:83) ~[java-util-0.10.1.jar:0.10.1]

at io.druid.query.CPUTimeMetricQueryRunner$1.wrap(CPUTimeMetricQueryRunner.java:74) ~[druid-processing-0.10.1.jar:0.10.1]

at io.druid.java.util.common.guava.WrappingSequence.toYielder(WrappingSequence.java:82) ~[java-util-0.10.1.jar:0.10.1]

at io.druid.java.util.common.guava.Yielders.each(Yielders.java:32) ~[java-util-0.10.1.jar:0.10.1]

at io.druid.sql.calcite.schema.DruidSchema.refreshSegmentsForDataSource(DruidSchema.java:411) ~[druid-sql-0.10.1.jar:0.10.1]

at io.druid.sql.calcite.schema.DruidSchema.refreshSegments(DruidSchema.java:377) ~[druid-sql-0.10.1.jar:0.10.1]

at io.druid.sql.calcite.schema.DruidSchema.access$800(DruidSchema.java:81) ~[druid-sql-0.10.1.jar:0.10.1]

at io.druid.sql.calcite.schema.DruidSchema$1.run(DruidSchema.java:184) [druid-sql-0.10.1.jar:0.10.1]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_151]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_151]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_151]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_151]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]

Caused by: java.util.concurrent.ExecutionException: org.jboss.netty.handler.timeout.ReadTimeoutException

at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) ~[guava-16.0.1.jar:?]

at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) ~[guava-16.0.1.jar:?]

at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-16.0.1.jar:?]

at io.druid.client.DirectDruidClient$JsonParserIterator.init(DirectDruidClient.java:601) ~[druid-server-0.10.1.jar:0.10.1]

… 29 more

Caused by: org.jboss.netty.handler.timeout.ReadTimeoutException

at org.jboss.netty.handler.timeout.ReadTimeoutHandler.(ReadTimeoutHandler.java:84) ~[netty-3.10.6.Final.jar:?]

at com.metamx.http.client.NettyHttpClient.go(NettyHttpClient.java:182) ~[http-client-1.1.0.jar:?]

at io.druid.client.DirectDruidClient.run(DirectDruidClient.java:443) ~[druid-server-0.10.1.jar:0.10.1]

at io.druid.client.CachingClusteredClient$5.addSequencesFromServer(CachingClusteredClient.java:540) ~[druid-server-0.10.1.jar:0.10.1]

at io.druid.client.CachingClusteredClient$5.get(CachingClusteredClient.java:433) ~[druid-server-0.10.1.jar:0.10.1]

at io.druid.client.CachingClusteredClient$5.get(CachingClusteredClient.java:427) ~[druid-server-0.10.1.jar:0.10.1]

Thanks,

Mani

Hi Team,

Could you please check on the below issue.

Thanks,

Mani

Hey,

Try setting higher timeouts in the query context. It will override the default query timeouts on broker and historical nodes or you can simply update the query timeout configuration on historical and broker nodes.