Unable to load data from historicals

Hi,

I am unable to view historical data on broker. I can’t see any error logs on any of the druid nodes.

But i found something like below on

#coordinator log ----------------------------------------

2019-02-20T13:32:17,853 INFO [Coordinator-Exec–0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Load Queues:

2019-02-20T13:32:38,731 INFO [DatabaseSegmentManager-Exec–0] org.apache.druid.metadata.SQLMetadataSegmentManager - Polled and found 11,322 segments in the database

2019-02-20T13:32:47,877 INFO [Coordinator-Exec–0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Starting coordination. Getting available segments.

2019-02-20T13:32:47,883 INFO [Coordinator-Exec–0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Found [11,322] available segments.

2019-02-20T13:32:47,888 WARN [Coordinator-Exec–0] org.apache.druid.server.coordinator.helper.DruidCoordinatorRuleRunner - Uh… I have no servers. Not assigning anything…

2019-02-20T13:32:47,889 INFO [Coordinator-Exec–0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Load Queues:

2019-02-20T13:33:08,000 INFO [DatabaseRuleManager-Exec–0] org.apache.druid.metadata.SQLMetadataRuleManager - Polled and found rules for 1 datasource(s)

2019-02-20T13:33:17,910 INFO [Coordinator-Exec–0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Starting coordination. Getting available segments.

2019-02-20T13:33:17,916 INFO [Coordinator-Exec–0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Found [11,322] available segments.

2019-02-20T13:33:17,921 WARN [Coordinator-Exec–0] org.apache.druid.server.coordinator.helper.DruidCoordinatorRuleRunner - Uh… I have no servers. Not assigning anything…

2019-02-20T13:33:17,922 INFO [Coordinator-Exec–0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Load Queues:

2019-02-20T13:33:40,147 INFO [DatabaseSegmentManager-Exec–0] org.apache.druid.metadata.SQLMetadataSegmentManager - Polled and found 11,322 segments in the database

2019-02-20T13:33:47,945 INFO [Coordinator-Exec–0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Starting coordination. Getting available segments.

2019-02-20T13:33:47,951 INFO [Coordinator-Exec–0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Found [11,322] available segments.

2019-02-20T13:33:47,957 WARN [Coordinator-Exec–0] org.apache.druid.server.coordinator.helper.DruidCoordinatorRuleRunner - Uh… I have no servers. Not assigning anything…

2019-02-20T13:33:47,957 INFO [Coordinator-Exec–0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Load Queues:

2019-02-20T13:34:08,002 INFO [DatabaseRuleManager-Exec–0] org.apache.druid.metadata.SQLMetadataRuleManager - Polled and found rules for 1 datasource(s)

2019-02-20T13:34:12,110 INFO [LookupCoordinatorManager–2] org.apache.druid.server.lookup.cache.LookupCoordinatorManager - Not updating lookups because no data exists

2019-02-20T13:34:17,979 INFO [Coordinator-Exec–0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Starting coordination. Getting available segments.

2019-02-20T13:34:17,986 INFO [Coordinator-Exec–0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Found [11,322] available segments.

2019-02-20T13:34:17,993 WARN [Coordinator-Exec–0] org.apache.druid.server.coordinator.helper.DruidCoordinatorRuleRunner - Uh… I have no servers. Not assigning anything…

#Historical log--------------------

2019-02-20T13:30:24,119 INFO [main-SendThread(10.50.68.55:2181)] org.apache.zookeeper.ClientCnxn - Opening socket connection to server 10.50.68.55/10.50.68.55:2181. Will not attempt to authenticate using SASL (unknown error)

2019-02-20T13:33:54,558 INFO [main-SendThread(10.50.68.55:2181)] org.apache.zookeeper.ClientCnxn - Socket connection established to 10.50.68.55/10.50.68.55:2181, initiating session

2019-02-20T13:36:57,195 WARN [main-SendThread(10.50.68.55:2181)] org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 202267ms for sessionid 0x101056676c8282a

2019-02-20T13:37:04,237 INFO [main-SendThread(10.50.68.55:2181)] org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 202267ms for sessionid 0x101056676c8282a, closing socket connection and attempting reconnect

Please suggest solution for this.

It sounds like your Historicals are having trouble maintaining a connection to ZK. It most commonly happens in case of network issues or in case of the Historical JVMs getting blocked on garbage collections or the like.