Historical Node issues

Hi all,

When I try to query my Historical Node all that comes back is an empty json array. Realtime node is working fine and segments are getting pushed to s3 but it doesn’t seem like Historical nodes are picking up that s3 data. I checked logs for my Historicals and there are no exceptions. However, in my Coordinator node I am getting this zookeeper exceptions:

org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /druid/sandbox/loadQueue/10.65.10.74:8083/wpUserInteraction_2015-10-12T07:15:00.000Z_2015-10-12T07:30:00.000Z_2015-10-12T07:15:00.000Z

at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) ~[zookeeper-3.4.6.jar:3.4.6-1569965]

at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[zookeeper-3.4.6.jar:3.4.6-1569965]

at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) ~[zookeeper-3.4.6.jar:3.4.6-1569965]

at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:691) ~[curator-framework-2.7.0.jar:?]

at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:675) ~[curator-framework-2.7.0.jar:?]

What could this exception mean? Is my Coordinator not able to find my Historical Nodes?

Thanks,

to be honest, my historical node logs say absolutely nothing.

yay. restarted historical node. seems like coordinator couldnt find them.

Hi Nicolas, what version of Druid was this? I know a similar class of problems to this was more common before but I haven’t seen these problems for some time now.

Hey Fangjin,

So yeah this happened again. Coordinator logs are filled with this exception. I will restart my historicals and see if they go away again.

I am using druid 0.8.0

I’ll post here if i find anything else

thanks,

Hi,

If this is happening often then I would doubt zookeeper configuration and zookeeper itself (try using latest stable version of zookeeper). Inside druid runtime.properties for each druid process, property “druid.zk.service.host” must have all zookeeper host names. And, each zookeeper node must be configured to know about all the zookeeper nodes in the system.(via server.x properties).

– Himanshu