Error on realtime

Hi,

I’ve a Druid 0.9.1.1 with Realtime connected to Kafka 0.8.2

Looking at logs I found some of these errors.

Can someone explain to me what’s happening.

My spec file is set as default described here http://druid.io/docs/latest/development/extensions-core/kafka-eight-firehose.html

Does I need to set rebalance.max.retries and increase to 10?

Here logs:

2017-03-09 04:50:35,576 DEBUG k.c.PartitionTopicInfo [chief-druid_ds1[0]] reset consume offset of druid_ds1:1: fetched offset = 69835032: consumed offset = 69835032 to 69835032

2017-03-09 04:50:35,579 DEBUG k.c.PartitionTopicInfo [ConsumerFetcherThread-druidaws6_aws-druid-realtime4-1489050204423-c3b97f9b-0-0] updated fetch offset of (druid_ds1:0: fetched offset = 69841267: consumed offset = 69841266) to 69841267

2017-03-09 04:50:35,579 DEBUG k.c.PartitionTopicInfo [chief-druid_ds1[0]] reset consume offset of druid_ds1:0: fetched offset = 69841267: consumed offset = 69841267 to 69841267

2017-03-09 04:50:37,286 DEBUG o.a.z.ClientCnxn [main-SendThread(10.80.4.56:2181)] Got ping response for sessionid: 0x55a84804f550258 after 0ms

2017-03-09 04:50:37,591 DEBUG o.a.z.ClientCnxn [chief-druid_ds2[0]-SendThread(10.80.4.56:2181)] Got ping response for sessionid: 0x65a848031810222 after 0ms

2017-03-09 04:50:37,592 DEBUG o.a.z.ClientCnxn [chief-druid_ds3[0]-SendThread(10.80.4.69:2181)] Got ping response for sessionid: 0x65a848031810224 after 0ms

2017-03-09 04:50:37,593 INFO k.c.ConsumerFetcherManager [Thread-71] [ConsumerFetcherManager-1489050204467] Stopping leader finder thread

2017-03-09 04:50:37,593 INFO k.c.ConsumerFetcherManager [Thread-71] [ConsumerFetcherManager-1489050204467] Stopping all fetchers

2017-03-09 04:50:37,593 INFO k.c.ConsumerFetcherManager [Thread-71] [ConsumerFetcherManager-1489050204467] All connections stopped

2017-03-09 04:50:37,593 DEBUG k.c.ZookeeperConsumerConnector [Thread-71] [realtime1-1489050204423-62df2a13], Clearing up queue

2017-03-09 04:50:37,593 DEBUG k.c.ZookeeperConsumerConnector [Thread-71] [realtime1-1489050204423-62df2a13], Cleared queue and sent shutdown command

2017-03-09 04:50:37,593 DEBUG o.I.z.ZkClient [Thread-71] Closing ZkClient…

2017-03-09 04:50:37,593 ERROR k.c.ZookeeperConsumerConnector [realtime1-1489050204423-62df2a13_watcher_executor] [realtime1-1489050204423-62df2a13], error during syncedRebalance

kafka.common.ConsumerRebalanceFailedException: realtime1-1489050204423-62df2a13 can’t rebalance after 4 retries

   at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:633) ~[kafka_2.10-0.8.2.1.jar:?]

   at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anon$1.run(ZookeeperConsumerConnector.scala:551) [kafka_2.10-0.8.2.1.jar:?]

2017-03-09 04:50:37,593 INFO k.c.ZookeeperConsumerConnector [realtime1-1489050204423-62df2a13_watcher_executor] [realtime1-1489050204423-62df2a13], stopping watcher executor thread for consumer realtime1-1489050204423-62df2a13

2017-03-09 04:50:37,593 DEBUG k.c.ConsumerIterator [chief-buck_retargeting[0]] Received the shutdown command

2017-03-09 04:50:37,594 INFO o.I.z.ZkEventThread [ZkClient-EventThread-181-10.80.4.69:2181,10.80.4.6:2181,10.80.4.56:2181] Terminate ZkClient event thread.

2017-03-09 04:50:37,594 DEBUG o.I.z.ZkConnection [Thread-71] Closing ZooKeeper connected to 10.80.4.69:2181,10.80.4.6:2181,10.80.4.56:2181

2017-03-09 04:50:37,594 DEBUG o.a.z.ZooKeeper [Thread-71] Closing session: 0x45a84803157023c

2017-03-09 04:50:37,594 DEBUG o.a.z.ClientCnxn [Thread-71] Closing client for session: 0x45a84803157023c

2017-03-09 04:50:37,595 DEBUG o.a.z.ClientCnxn [chief-druid_ds1[0]-SendThread(10.80.4.56:2181)] Got ping response for sessionid: 0x45a84803157023d after 0ms

2017-03-09 04:50:37,596 DEBUG o.a.z.ClientCnxn [chief-buck_retargeting[0]-SendThread(10.80.4.6:2181)] Got notification sessionid:0x45a84803157023c

2017-03-09 04:50:37,596 DEBUG o.a.z.ClientCnxn [chief-buck_retargeting[0]-SendThread(10.80.4.6:2181)] Got WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/consumers/druidgroup1/ids for sessionid 0x45a84803157023c

2017-03-09 04:50:37,596 DEBUG o.I.z.ZkClient [chief-buck_retargeting[0]-EventThread] Received event: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/consumers/druidgroup1/ids

2017-03-09 04:50:37,596 DEBUG o.a.z.ClientCnxn [chief-druid_ds3[0]-SendThread(10.80.4.69:2181)] Got notification sessionid:0x65a848031810224

2017-03-09 04:50:37,596 DEBUG o.a.z.ClientCnxn [chief-druid_ds3[0]-SendThread(10.80.4.69:2181)] Got WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/consumers/druidgroup1/ids for sessionid 0x65a848031810224

2017-03-09 04:50:37,596 DEBUG o.a.z.ClientCnxn [chief-druid_ds1[0]-SendThread(10.80.4.56:2181)] Got notification sessionid:0x45a84803157023d

2017-03-09 04:50:37,596 DEBUG o.a.z.ClientCnxn [chief-druid_ds1[0]-SendThread(10.80.4.56:2181)] Got WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/consumers/druidgroup1/ids for sessionid 0x45a84803157023d

2017-03-09 04:50:37,596 DEBUG o.a.z.ClientCnxn [chief-buck_retargeting[0]-SendThread(10.80.4.6:2181)] Reading reply sessionid:0x45a84803157023c, packet:: clientPath:null serverPath:null finished:false header:: 270,-11 replyHeader:: 270,124554255009,0 request:: null response:: null

2017-03-09 04:50:37,596 DEBUG o.I.z.ZkClient [chief-druid_ds3[0]-EventThread] Received event: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/consumers/druidgroup1/ids

2017-03-09 04:50:37,596 DEBUG o.a.z.ClientCnxn [chief-druid_ds2[0]-SendThread(10.80.4.56:2181)] Got notification sessionid:0x65a848031810222

2017-03-09 04:50:37,596 DEBUG o.I.z.ZkClient [chief-druid_ds1[0]-EventThread] Received event: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/consumers/druidgroup1/ids

2017-03-09 04:50:37,596 DEBUG o.a.z.ClientCnxn [chief-druid_ds2[0]-SendThread(10.80.4.56:2181)] Got WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/consumers/druidgroup1/ids for sessionid 0x65a848031810222

2017-03-09 04:50:37,596 DEBUG o.I.z.ZkEventThread [chief-druid_ds3[0]-EventThread] New event: ZkEvent[Children of /consumers/druidgroup1/ids changed sent to kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener@47b47ccf]

2017-03-09 04:50:37,596 DEBUG o.a.z.ClientCnxn [Thread-71] Disconnecting client for session: 0x45a84803157023c

2017-03-09 04:50:37,596 DEBUG o.I.z.ZkEventThread [chief-druid_ds1[0]-EventThread] New event: ZkEvent[Children of /consumers/druidgroup1/ids changed sent to kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener@357fe79d]

2017-03-09 04:50:37,596 DEBUG o.I.z.ZkClient [chief-druid_ds3[0]-EventThread] Leaving process event

2017-03-09 04:50:37,596 INFO o.a.z.ZooKeeper [Thread-71] Session: 0x45a84803157023c closed

2017-03-09 04:50:37,596 DEBUG o.I.z.ZkClient [Thread-71] Closing ZkClient…done

2017-03-09 04:50:37,596 DEBUG o.I.z.ZkClient [chief-druid_ds1[0]-EventThread] Leaving process event

2017-03-09 04:50:37,596 INFO k.c.ZookeeperConsumerConnector [Thread-71] [realtime1-1489050204423-62df2a13], ZKConsumerConnector shutdown completed in 20124 ms

2017-03-09 04:50:37,596 DEBUG o.I.z.ZkClient [chief-buck_retargeting[0]-EventThread] ignoring event ‘{NodeChildrenChanged | /consumers/druidgroup1/ids}’ since shutdown triggered

2017-03-09 04:50:37,596 DEBUG o.I.z.ZkEventThread [ZkClient-EventThread-192-10.80.4.69:2181,10.80.4.6:2181,10.80.4.56:2181] Delivering event #43 ZkEvent[Children of /consumers/druidgroup1/ids changed sent to kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener@47b47ccf]

2017-03-09 04:50:37,596 DEBUG o.I.z.ZkClient [chief-buck_retargeting[0]-EventThread] Leaving process event

2017-03-09 04:50:37,596 DEBUG o.I.z.ZkClient [chief-druid_ds2[0]-EventThread] Received event: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/consumers/druidgroup1/ids

2017-03-09 04:50:37,596 DEBUG o.I.z.ZkEventThread [ZkClient-EventThread-193-10.80.4.69:2181,10.80.4.6:2181,10.80.4.56:2181] Delivering event #44 ZkEvent[Children of /consumers/druidgroup1/ids changed sent to kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener@357fe79d]

``

Thanks
Alessandro Conti