insane data transfer out from s3

We have 4 historical nodes with about 60gb data each.

We see that this month datatransfer out from s3 for the druid bucket has been over 1,700 GB this month so far (and its only 23rd)!

Using druid 0.8.3.

Is there a reason why druid is transferring so much data from s3?

What is it transferring anyway?

investigating this further, in one of the historicals which has a ton of ‘loading segments’ msgs in the logs, I see a ton of these exceptions:

java.lang.RuntimeException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /druidprod/base/loadQueue/10.0.0.33:8080/dripstat_2016-04-20T20:00:00.000Z_2016-04-20T21:00:00.000Z_2016-04-20T20:00:00.000Z
at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.1.jar: ?]
at io.druid.server.coordination.BaseZkCoordinator$1$1.execute(BaseZkCoordinator.java:139) ~[druid-server-0.8.3.jar:0.8.3 ]
at io.druid.server.coordination.ZkCoordinator.addSegment(ZkCoordinator.java:186) ~[druid-server-0.8.3.jar:0.8.3 ]
at io.druid.server.coordination.SegmentChangeRequestLoad.go(SegmentChangeRequestLoad.java:42) ~[druid-server-0.8.3.jar:0.8.3 ]
at io.druid.server.coordination.BaseZkCoordinator$1.childEvent(BaseZkCoordinator.java:115) [druid-server-0.8.3.jar:0.8.3 ]
at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:518) [curator-recipes-2.9.1.jar: ?]
at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:512) [curator-recipes-2.9.1.jar: ?]
at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92) [curator-framework-2.9.1.jar: ?]
at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) [guava-16.0.1.jar: ?]
at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:83) [curator-framework-2.9.1.jar: ?]
at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:509) [curator-recipes-2.9.1.jar: ?]
at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35) [curator-recipes-2.9.1.jar: ?]
at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:766) [curator-recipes-2.9.1.jar: ?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_77 ]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_77 ]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_77 ]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_77 ]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_77 ]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_77 ]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /druidprod/base/loadQueue/10.0.0.33:8080/dripstat_2016-04-20T20:00:00.000Z_2016-04-20T21:00:00.000Z_2016-04-20T20:00:00.000Z
at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) ~[zookeeper-3.4.6.jar:3.4.6-1569965 ]
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[zookeeper-3.4.6.jar:3.4.6-1569965 ]
at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873) ~[zookeeper-3.4.6.jar:3.4.6-1569965 ]
at org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:242) ~[curator-framework-2.9.1.jar: ?]
at org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:236) ~[curator-framework-2.9.1.jar: ?]
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) ~[curator-client-2.9.1.jar: ?]
at org.apache.curator.framework.imps.DeleteBuilderImpl.pathInForeground(DeleteBuilderImpl.java:232) ~[curator-framework-2.9.1.jar: ?]
at org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:217) ~[curator-framework-2.9.1.jar: ?]
at org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:41) ~[curator-framework-2.9.1.jar: ?]
at io.druid.server.coordination.BaseZkCoordinator$1$1.execute(BaseZkCoordinator.java:126) ~[druid-server-0.8.3.jar:0.8.3 ]
… 18 more

While restarting the historical node seems to have fixed the exception, looking at the logs of all the historical nodes, I see tons of these msgs:

Segment [%s] size is different than expected size.

What does this mean?

I am ingesting through realtime nodes.

Actually it seems even after rebooting i continue getting this exception:

2016-04-24 01:07:30,680 ERROR i.d.s.c.ZkCoordinator [ZkCoordinator-0] Failed to delete zNode[/druidprod/base/loadQueue/10.0.0.33:8080/dripstat_2016-03-09T15:00:00.000Z_2016-03-09T16:00:00.000Z_2016-03-09T15:00:00.000Z], but ignoring exception.

org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /druidprod/base/loadQueue/10.0.0.33:8080/dripstat_2016-03-09T15:00:00.000Z_2016-03-09T16:00:00.000Z_2016-03-09T15:00:00.000Z

at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) ~[zookeeper-3.4.6.jar:3.4.6-1569965]

at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[zookeeper-3.4.6.jar:3.4.6-1569965]

at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873) ~[zookeeper-3.4.6.jar:3.4.6-1569965]

at org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:242) ~[curator-framework-2.9.1.jar:?]

at org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:236) ~[curator-framework-2.9.1.jar:?]

at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) ~[curator-client-2.9.1.jar:?]

at org.apache.curator.framework.imps.DeleteBuilderImpl.pathInForeground(DeleteBuilderImpl.java:232) ~[curator-framework-2.9.1.jar:?]

at org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:217) ~[curator-framework-2.9.1.jar:?]

at org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:41) ~[curator-framework-2.9.1.jar:?]

at io.druid.server.coordination.BaseZkCoordinator$1$1.execute(BaseZkCoordinator.java:133) [druid-server-0.8.3.jar:0.8.3]

at io.druid.server.coordination.ZkCoordinator.addSegment(ZkCoordinator.java:186) [druid-server-0.8.3.jar:0.8.3]

at io.druid.server.coordination.SegmentChangeRequestLoad.go(SegmentChangeRequestLoad.java:42) [druid-server-0.8.3.jar:0.8.3]

at io.druid.server.coordination.BaseZkCoordinator$1.childEvent(BaseZkCoordinator.java:115) [druid-server-0.8.3.jar:0.8.3]

at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:518) [curator-recipes-2.9.1.jar:?]

at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:512) [curator-recipes-2.9.1.jar:?]

at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92) [curator-framework-2.9.1.jar:?]

at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) [guava-16.0.1.jar:?]

at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:83) [curator-framework-2.9.1.jar:?]

at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:509) [curator-recipes-2.9.1.jar:?]

at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35) [curator-recipes-2.9.1.jar:?]

at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:766) [curator-recipes-2.9.1.jar:?]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_77]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_77]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_77]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_77]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_77]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_77]

at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]

2016-04-24 01:07:30,694 ERROR i.d.s.c.ZkCoordinator [ZkCoordinator-0] Exception while removing zNode[/druidprod/base/loadQueue/10.0.0.33:8080/dripstat_2016-03-09T15:00:00.000Z_2016-03-09T16:00:00.000Z_2016-03-09T15:00:00.000Z]

org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /druidprod/base/loadQueue/10.0.0.33:8080/dripstat_2016-03-09T15:00:00.000Z_2016-03-09T16:00:00.000Z_2016-03-09T15:00:00.000Z

at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) ~[zookeeper-3.4.6.jar:3.4.6-1569965]

at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[zookeeper-3.4.6.jar:3.4.6-1569965]

at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873) ~[zookeeper-3.4.6.jar:3.4.6-1569965]

at org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:242) ~[curator-framework-2.9.1.jar:?]

at org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:236) ~[curator-framework-2.9.1.jar:?]

at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) ~[curator-client-2.9.1.jar:?]

at org.apache.curator.framework.imps.DeleteBuilderImpl.pathInForeground(DeleteBuilderImpl.java:232) ~[curator-framework-2.9.1.jar:?]

at org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:217) ~[curator-framework-2.9.1.jar:?]

at org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:41) ~[curator-framework-2.9.1.jar:?]

at io.druid.server.coordination.BaseZkCoordinator$1$1.execute(BaseZkCoordinator.java:126) [druid-server-0.8.3.jar:0.8.3]

at io.druid.server.coordination.ZkCoordinator.addSegment(ZkCoordinator.java:186) [druid-server-0.8.3.jar:0.8.3]

at io.druid.server.coordination.SegmentChangeRequestLoad.go(SegmentChangeRequestLoad.java:42) [druid-server-0.8.3.jar:0.8.3]

at io.druid.server.coordination.BaseZkCoordinator$1.childEvent(BaseZkCoordinator.java:115) [druid-server-0.8.3.jar:0.8.3]

at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:518) [curator-recipes-2.9.1.jar:?]

at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:512) [curator-recipes-2.9.1.jar:?]

at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92) [curator-framework-2.9.1.jar:?]

at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) [guava-16.0.1.jar:?]

at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:83) [curator-framework-2.9.1.jar:?]

at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:509) [curator-recipes-2.9.1.jar:?]

at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35) [curator-recipes-2.9.1.jar:?]

at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:766) [curator-recipes-2.9.1.jar:?]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_77]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_77]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_77]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_77]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_77]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_77]

at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]

2016-04-24 01:07:30,732 ERROR i.d.s.c.ZkCoordinator [ZkCoordinator-0] Failed to delete zNode[/druidprod/base/loadQueue/10.0.0.33:8080/dripstat_2016-03-09T15:00:00.000Z_2016-03-09T16:00:00.000Z_2016-03-09T15:00:00.000Z], but ignoring exception.

org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /druidprod/base/loadQueue/10.0.0.33:8080/dripstat_2016-03-09T15:00:00.000Z_2016-03-09T16:00:00.000Z_2016-03-09T15:00:00.000Z

at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) ~[zookeeper-3.4.6.jar:3.4.6-1569965]

at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[zookeeper-3.4.6.jar:3.4.6-1569965]

at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873) ~[zookeeper-3.4.6.jar:3.4.6-1569965]

at org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:242) ~[curator-framework-2.9.1.jar:?]

at org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:236) ~[curator-framework-2.9.1.jar:?]

at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) ~[curator-client-2.9.1.jar:?]

at org.apache.curator.framework.imps.DeleteBuilderImpl.pathInForeground(DeleteBuilderImpl.java:232) ~[curator-framework-2.9.1.jar:?]

at org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:217) ~[curator-framework-2.9.1.jar:?]

at org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:41) ~[curator-framework-2.9.1.jar:?]

at io.druid.server.coordination.BaseZkCoordinator$1.childEvent(BaseZkCoordinator.java:147) [druid-server-0.8.3.jar:0.8.3]

at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:518) [curator-recipes-2.9.1.jar:?]

at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:512) [curator-recipes-2.9.1.jar:?]

at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92) [curator-framework-2.9.1.jar:?]

at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) [guava-16.0.1.jar:?]

at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:83) [curator-framework-2.9.1.jar:?]

at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:509) [curator-recipes-2.9.1.jar:?]

at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35) [curator-recipes-2.9.1.jar:?]

at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:766) [curator-recipes-2.9.1.jar:?]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_77]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_77]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_77]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_77]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_77]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_77]

at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]

2016-04-24 01:07:30,745 ERROR i.d.s.c.ZkCoordinator [ZkCoordinator-0] Segment load/unload: uncaught exception.: {class=io.druid.server.coordination.ZkCoordinator, exceptionType=class java.lang.RuntimeException, exceptionMessage=org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /druidprod/base/loadQueue/10.0.0.33:8080/dripstat_2016-03-09T15:00:00.000Z_2016-03-09T16:00:00.000Z_2016-03-09T15:00:00.000Z, node=/druidprod/base/loadQueue/10.0.0.33:8080/dripstat_2016-03-09T15:00:00.000Z_2016-03-09T16:00:00.000Z_2016-03-09T15:00:00.000Z, nodeProperties=SegmentChangeRequestLoad{segment=DataSegment{size=59937490, shardSpec=LinearShardSpec{partitionNum=0}, metrics=[count, min, max, callcount, errcount, totaltime, value, traceduration], dimensions=[accountid, appid, extraparentmetricid, jvmid, lowercategory, metricid, metricobjectid, metrictype, middlecategory, minpresent, name, parentName, parentmetricid, uppercategory], version=‘2016-03-09T15:00:00.000Z’, loadSpec={type=s3_zip, bucket=dsprod, key=dripstat/2016-03-09T15:00:00.000Z_2016-03-09T16:00:00.000Z/2016-03-09T15:00:00.000Z/0/index.zip}, interval=2016-03-09T15:00:00.000Z/2016-03-09T16:00:00.000Z, dataSource=‘dripstat’, binaryVersion=‘9’}}}

java.lang.RuntimeException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /druidprod/base/loadQueue/10.0.0.33:8080/dripstat_2016-03-09T15:00:00.000Z_2016-03-09T16:00:00.000Z_2016-03-09T15:00:00.000Z

at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.1.jar:?]

at io.druid.server.coordination.BaseZkCoordinator$1$1.execute(BaseZkCoordinator.java:139) ~[druid-server-0.8.3.jar:0.8.3]

at io.druid.server.coordination.ZkCoordinator.addSegment(ZkCoordinator.java:186) ~[druid-server-0.8.3.jar:0.8.3]

at io.druid.server.coordination.SegmentChangeRequestLoad.go(SegmentChangeRequestLoad.java:42) ~[druid-server-0.8.3.jar:0.8.3]

at io.druid.server.coordination.BaseZkCoordinator$1.childEvent(BaseZkCoordinator.java:115) [druid-server-0.8.3.jar:0.8.3]

at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:518) [curator-recipes-2.9.1.jar:?]

at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:512) [curator-recipes-2.9.1.jar:?]

at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92) [curator-framework-2.9.1.jar:?]

at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) [guava-16.0.1.jar:?]

at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:83) [curator-framework-2.9.1.jar:?]

at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:509) [curator-recipes-2.9.1.jar:?]

at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35) [curator-recipes-2.9.1.jar:?]

at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:766) [curator-recipes-2.9.1.jar:?]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_77]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_77]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_77]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_77]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_77]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_77]

at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]

Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /druidprod/base/loadQueue/10.0.0.33:8080/dripstat_2016-03-09T15:00:00.000Z_2016-03-09T16:00:00.000Z_2016-03-09T15:00:00.000Z

at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) ~[zookeeper-3.4.6.jar:3.4.6-1569965]

at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[zookeeper-3.4.6.jar:3.4.6-1569965]

at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873) ~[zookeeper-3.4.6.jar:3.4.6-1569965]

at org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:242) ~[curator-framework-2.9.1.jar:?]

at org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:236) ~[curator-framework-2.9.1.jar:?]

at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) ~[curator-client-2.9.1.jar:?]

at org.apache.curator.framework.imps.DeleteBuilderImpl.pathInForeground(DeleteBuilderImpl.java:232) ~[curator-framework-2.9.1.jar:?]

at org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:217) ~[curator-framework-2.9.1.jar:?]

at org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:41) ~[curator-framework-2.9.1.jar:?]

at io.druid.server.coordination.BaseZkCoordinator$1$1.execute(BaseZkCoordinator.java:126) ~[druid-server-0.8.3.jar:0.8.3]

… 18 more

created bug report here:

Hi Prashant, did you make changes to your production ZK clusters?