Tranquility Servers behind Load Balancer

When I had a single Tranquility server, things are working fine. As soon as I place an additional server and place both of them behind a Load Balancer. I am noticing error on the tranquility server log. Also notice all the tranquility post requests are returning 500 status. What could be wrong? Thanks.

On my tranquility server.json I have the zookeeper.connect set to 3 IP addresses used for the zk. e.g. “zk1_ip, zk2_ip, zk3_ip”

2019-05-30 22:43:09,626 [ClusteredBeam-ZkFuturePool-37be496d-a8ca-49dc-a31d-000645e87360] INFO c.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“alerts”,“timestamp”:“2019-05-30T22:43:09.626Z”,“service”:“tranquility”,“host”:“localhost”,“severity”:“anomaly”,“description”:“Failed to create merged beam: druid:overlord/_event_stats”,“data”:{“exceptionType”:“java.lang.IllegalStateException”,“exceptionMessage”:“Failed to save new beam for identifier[druid:overlord/_event_stats] timestamp[2019-05-30T22:00:00.000Z]”,“exceptionStackTrace”:“java.lang.IllegalStateException: Failed to save new beam for identifier[druid:overlord/_event_stats] timestamp[2019-05-30T22:00:00.000Z]\n\tat com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:285)\n\tat com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:282)\n\tat com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:1119)\n\tat com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:1117)\n\tat com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:107)\n\tat com.twitter.util.Promise$Transformer.k(Promise.scala:107)\n\tat com.twitter.util.Promise$Transformer.apply(Promise.scala:117)\n\tat com.twitter.util.Promise$Transformer.apply(Promise.scala:98)\n\tat com.twitter.util.Promise$$anon$1.run(Promise.scala:421)\n\tat com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:200)\n\tat com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:158)\n\tat com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:272)\n\tat com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:108)\n\tat com.twitter.util.Promise.runq(Promise.scala:405)\n\tat com.twitter.util.Promise.updateIfEmpty(Promise.scala:801)\n\tat com.twitter.util.ExecutorServiceFuturePool$$anon$4.run(FuturePool.scala:141)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: com.metamx.tranquility.druid.IndexServicePermanentException: Service[druid:overlord] call failed with status: 400 Bad Request\n\tat com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$18.apply(IndexService.scala:167)\n\tat com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$18.apply(IndexService.scala:141)\n\tat com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$3.apply(Future.scala:1145)\n\tat com.twitter.util.Try$.apply(Try.scala:15)\n\tat com.twitter.util.Future$.apply(Future.scala:163)\n\tat com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:1145)\n\tat com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:1144)\n\tat com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:107)\n\tat com.twitter.util.Promise$Transformer.k(Promise.scala:107)\n\tat com.twitter.util.Promise$Transformer.apply(Promise.scala:117)\n\tat com.twitter.util.Promise$Transformer.apply(Promise.scala:98)\n\tat com.twitter.util.Promise$$anon$1.run(Promise.scala:421)\n\tat com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:200)\n\tat com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:158)\n\tat com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:272)\n\tat com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:108)\n\tat com.twitter.util.Promise.runq(Promise.scala:405)\n\tat com.twitter.util.Promise.updateIfEmpty(Promise.scala:801)\n\tat com.twitter.util.Promise.update(Promise.scala:775)\n\tat com.twitter.util.Promise.setValue(Promise.scala:751)\n\tat com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:120)\n\tat com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:56)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108)\n\tat org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:124)\n\tat org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)\n\tat org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:194)\n\tat org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)\n\tat org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)\n\tat org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)\n\tat org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)\n\tat org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)\n\tat org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142)\n\tat com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:68)\n\tat org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142)\n\tat com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:32)\n\tat org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)\n\tat org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)\n\tat org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)\n\tat org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)\n\tat org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)\n\tat org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)\n\tat org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)\n\tat org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)\n\tat org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)\n\tat org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat com.twitter.finagle.util.BlockingTimeTrackingThreadFactory$$anon$1.run(BlockingTimeTrackingThreadFactory.scala:24)\n\t… 1 more\n”}}]

2019-05-30 22:43:09,626 [qtp22006205-41] WARN c.m.t.server.http.TranquilityServlet - Server error serving request to http://LB-name.us-east-1.elb.amazonaws.com:8200/v1/post/_event_stats

java.lang.IllegalStateException: Failed to create merged beam: druid:overlord/_event_stats

    at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$sendAll$2$$anonfun$26.apply(ClusteredBeam.scala:379) ~[io.druid.tranquility-core-0.8.3.jar:0.8.3]

    at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$sendAll$2$$anonfun$26.apply(ClusteredBeam.scala:375) ~[io.druid.tranquility-core-0.8.3.jar:0.8.3]

    at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:107) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0]

    at com.twitter.util.Promise$Transformer.k(Promise.scala:107) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0]

    at com.twitter.util.Promise$Transformer.apply(Promise.scala:117) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0]

    at com.twitter.util.Promise$Transformer.apply(Promise.scala:98) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0]

    at com.twitter.util.Promise$$anon$1.run(Promise.scala:421) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0]

    at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:200) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0]

    at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:158) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0]

    at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:272) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0]

    at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:108) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0]

    at com.twitter.util.Promise.runq(Promise.scala:405) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0]

    at com.twitter.util.Promise.updateIfEmpty(Promise.scala:801) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0]

    at com.twitter.util.ExecutorServiceFuturePool$$anon$4.run(FuturePool.scala:141) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0]

    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_201]

    at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_201]

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_201]

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_201]

    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_201]

Caused by: java.lang.IllegalStateException: Failed to save new beam for identifier[druid:overlord/_event_stats] timestamp[2019-05-30T22:00:00.000Z]

    at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:285) ~[io.druid.tranquility-core-0.8.3.jar:0.8.3]

    at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:282) ~[io.druid.tranquility-core-0.8.3.jar:0.8.3]

    at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:1119) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0]

    at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:1117) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0]

    ... 17 common frames omitted

Caused by: com.metamx.tranquility.druid.IndexServicePermanentException: Service[druid:overlord] call failed with status: 400 Bad Request

    at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$18.apply(IndexService.scala:167) ~[io.druid.tranquility-core-0.8.3.jar:0.8.3]

    at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$18.apply(IndexService.scala:141) ~[io.druid.tranquility-core-0.8.3.jar:0.8.3]

    at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$3.apply(Future.scala:1145) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0]

    at com.twitter.util.Try$.apply(Try.scala:15) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0]

    at com.twitter.util.Future$.apply(Future.scala:163) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0]

    at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:1145) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0]

    at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:1144) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0]

    at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:107) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0]

    at com.twitter.util.Promise$Transformer.k(Promise.scala:107) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0]

    at com.twitter.util.Promise$Transformer.apply(Promise.scala:117) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0]

    at com.twitter.util.Promise$Transformer.apply(Promise.scala:98) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0]

    at com.twitter.util.Promise$$anon$1.run(Promise.scala:421) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0]

    at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:200) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0]

    at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:158) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0]

    at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:272) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0]

    at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:108) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0]

    at com.twitter.util.Promise.runq(Promise.scala:405) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0]

    at com.twitter.util.Promise.updateIfEmpty(Promise.scala:801) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0]

    at com.twitter.util.Promise.update(Promise.scala:775) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0]

    at com.twitter.util.Promise.setValue(Promise.scala:751) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0]

    at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:120) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0]

    at com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:56) ~[com.twitter.finagle-core_2.11-6.43.0.jar:6.43.0]

    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na]

    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na]

    at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) ~[io.netty.netty-3.10.5.Final.jar:na]

    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na]

    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na]

    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na]

    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:124) ~[io.netty.netty-3.10.5.Final.jar:na]

    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na]

    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na]

    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na]

    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[io.netty.netty-3.10.5.Final.jar:na]

    at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:194) ~[io.netty.netty-3.10.5.Final.jar:na]

    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na]

    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na]

    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na]

    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[io.netty.netty-3.10.5.Final.jar:na]

    at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) ~[io.netty.netty-3.10.5.Final.jar:na]

    at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) ~[io.netty.netty-3.10.5.Final.jar:na]

    at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) ~[io.netty.netty-3.10.5.Final.jar:na]

    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na]

    at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) ~[io.netty.netty-3.10.5.Final.jar:na]

    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na]

    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na]

    at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na]

    at com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:68) ~[com.twitter.finagle-core_2.11-6.43.0.jar:6.43.0]

    at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na]

    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na]

    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na]

    at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na]

    at com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:32) ~[com.twitter.finagle-core_2.11-6.43.0.jar:6.43.0]

    at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na]

    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na]

    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) ~[io.netty.netty-3.10.5.Final.jar:na]

    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) ~[io.netty.netty-3.10.5.Final.jar:na]

    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) ~[io.netty.netty-3.10.5.Final.jar:na]

    at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) ~[io.netty.netty-3.10.5.Final.jar:na]

    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) ~[io.netty.netty-3.10.5.Final.jar:na]

    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) ~[io.netty.netty-3.10.5.Final.jar:na]

    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) ~[io.netty.netty-3.10.5.Final.jar:na]

    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[io.netty.netty-3.10.5.Final.jar:na]

    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[io.netty.netty-3.10.5.Final.jar:na]

    at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) ~[io.netty.netty-3.10.5.Final.jar:na]

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_201]

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_201]

    at com.twitter.finagle.util.BlockingTimeTrackingThreadFactory$$anon$1.run(BlockingTimeTrackingThreadFactory.scala:24) ~[com.twitter.finagle-core_2.11-6.43.0.jar:6.43.0]

    ... 1 common frames omitted

``

Correction, there are combination of 200 and 500 to the Post requests to the Tranquility.

Would like to hear any feedback if you place tranquility servers behind the load balancer. Want to know if i misconfigured something. Thanks.

It looks like tranquility server is getting an error from the overlord, causing the event post request to tranquility to fail, you could check the overlord logs to see if there’s anything useful there.