Emitting alert: [anomaly] Failed to propagate events: druid:overlord/xxx (500 Internal Server Error)

Hi Fangjin,

I’m stuck on Emitting alert: [anomaly] Failed to propagate events: druid:overlord/xxx (500 Internal Server Error), I googled relative issues, but I did not found any useful answer. Could you help to solve this issue?

2018-05-18 16:58:46,153 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@614da024 checkpointed with 0 Records and 0 Swap Files in 14 milliseconds (Stop-the-world time = 10 milliseconds, Clear Edit Logs time = 3 millis), max Transaction ID -1

2018-05-18 16:59:02,057 INFO [pool-10-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository

2018-05-18 16:59:02,058 INFO [pool-10-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 26 records in 0 milliseconds

2018-05-18 16:59:21,798 WARN [finagle/netty3-16] c.metamx.tranquility.beam.ClusteredBeam Emitting alert: [anomaly] Failed to propagate events: druid:overlord/tweet_demo

{

“eventCount” : 1,

“timestamp” : “2018-05-18T08:50:00.000Z”,

“beams” : “MergingPartitioningBeam(DruidBeam(interval = 2018-05-18T08:50:00.000Z/2018-05-18T09:00:00.000Z, partition = 0, tasks = [index_realtime_tweet_demo_2018-05-18T08:50:00.000Z_0_0/tweet_demo-170-0000-0000]))”

}

java.io.IOException: Failed to send request to task[index_realtime_tweet_demo_2018-05-18T08:50:00.000Z_0_0]: 500 Internal Server Error

at com.metamx.tranquility.druid.TaskClient$$anonfun$apply$2$$anonfun$apply$3.apply(TaskClient.scala:87)

at com.metamx.tranquility.druid.TaskClient$$anonfun$apply$2$$anonfun$apply$3.apply(TaskClient.scala:73)

at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950)

at com.twitter.util.Try$.apply(Try.scala:13)

at com.twitter.util.Future$.apply(Future.scala:97)

at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950)

at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949)

at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112)

at com.twitter.util.Promise$Transformer.k(Promise.scala:112)

at com.twitter.util.Promise$Transformer.apply(Promise.scala:122)

at com.twitter.util.Promise$Transformer.apply(Promise.scala:103)

at com.twitter.util.Promise$$anon$1.run(Promise.scala:366)

at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178)

at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136)

at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207)

at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92)

at com.twitter.util.Promise.runq(Promise.scala:350)

at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721)

at com.twitter.util.Promise.update(Promise.scala:694)

at com.twitter.util.Promise.setValue(Promise.scala:670)

at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111)

at com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55)

at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)

at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)

at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108)

at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)

at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)

at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)

at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145)

at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)

at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)

at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)

at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)

at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)

at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)

at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)

at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)

at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92)

at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)

at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)

at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142)

at com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78)

at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)

at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)

at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)

at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142)

at com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35)

at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)

at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)

at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)

at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)

at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)

at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)

at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)

at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)

at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)

at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)

at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)

at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

2018-05-18 16:59:21,800 INFO [finagle/netty3-16] com.metamx.emitter.core.LoggingEmitter Event [{“feed”:“alerts”,“timestamp”:“2018-05-18T16:59:21.800+08:00”,“service”:“tranquility”,“host”:“localhost”,“severity”:“anomaly”,“description”:“Failed to propagate events: druid:overlord/tweet_demo”,“data”:{“exceptionType”:“java.io.IOException”,“exceptionStackTrace”:“java.io.IOException: Failed to send request to task[index_realtime_tweet_demo_2018-05-18T08:50:00.000Z_0_0]: 500 Internal Server Error\n\tat com.metamx.tranquility.druid.TaskClient$$anonfun$apply$2$$anonfun$apply$3.apply(TaskClient.scala:87)\n\tat com.metamx.tranquility.druid.TaskClient$$anonfun$apply$2$$anonfun$apply$3.apply(TaskClient.scala:73)\n\tat com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950)\n\tat com.twitter.util.Try$.apply(Try.scala:13)\n\tat com.twitter.util.Future$.apply(Future.scala:97)\n\tat com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950)\n\tat com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949)\n\tat com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112)\n\tat com.twitter.util.Promise$Transformer.k(Promise.scala:112)\n\tat com.twitter.util.Promise$Transformer.apply(Promise.scala:122)\n\tat com.twitter.util.Promise$Transformer.apply(Promise.scala:103)\n\tat com.twitter.util.Promise$$anon$1.run(Promise.scala:366)\n\tat com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178)\n\tat com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136)\n\tat com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207)\n\tat com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92)\n\tat com.twitter.util.Promise.runq(Promise.scala:350)\n\tat com.twitter.util.Promise.updateIfEmpty(Promise.scala:721)\n\tat com.twitter.util.Promise.update(Promise.scala:694)\n\tat com.twitter.util.Promise.setValue(Promise.scala:670)\n\tat com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111)\n\tat com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108)\n\tat org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145)\n\tat org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)\n\tat org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)\n\tat org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)\n\tat org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)\n\tat org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)\n\tat org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142)\n\tat com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78)\n\tat org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142)\n\tat com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35)\n\tat org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)\n\tat org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)\n\tat org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)\n\tat org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)\n\tat org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)\n\tat org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)\n\tat org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)\n\tat org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)\n\tat org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)\n\tat org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\n”,“timestamp”:“2018-05-18T08:50:00.000Z”,“beams”:“MergingPartitioningBeam(DruidBeam(interval = 2018-05-18T08:50:00.000Z/2018-05-18T09:00:00.000Z, partition = 0, tasks = [index_realtime_tweet_demo_2018-05-18T08:50:00.000Z_0_0/tweet_demo-170-0000-0000]))”,“eventCount”:1,“exceptionMessage”:“Failed to send request to task[index_realtime_tweet_demo_2018-05-18T08:50:00.000Z_0_0]: 500 Internal Server Error”}}]

2018-05-18 16:59:21,801 DEBUG [finagle/netty3-16] o.a.nifi.processors.druid.PutDruidRecord PutDruidRecord[id=95c5b287-61ff-3d53-2e0a-8a7cd874bc9a] [Terminated Process] - DEBUG - Record Dropped due to MessageDroppedException: Message dropped, transferring record to dropped.: com.metamx.tranquility.tranquilizer.MessageDroppedException: Message dropped

com.metamx.tranquility.tranquilizer.MessageDroppedException: Message dropped

at com.twitter.finagle.NoStacktrace(Unknown Source)

2018-05-18 16:59:21,803 INFO [Timer-Driven Process Thread-2] o.a.n.c.r.StandardProcessSession StandardFlowFileRecord[uuid=fa7245dd-68a1-473b-b5c3-19472a78c11a,claim=,offset=0,name=2553854205888.json,size=0] terminated by PutDruidRecord[id=95c5b287-61ff-3d53-2e0a-8a7cd874bc9a]; life of FlowFile = 65132 ms

2018-05-18 16:59:21,803 INFO [Timer-Driven Process Thread-2] o.a.n.c.r.StandardProcessSession StandardFlowFileRecord[uuid=a55501c9-73c2-40f5-a23b-9c139b02fb86,claim=,offset=0,name=2553854205888.json,size=0] terminated by PutDruidRecord[id=95c5b287-61ff-3d53-2e0a-8a7cd874bc9a]; life of FlowFile = 65132 ms

2018-05-18 16:59:21,803 INFO [Timer-Driven Process Thread-2] o.a.n.c.r.StandardProcessSession StandardFlowFileRecord[uuid=fa037d59-8d72-4fad-90ee-0d978f167a99,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1526631426651-1, container=default, section=1], offset=14904, length=362],offset=0,name=2553854205888.json,size=362] terminated by PutDruidRecord[id=95c5b287-61ff-3d53-2e0a-8a7cd874bc9a]; life of FlowFile = 259779 ms

2018-05-18 16:59:21,805 INFO [Timer-Driven Process Thread-2] o.a.n.c.r.StandardProcessSession StandardProcessSession[id=371] for PutDruidRecord[id=95c5b287-61ff-3d53-2e0a-8a7cd874bc9a], committed the following events: created FlowFiles [1117c61c-a963-4739-8616-9692ad682ff3, fa7245dd-68a1-473b-b5c3-19472a78c11a, a55501c9-73c2-40f5-a23b-9c139b02fb86], modified FlowFiles [1117c61c-a963-4739-8616-9692ad682ff3], removed FlowFiles [fa7245dd-68a1-473b-b5c3-19472a78c11a, fa037d59-8d72-4fad-90ee-0d978f167a99, a55501c9-73c2-40f5-a23b-9c139b02fb86], Transferred FlowFiles [1117c61c-a963-4739-8616-9692ad682ff3] to ‘dropped’, Processing Time = 65 seconds, 132 millis (65132746154 nanos)

2018-05-18 16:59:21,805 DEBUG [Timer-Driven Process Thread-2] o.a.n.c.r.StandardProcessSession Session commit for StandardProcessSession[id=371] [PutDruidRecord[id=95c5b287-61ff-3d53-2e0a-8a7cd874bc9a]] took 3 millis (3883504 nanos); FlowFile Repository Update took 1 millis (1362417 nanos); Claim Removal took 361886 nanos (361886 nanos); FlowFile Event Update took 44055 nanos (44055 nanos); Enqueuing FlowFiles took 86937 nanos (86937 nanos); Updating Provenance Event Repository took 1 millis (1303459 nanos)

2018-05-18 16:59:21,807 DEBUG [Timer-Driven Process Thread-2] o.a.nifi.processors.druid.PutDruidRecord PutDruidRecord[id=95c5b287-61ff-3d53-2e0a-8a7cd874bc9a] [Terminated Process] - DEBUG - Tranquilizer Status: Open

2018-05-18 16:59:21,808 DEBUG [Timer-Driven Process Thread-2] o.a.nifi.processors.druid.PutDruidRecord PutDruidRecord[id=95c5b287-61ff-3d53-2e0a-8a7cd874bc9a] [Terminated Process] - DEBUG - Sent Payload to Druid: {msg=RT @jrthguion: Prix Nobel de la Forfaiture pour @MichelKafando :«J’aurai tout fait pour que la paix règne dans ce pays». Humour ou ironie?…, created_time=Fri May 18 08:55:03 +0000 2018, created_unixtime=1526633703408, tweet_id=997400159196762113, lang=fr, time_zone=London, username=Burkina BOT :burkina_faso:}

2018-05-18 16:59:21,812 DEBUG [Timer-Driven Process Thread-9] o.a.n.processors.standard.LogAttribute LogAttribute[id=0c0f75cc-9981-3c1a-a6e2-c5839439cf26] logging for flow file StandardFlowFileRecord[uuid=1117c61c-a963-4739-8616-9692ad682ff3,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1526631426760-2, container=default, section=2], offset=4086, length=320],offset=0,name=2553854205888.json,size=320]