Tranquility Kafka - Failed to send request to task: 404 Not Found

Hi,
I’m working on getting a Tranquility Kafka stream ingestion process up and running, and today, I’ve noticed it intermittently seems to have a non fatal error (see stack trace below):

Could you help me understand the significance of this error please? Is it indicating Tranquility has failed to send a batch of events to the indexing task, or something different?

Thanks,

Tim

2017-05-10 15:15:58,621 [finagle/netty3-1] INFO c.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“alerts”,“timestamp”:“2017-05-10T15:15:58.617Z”,“service”:“tranquility”,“host”:“localhost”,“severity”:“anomaly”,“description”:“Failed to propagate events: druid:overlord/data_source”,“data”:{“exceptionType”:“java.io.IOException”,“exceptionStackTrace”:"java.io.IOException: Failed to send request to task[<TASK_ID>]: 404 Not Found

at com.metamx.tranquility.druid.TaskClient$$anonfun$apply$2$$anonfun$apply$3.apply(TaskClient.scala:87)

at com.metamx.tranquility.druid.TaskClient$$anonfun$apply$2$$anonfun$apply$3.apply(TaskClient.scala:73)

at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950)

at com.twitter.util.Try$.apply(Try.scala:13)

at com.twitter.util.Future$.apply(Future.scala:97)

at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950)

at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949)

at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112)

at com.twitter.util.Promise$Transformer.k(Promise.scala:112)

at com.twitter.util.Promise$Transformer.apply(Promise.scala:122)

at com.twitter.util.Promise$Transformer.apply(Promise.scala:103)

at com.twitter.util.Promise$$anon$1.run(Promise.scala:366)

at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178)

at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136)

at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207)

at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92)

at com.twitter.util.Promise.runq(Promise.scala:350)

at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721)

at com.twitter.util.Promise.update(Promise.scala:694)

at com.twitter.util.Promise.setValue(Promise.scala:670)

at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111)

at com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55)

at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)

at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)

at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108)

at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)

at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)

at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)

at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145)

at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)

at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)

at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)

at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)

at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)

at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)

at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)

at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)

at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92)

at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)

at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)

at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142)

at com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78)

at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)

at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)

at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)

at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142)

at com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35)

at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)

at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)

at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)

at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)

at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)

at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)

at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)

at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)

at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)

at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)

at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)

at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)\n",“timestamp”:“2017-05-10T15:00:00.000Z”,“beams”:“MergingPartitioningBeam(DruidBeam(interval = 2017-05-10T15:00:00.000Z/2017-05-10T16:00:00.000Z, partition = 0, tasks = [<…>]))”,“eventCount”:13625,“exceptionMessage”:“Failed to send request to task[<…>]: 404 Not Found”}}]

That’s weird. I’d start by looking at the logs for the task mentioned in the error, around the time the error happened, and see if it looks like it was fully started up or not.