Ingest to tranquility server error

Hi,

I am having a terrible time reading through the online docs. Esp when it comes to ingesting. When I post to tranquility servers the errors are all of the place. The full trace is at the very bottom. From, the docs…what am I missing?

This is the data I am trying to ingest to this url : http://tranquilty-server.com:8200/v1/post/foo

{‘timestamp’: ‘2019-03-24T12:06:46.000Z’, ‘impressions’: 1, ‘dim1’: ‘1’, ‘dim2’: ‘2’, ‘dim3’: ‘3’}

I assume I resolve this error things will be better. What is the issue?

2019-03-24 09:24:43,435 [ClusteredBeam-ZkFuturePool-54281ecc-49cb-4108-bcc9-5eeb9250b8e6] INFO c.m.c.s.net.finagle.DiscoResolver - Updating instances for service[druid:overlord] to Set(ServiceInstance{name=‘druid:overlord’, id=‘ccdeaa35-7992-4b55-8b09-5164db19bd76’, address=‘172.31.37.177’, port=8090, sslPort=-1, payload=null, registrationTimeUTC=1551772526993, serviceType=DYNAMIC, uriSpec=null})

2019-03-24 09:24:43,573 [ClusteredBeam-ZkFuturePool-54281ecc-49cb-4108-bcc9-5eeb9250b8e6] INFO c.m.t.finagle.FinagleRegistry - Created client for service: disco!druid:overlord

ERROR com.metamx.common.scala.control$ - Non-retryable response for uri[/druid/indexer/v1/task] with content: {“error”:“Instantiation of [simple type, class org.apache.druid.segment.indexing.RealtimeTuningConfig] value failed: Failed to create directory within 10000 attempts (tried 1553419484189-0 to 1553419484189-9999) (through reference chain: org.apache.druid.indexing.common.task.RealtimeIndexTask[“spec”]->org.apache.druid.segment.realtime.FireDepartment[“tuningConfig”])”}

2019-03-24 09:24:44,262 [ClusteredBeam-ZkFuturePool-54281ecc-49cb-4108-bcc9-5eeb9250b8e6] ERROR c.m.tranquility.beam.ClusteredBeam - Failed to update cluster state: druid:overlord/foo

com.metamx.tranquility.druid.IndexServicePermanentException: Service[druid:overlord] call failed with status: 400 Bad Request

Here is the error when I post to tranquility server:

Server error

At the very bottom are the logs

I have ever server setup on dedicated servers… I get no errors on start up:

historical

middleManager

coordinator

broker

overlord

kafka

tranquility

Here is overload runtime.properties

druid.service=druid/overlord

druid.plaintextPort=8090

druid.indexer.queue.startDelay=PT30S

druid.indexer.runner.type=remote

druid.indexer.storage.type=metadata

Here is the tranqulity server.json file:

{

“dataSources” : [

{

“spec” : {

“dataSchema” : {

“dataSource” : “foo”,

“metricsSpec” : [

{ “type” : “count”, “name” : “impressions” },

{ “type” : “doubleSum”, “fieldName” : “cpi”, “name” : “cpi” }

],

“granularitySpec” : {

“segmentGranularity” : “hour”,

“queryGranularity” : “none”,

“type” : “uniform”

},

“parser” : {

“type” : “string”,

“parseSpec” : {

“format” : “json”,

“timestampSpec” : { “column” : “timestamp”, “format” : “auto” },

“dimensionsSpec” : {

“dimensions” : [“dim1”, “dim2”, “dim3”]

}

}

}

},

“tuningConfig” : {

“type” : “realtime”,

“windowPeriod” : “PT10M”,

“intermediatePersistPeriod” : “PT10M”,

“maxRowsInMemory” : “100000”

}

},

“properties” : {

“task.partitions” : “1”,

“task.replicants” : “1”

}

}

],

“properties” : {

“zookeeper.connect” : “localhost”

}

}

2019-03-24 09:24:42,780 [ClusteredBeam-ZkFuturePool-54281ecc-49cb-4108-bcc9-5eeb9250b8e6] WARN org.apache.curator.utils.ZKPaths - The version of ZooKeeper being used doesn’t support Container nodes. CreateMode.PERSISTENT will be used instead.
2019-03-24 09:24:42,860 [ClusteredBeam-ZkFuturePool-54281ecc-49cb-4108-bcc9-5eeb9250b8e6] INFO c.m.tranquility.beam.ClusteredBeam - Creating new merged beam for identifier[druid:overlord/foo] timestamp[2019-03-24T09:00:00.000Z] (target = 1, actual = 0)
2019-03-24 09:24:43,068 [ClusteredBeam-ZkFuturePool-54281ecc-49cb-4108-bcc9-5eeb9250b8e6] INFO com.metamx.common.scala.control$ - Creating druid indexing task (service = druid:overlord): {
“type” : “index_realtime”,
“id” : “index_realtime_foo_2019-03-24T09:00:00.000Z_0_0”,
“resource” : {
“availabilityGroup” : “foo-2019-03-24T09:00:00.000Z-0000”,
“requiredCapacity” : 1
},
“spec” : {
“dataSchema” : {
“dataSource” : “foo”,
“parser” : {
“type” : “map”,
“parseSpec” : {
“format” : “json”,
“timestampSpec” : {
“column” : “timestamp”,
“format” : “millis”,
“missingValue” : null
},
“dimensionsSpec” : {
“dimensions” : [ “dim1”, “dim2”, “dim3” ],
“spatialDimensions” :
}
}
},
“metricsSpec” : [ {
“type” : “count”,
“name” : “impressions”
}, {
“type” : “doubleSum”,
“name” : “cpi”,
“fieldName” : “cpi”
} ],
“granularitySpec” : {
“type” : “uniform”,
“segmentGranularity” : “HOUR”,
“queryGranularity” : {
“type” : “none”
}
}
},
“ioConfig” : {
“type” : “realtime”,
“plumber” : null,
“firehose” : {
“type” : “clipped”,
“interval” : “2019-03-24T09:00:00.000Z/2019-03-24T10:00:00.000Z”,
“delegate” : {
“type” : “timed”,
“shutoffTime” : “2019-03-24T10:15:00.000Z”,
“delegate” : {
“type” : “receiver”,
“serviceName” : “firehose:druid:overlord:foo-009-0000-0000”,
“bufferSize” : 100000
}
}
}
},
“tuningConfig” : {
“shardSpec” : {
“type” : “linear”,
“partitionNum” : 0
},
“rejectionPolicy” : {
“type” : “none”
},
“buildV9Directly” : false,
“maxPendingPersists” : 0,
“intermediatePersistPeriod” : “PT10M”,
“windowPeriod” : “PT10M”,
“type” : “realtime”,
“maxRowsInMemory” : “100000”
}
}
}
2019-03-24 09:24:43,435 [ClusteredBeam-ZkFuturePool-54281ecc-49cb-4108-bcc9-5eeb9250b8e6] INFO c.m.c.s.net.finagle.DiscoResolver - Updating instances for service[druid:overlord] to Set(ServiceInstance{name=‘druid:overlord’, id=‘ccdeaa35-7992-4b55-8b09-5164db19bd76’, address=‘172.31.37.177’, port=8090, sslPort=-1, payload=null, registrationTimeUTC=1551772526993, serviceType=DYNAMIC, uriSpec=null})
2019-03-24 09:24:43,573 [ClusteredBeam-ZkFuturePool-54281ecc-49cb-4108-bcc9-5eeb9250b8e6] INFO c.m.t.finagle.FinagleRegistry - Created client for service: disco!druid:overlord
2019-03-24 09:24:44,254 [finagle/netty3-1] ERROR com.metamx.common.scala.control$ - Non-retryable response for uri[/druid/indexer/v1/task] with content: {“error”:“Instantiation of [simple type, class org.apache.druid.segment.indexing.RealtimeTuningConfig] value failed: Failed to create directory within 10000 attempts (tried 1553419484189-0 to 1553419484189-9999) (through reference chain: org.apache.druid.indexing.common.task.RealtimeIndexTask[“spec”]->org.apache.druid.segment.realtime.FireDepartment[“tuningConfig”])”}
2019-03-24 09:24:44,262 [ClusteredBeam-ZkFuturePool-54281ecc-49cb-4108-bcc9-5eeb9250b8e6] ERROR c.m.tranquility.beam.ClusteredBeam - Failed to update cluster state: druid:overlord/foo
com.metamx.tranquility.druid.IndexServicePermanentException: Service[druid:overlord] call failed with status: 400 Bad Request
at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:158) ~[io.druid.tranquility-core-0.8.2.jar:0.8.2]
at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132) ~[io.druid.tranquility-core-0.8.2.jar:0.8.2]
at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Try$.apply(Try.scala:13) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Future$.apply(Future.scala:97) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise$Transformer.k(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise.runq(Promise.scala:350) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise.update(Promise.scala:694) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise.setValue(Promise.scala:670) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:194) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na]
at com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0]
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na]
at com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0]
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) ~[io.netty.netty-3.10.5.Final.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_191]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_191]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_191]
2019-03-24 09:24:44,275 [ClusteredBeam-ZkFuturePool-54281ecc-49cb-4108-bcc9-5eeb9250b8e6] WARN c.m.tranquility.beam.ClusteredBeam - Emitting alert: [anomaly] Failed to create merged beam: druid:overlord/foo
{ }
java.lang.IllegalStateException: Failed to save new beam for identifier[druid:overlord/foo] timestamp[2019-03-24T09:00:00.000Z]
at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:289) ~[io.druid.tranquility-core-0.8.2.jar:0.8.2]
at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:286) ~[io.druid.tranquility-core-0.8.2.jar:0.8.2]
at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:924) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:922) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise$Transformer.k(Promise.scala:112) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise.runq(Promise.scala:350) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.ExecutorServiceFuturePool$$anon$2.run(FuturePool.scala:107) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_191]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_191]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_191]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_191]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_191]
Caused by: com.metamx.tranquility.druid.IndexServicePermanentException: Service[druid:overlord] call failed with status: 400 Bad Request
at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:158) ~[io.druid.tranquility-core-0.8.2.jar:0.8.2]
at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132) ~[io.druid.tranquility-core-0.8.2.jar:0.8.2]
at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Try$.apply(Try.scala:13) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Future$.apply(Future.scala:97) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise$Transformer.k(Promise.scala:112) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise.runq(Promise.scala:350) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise.update(Promise.scala:694) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise.setValue(Promise.scala:670) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:194) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na]
at com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0]
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na]
at com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0]
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) ~[io.netty.netty-3.10.5.Final.jar:na]
… 3 common frames omitted
2019-03-24 09:24:44,302 [ClusteredBeam-ZkFuturePool-54281ecc-49cb-4108-bcc9-5eeb9250b8e6] INFO c.metamx.emitter.core.LoggingEmitter - Event [{“feed”:“alerts”,“timestamp”:“2019-03-24T09:24:44.291Z”,“service”:“tranquility”,“host”:“localhost”,“severity”:“anomaly”,“description”:“Failed to create merged beam: druid:overlord/foo”,“data”:{“exceptionType”:“java.lang.IllegalStateException”,“exceptionMessage”:“Failed to save new beam for identifier[druid:overlord/foo] timestamp[2019-03-24T09:00:00.000Z]”,“exceptionStackTrace”:“java.lang.IllegalStateException: Failed to save new beam for identifier[druid:overlord/foo] timestamp[2019-03-24T09:00:00.000Z]\n\tat com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:289)\n\tat com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:286)\n\tat com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:924)\n\tat com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:922)\n\tat com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112)\n\tat com.twitter.util.Promise$Transformer.k(Promise.scala:112)\n\tat com.twitter.util.Promise$Transformer.apply(Promise.scala:122)\n\tat com.twitter.util.Promise$Transformer.apply(Promise.scala:103)\n\tat com.twitter.util.Promise$$anon$1.run(Promise.scala:366)\n\tat com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178)\n\tat com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136)\n\tat com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207)\n\tat com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92)\n\tat com.twitter.util.Promise.runq(Promise.scala:350)\n\tat com.twitter.util.Promise.updateIfEmpty(Promise.scala:721)\n\tat com.twitter.util.ExecutorServiceFuturePool$$anon$2.run(FuturePool.scala:107)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: com.metamx.tranquility.druid.IndexServicePermanentException: Service[druid:overlord] call failed with status: 400 Bad Request\n\tat com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:158)\n\tat com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132)\n\tat com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950)\n\tat com.twitter.util.Try$.apply(Try.scala:13)\n\tat com.twitter.util.Future$.apply(Future.scala:97)\n\tat com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950)\n\tat com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949)\n\tat com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112)\n\tat com.twitter.util.Promise$Transformer.k(Promise.scala:112)\n\tat com.twitter.util.Promise$Transformer.apply(Promise.scala:122)\n\tat com.twitter.util.Promise$Transformer.apply(Promise.scala:103)\n\tat com.twitter.util.Promise$$anon$1.run(Promise.scala:366)\n\tat com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178)\n\tat com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136)\n\tat com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207)\n\tat com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92)\n\tat com.twitter.util.Promise.runq(Promise.scala:350)\n\tat com.twitter.util.Promise.updateIfEmpty(Promise.scala:721)\n\tat com.twitter.util.Promise.update(Promise.scala:694)\n\tat com.twitter.util.Promise.setValue(Promise.scala:670)\n\tat com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111)\n\tat com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108)\n\tat org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)\n\tat org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:194)\n\tat org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)\n\tat org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)\n\tat org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)\n\tat org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)\n\tat org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)\n\tat org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142)\n\tat com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78)\n\tat org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142)\n\tat com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35)\n\tat org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)\n\tat org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)\n\tat org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)\n\tat org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)\n\tat org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)\n\tat org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)\n\tat org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)\n\tat org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)\n\tat org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)\n\tat org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)\n\t… 3 more\n”}}]
2019-03-24 09:24:44,306 [qtp59295001-20] WARN c.m.t.server.http.TranquilityServlet - Server error serving request to http://ec2-34-207-97-159.compute-1.amazonaws.com:8200/v1/post/foo
java.lang.IllegalStateException: Failed to create merged beam: druid:overlord/foo
at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$sendAll$2$$anonfun$26.apply(ClusteredBeam.scala:383) ~[io.druid.tranquility-core-0.8.2.jar:0.8.2]
at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$sendAll$2$$anonfun$26.apply(ClusteredBeam.scala:379) ~[io.druid.tranquility-core-0.8.2.jar:0.8.2]
at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise$Transformer.k(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise.runq(Promise.scala:350) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.ExecutorServiceFuturePool$$anon$2.run(FuturePool.scala:107) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_191]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_191]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_191]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_191]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_191]
Caused by: java.lang.IllegalStateException: Failed to save new beam for identifier[druid:overlord/foo] timestamp[2019-03-24T09:00:00.000Z]
at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:289) ~[io.druid.tranquility-core-0.8.2.jar:0.8.2]
at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:286) ~[io.druid.tranquility-core-0.8.2.jar:0.8.2]
at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:924) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:922) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
… 17 common frames omitted
Caused by: com.metamx.tranquility.druid.IndexServicePermanentException: Service[druid:overlord] call failed with status: 400 Bad Request
at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:158) ~[io.druid.tranquility-core-0.8.2.jar:0.8.2]
at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132) ~[io.druid.tranquility-core-0.8.2.jar:0.8.2]
at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Try$.apply(Try.scala:13) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Future$.apply(Future.scala:97) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise$Transformer.k(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise.runq(Promise.scala:350) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise.update(Promise.scala:694) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.util.Promise.setValue(Promise.scala:670) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
at com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:194) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na]
at com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0]
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na]
at com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0]
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[io.netty.netty-3.10.5.Final.jar:na]
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) ~[io.netty.netty-3.10.5.Final.jar:na]
… 3 common frames omitted

Correcetion:
This is my overlord file:

druid.zk.service.host=127.0.0.1

druid.service=druid/overlord

druid.plaintextPort=8090

druid.host = 172.31:8090

druid.indexer.queue.startDelay=PT5S

druid.indexer.runner.type=remote

druid.indexer.storage.type=metadata

druid.extensions.loadList=[“mysql-metadata-storage”,“druid-kafka-indexing-service”]

druid.metadata.storage.type=mysql

druid.metadata.storage.connector.connectURI=jdbc:mysql://druid-development-east…us-east-1.rds.amazonaws.com/druid

#druid.metadata.storage.connector.user=druid

#druid.metadata.storage.connector.password=diurd

druid.metadata.storage.connector.user=root

druid.metadata.storage.connector.password=GxcXCZXC

druid.processing.tmpDir=/tmp/druid

PS I added -Djava.io.tmpdir=/tmp to overlord jvm config and worked and now when I ingest I get the below:

{“result”:{“received”:1,“sent”:0}}

I assume that means it worked?

Thanks