Tranquility failed to create directory

Hi guys,

I had Tranquility server running for a while and suddently it stopped. From the log it says

2018-10-17 22:31:10,890 [ClusteredBeam-ZkFuturePool-532e4d75-fd0a-4982-a6ff-25ac7f8b4ced] INFO c.m.t.finagle.FinagleRegistry - Created client for service: disco!druid:overlord
2018-10-17 22:31:12,150 [finagle/netty3-1] ERROR com.metamx.common.scala.control$ - Non-retryable response for uri[/druid/indexer/v1/task] with content: {“error”:“Instantiation of [simple type, class io.druid.segment.indexing.RealtimeTuningConfig] value failed: Failed to create directory within 10000 attempts (tried 1539815471149-0 to 1539815471149-9999)”}

``

the thing is that druid has permission to write to the temporary folder var/tmp (I run the middleManager with "-Djava.io.tmpdir=var/tmp). Also I checked the disk space and it was okay.

I restarted the server several times too, but same story.

Could anyone tell me what could possibly went wrong?

thx guys!

Below the log (full log attached):

2018-10-17 22:30:18,540 [main] INFO c.m.t.server.http.ServerMain$ - Reading configuration from file[/home/serg/mycompany/druid-conf/conf/tranquility/tranq-conf-DataProduction-cluster.json].

2018-10-17 22:30:18,660 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [20] for [http.threads] on [com.metamx.tranquility.server.PropertiesBasedServerConfig#httpThreads()]

2018-10-17 22:30:18,663 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [8200] for [http.port] on [com.metamx.tranquility.server.PropertiesBasedServerConfig#httpPort()]

2018-10-17 22:30:18,663 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [http.idleTimeout] on [com.metamx.tranquility.server.PropertiesBasedServerConfig#httpIdleTimeout()]

2018-10-17 22:30:18,703 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [druid/overlord] for [druid.selectors.indexing.serviceName] on [com.metamx.tranquility.config.PropertiesBasedConfig#druidIndexingServiceName()]

2018-10-17 22:30:18,703 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [task.partitions] on [com.metamx.tranquility.config.PropertiesBasedConfig#taskPartitions()]

2018-10-17 22:30:18,703 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [task.replicants] on [com.metamx.tranquility.config.PropertiesBasedConfig#taskReplicants()]

2018-10-17 22:30:18,704 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [task.warmingPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#taskWarmingPeriod()]

2018-10-17 22:30:18,704 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [serialization.format] on [com.metamx.tranquility.config.PropertiesBasedConfig#serializationFormat()]

2018-10-17 22:30:18,704 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [192.168.8.203] for [zookeeper.connect] on [com.metamx.tranquility.config.PropertiesBasedConfig#zookeeperConnect()]

2018-10-17 22:30:18,704 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [zookeeper.timeout] on [com.metamx.tranquility.config.PropertiesBasedConfig#zookeeperTimeout()]

2018-10-17 22:30:18,704 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [tranquility.maxBatchSize] on [com.metamx.tranquility.config.PropertiesBasedConfig#tranquilityMaxBatchSize()]

2018-10-17 22:30:18,705 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [tranquility.maxPendingBatches] on [com.metamx.tranquility.config.PropertiesBasedConfig#tranquilityMaxPendingBatches()]

2018-10-17 22:30:18,705 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [tranquility.lingerMillis] on [com.metamx.tranquility.config.PropertiesBasedConfig#tranquilityLingerMillis()]

2018-10-17 22:30:18,705 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [/druid/discovery] for [druid.discovery.curator.path] on [com.metamx.tranquility.config.PropertiesBasedConfig#discoPath()]

2018-10-17 22:30:18,705 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseGracePeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseGracePeriod()]

2018-10-17 22:30:18,708 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseQuietPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseQuietPeriod()]

2018-10-17 22:30:18,709 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseRetryPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseRetryPeriod()]

2018-10-17 22:30:18,709 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseChunkSize] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseChunkSize()]

2018-10-17 22:30:18,709 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.randomizeTaskId] on [com.metamx.tranquility.config.PropertiesBasedConfig#randomizeTaskId()]

2018-10-17 22:30:18,709 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.indexRetryPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#indexRetryPeriod()]

2018-10-17 22:30:18,710 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseBufferSize] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseBufferSize()]

2018-10-17 22:30:18,710 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.overlordLocator] on [com.metamx.tranquility.config.PropertiesBasedConfig#overlordLocator()]

2018-10-17 22:30:18,710 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.taskLocator] on [com.metamx.tranquility.config.PropertiesBasedConfig#taskLocator()]

2018-10-17 22:30:18,710 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.overlordPollPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#overlordPollPeriod()]

2018-10-17 22:30:18,811 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [20] for [http.threads] on [com.metamx.tranquility.server.PropertiesBasedServerConfig#httpThreads()]

2018-10-17 22:30:18,811 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [8200] for [http.port] on [com.metamx.tranquility.server.PropertiesBasedServerConfig#httpPort()]

2018-10-17 22:30:18,811 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [http.idleTimeout] on [com.metamx.tranquility.server.PropertiesBasedServerConfig#httpIdleTimeout()]

2018-10-17 22:30:18,811 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [druid/overlord] for [druid.selectors.indexing.serviceName] on [com.metamx.tranquility.config.PropertiesBasedConfig#druidIndexingServiceName()]

2018-10-17 22:30:18,811 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [1] for [task.partitions] on [com.metamx.tranquility.config.PropertiesBasedConfig#taskPartitions()]

2018-10-17 22:30:18,811 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [1] for [task.replicants] on [com.metamx.tranquility.config.PropertiesBasedConfig#taskReplicants()]

2018-10-17 22:30:18,811 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [task.warmingPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#taskWarmingPeriod()]

2018-10-17 22:30:18,811 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [serialization.format] on [com.metamx.tranquility.config.PropertiesBasedConfig#serializationFormat()]

2018-10-17 22:30:18,812 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [192.168.8.203] for [zookeeper.connect] on [com.metamx.tranquility.config.PropertiesBasedConfig#zookeeperConnect()]

2018-10-17 22:30:18,812 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [zookeeper.timeout] on [com.metamx.tranquility.config.PropertiesBasedConfig#zookeeperTimeout()]

2018-10-17 22:30:18,812 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [tranquility.maxBatchSize] on [com.metamx.tranquility.config.PropertiesBasedConfig#tranquilityMaxBatchSize()]

2018-10-17 22:30:18,812 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [tranquility.maxPendingBatches] on [com.metamx.tranquility.config.PropertiesBasedConfig#tranquilityMaxPendingBatches()]

2018-10-17 22:30:18,812 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [tranquility.lingerMillis] on [com.metamx.tranquility.config.PropertiesBasedConfig#tranquilityLingerMillis()]

2018-10-17 22:30:18,812 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [/druid/discovery] for [druid.discovery.curator.path] on [com.metamx.tranquility.config.PropertiesBasedConfig#discoPath()]

2018-10-17 22:30:18,812 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseGracePeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseGracePeriod()]

2018-10-17 22:30:18,812 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseQuietPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseQuietPeriod()]

2018-10-17 22:30:18,813 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseRetryPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseRetryPeriod()]

2018-10-17 22:30:18,813 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseChunkSize] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseChunkSize()]

2018-10-17 22:30:18,813 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.randomizeTaskId] on [com.metamx.tranquility.config.PropertiesBasedConfig#randomizeTaskId()]

2018-10-17 22:30:18,813 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.indexRetryPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#indexRetryPeriod()]

2018-10-17 22:30:18,813 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseBufferSize] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseBufferSize()]

2018-10-17 22:30:18,813 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.overlordLocator] on [com.metamx.tranquility.config.PropertiesBasedConfig#overlordLocator()]

2018-10-17 22:30:18,813 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.taskLocator] on [com.metamx.tranquility.config.PropertiesBasedConfig#taskLocator()]

2018-10-17 22:30:18,820 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.overlordPollPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#overlordPollPeriod()]

2018-10-17 22:30:18,827 [main] INFO c.m.t.server.http.ServerMain$ - Read configuration for dataSources[DataProduction].

2018-10-17 22:30:18,900 [main] WARN o.a.c.retry.ExponentialBackoffRetry - maxRetries too large (30). Pinning to 29

2018-10-17 22:30:19,085 [main] INFO c.m.t.finagle.FinagleRegistry - Adding resolver for scheme[disco].

2018-10-17 22:30:20,741 [main] INFO o.h.validator.internal.util.Version - HV000001: Hibernate Validator 5.1.3.Final

2018-10-17 22:30:21,075 [main] INFO io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, directory=‘extensions’, hadoopDependenciesDir=‘hadoop-dependencies’, hadoopContainerDruidClasspath=‘null’, loadList=null}]

2018-10-17 22:30:21,568 [main] INFO c.metamx.emitter.core.LoggingEmitter - Start: started [true]

2018-10-17 22:30:22,036 [main] INFO org.eclipse.jetty.util.log - Logging initialized @4440ms

2018-10-17 22:30:22,095 [main] INFO o.a.c.f.imps.CuratorFrameworkImpl - Starting

2018-10-17 22:30:22,105 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.8–1, built on 02/06/2016 03:18 GMT

2018-10-17 22:30:22,105 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=druid-historical-vm

2018-10-17 22:30:22,105 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_181

2018-10-17 22:30:22,105 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation

2018-10-17 22:30:22,105 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-8-oracle/jre

2018-10-17 22:30:22,105 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/home/serg/tranquility/lib/io.druid.tranquility-distribution-0.8.2.jar:/home/serg/tranquility/lib/io.druid.tranquility-kafka-0.8.2.jar:/home/serg/tranquility/lib/io.druid.tranquility-core-0.8.2.jar:/home/serg/tranquility/lib/io.druid.tranquility-server-0.8.2.jar:/home/serg/tranquility/lib/org.scala-lang.scala-library-2.11.7.jar:/home/serg/tranquility/lib/com.metamx.scala-util_2.11-1.11.6.jar:/home/serg/tranquility/lib/com.metamx.loglady_2.11-1.1.0-mmx.jar:/home/serg/tranquility/lib/org.skife.config.config-magic-0.9.jar:/home/serg/tranquility/lib/com.google.guava.guava-16.0.1.jar:/home/serg/tranquility/lib/com.fasterxml.jackson.core.jackson-annotations-2.4.6.jar:/home/serg/tranquility/lib/com.fasterxml.jackson.core.jackson-core-2.4.6.jar:/home/serg/tranquility/lib/com.fasterxml.jackson.core.jackson-databind-2.4.6.jar:/home/serg/tranquility/lib/net.sf.opencsv.opencsv-2.3.jar:/home/serg/tranquility/lib/io.netty.netty-3.10.5.Final.jar:/home/serg/tranquility/lib/javax.validation.validation-api-1.1.0.Final.jar:/home/serg/tranquility/lib/commons-lang.commons-lang-2.6.jar:/home/serg/tranquility/lib/org.scalaj.scalaj-time_2.11-0.5.jar:/home/serg/tranquility/lib/org.yaml.snakeyaml-1.11.jar:/home/serg/tranquility/lib/com.fasterxml.jackson.dataformat.jackson-dataformat-smile-2.4.6.jar:/home/serg/tranquility/lib/com.fasterxml.jackson.datatype.jackson-datatype-joda-2.4.6.jar:/home/serg/tranquility/lib/com.fasterxml.jackson.module.jackson-module-scala_2.11-2.4.5.jar:/home/serg/tranquility/lib/org.scala-lang.scala-reflect-2.11.2.jar:/home/serg/tranquility/lib/com.thoughtworks.paranamer.paranamer-2.6.jar:/home/serg/tranquility/lib/com.google.code.findbugs.jsr305-2.0.1.jar:/home/serg/tranquility/lib/mysql.mysql-connector-java-5.1.18.jar:/home/serg/tranquility/lib/com.h2database.h2-1.3.158.jar:/home/serg/tranquility/lib/c3p0.c3p0-0.9.1.2.jar:/home/serg/tranquility/lib/jline.jline-0.9.94.jar:/home/serg/tranquility/lib/junit.junit-3.8.1.jar:/home/serg/tranquility/lib/org.codehaus.jackson.jackson-mapper-asl-1.9.13.jar:/home/serg/tranquility/lib/org.codehaus.jackson.jackson-core-asl-1.9.13.jar:/home/serg/tranquility/lib/com.twitter.util-core_2.11-6.30.0.jar:/home/serg/tranquility/lib/com.twitter.util-function_2.11-6.30.0.jar:/home/serg/tranquility/lib/com.twitter.jsr166e-1.0.0.jar:/home/serg/tranquility/lib/org.scala-lang.modules.scala-parser-combinators_2.11-1.0.4.jar:/home/serg/tranquility/lib/com.twitter.finagle-core_2.11-6.31.0.jar:/home/serg/tranquility/lib/com.twitter.util-app_2.11-6.30.0.jar:/home/serg/tranquility/lib/com.twitter.util-registry_2.11-6.30.0.jar:/home/serg/tranquility/lib/com.twitter.util-cache_2.11-6.30.0.jar:/home/serg/tranquility/lib/com.twitter.util-codec_2.11-6.30.0.jar:/home/serg/tranquility/lib/commons-codec.commons-codec-1.9.jar:/home/serg/tranquility/lib/com.twitter.util-collection_2.11-6.30.0.jar:/home/serg/tranquility/lib/javax.inject.javax.inject-1.jar:/home/serg/tranquility/lib/commons-collections.commons-collections-3.2.1.jar:/home/serg/tranquility/lib/com.twitter.util-hashing_2.11-6.30.0.jar:/home/serg/tranquility/lib/com.twitter.util-jvm_2.11-6.30.0.jar:/home/serg/tranquility/lib/com.twitter.util-lint_2.11-6.30.0.jar:/home/serg/tranquility/lib/com.twitter.util-logging_2.11-6.30.0.jar:/home/serg/tranquility/lib/com.twitter.util-stats_2.11-6.30.0.jar:/home/serg/tranquility/lib/com.twitter.finagle-http_2.11-6.31.0.jar:/home/serg/tranquility/lib/org.slf4j.slf4j-api-1.7.12.jar:/home/serg/tranquility/lib/org.slf4j.jul-to-slf4j-1.7.12.jar:/home/serg/tranquility/lib/commons-logging.commons-logging-1.1.3.jar:/home/serg/tranquility/lib/io.druid.druid-server-0.9.1.jar:/home/serg/tranquility/lib/io.druid.druid-processing-0.9.1.jar:/home/serg/tranquility/lib/io.druid.druid-common-0.9.1.jar:/home/serg/tranquility/lib/io.druid.druid-api-0.9.1.jar:/home/serg/tranquility/lib/aopalliance.aopalliance-1.0.jar:/home/serg/tranquility/lib/io.airlift.airline-0.7.jar:/home/serg/tranquility/lib/com.google.code.findbugs.annotations-2.0.3.jar:/home/serg/tranquility/lib/org.hibernate.hibernate-validator-5.1.3.Final.jar:/home/serg/tranquility/lib/org.jboss.logging.jboss-logging-3.1.3.GA.jar:/home/serg/tranquility/lib/com.fasterxml.classmate-1.0.0.jar:/home/serg/tranquility/lib/commons-io.commons-io-2.4.jar:/home/serg/tranquility/lib/org.apache.commons.commons-dbcp2-2.0.1.jar:/home/serg/tranquility/lib/org.apache.commons.commons-pool2-2.2.jar:/home/serg/tranquility/lib/commons-pool.commons-pool-1.6.jar:/home/serg/tranquility/lib/javax.el.javax.el-api-3.0.0.jar:/home/serg/tranquility/lib/com.fasterxml.jackson.datatype.jackson-datatype-guava-2.4.6.jar:/home/serg/tranquility/lib/org.jdbi.jdbi-2.63.1.jar:/home/serg/tranquility/lib/joda-time.joda-time-2.8.2.jar:/home/serg/tranquility/lib/org.apache.logging.log4j.log4j-jul-2.5.jar:/home/serg/tranquility/lib/org.slf4j.jcl-over-slf4j-1.7.12.jar:/home/serg/tranquility/lib/net.java.dev.jets3t.jets3t-0.9.4.jar:/home/serg/tranquility/lib/javax.activation.activation-1.1.1.jar:/home/serg/tranquility/lib/org.bouncycastle.bcprov-jdk15on-1.52.jar:/home/serg/tranquility/lib/com.jamesmurty.utils.java-xmlbuilder-1.1.jar:/home/serg/tranquility/lib/net.iharder.base64-2.3.8.jar:/home/serg/tranquility/lib/org.antlr.antlr4-runtime-4.5.1.jar:/home/serg/tranquility/lib/com.metamx.bytebuffer-collections-0.2.4.jar:/home/serg/tranquility/lib/com.metamx.extendedset-1.3.9.jar:/home/serg/tranquility/lib/org.roaringbitmap.RoaringBitmap-0.5.16.jar:/home/serg/tranquility/lib/com.metamx.emitter-0.3.6.jar:/home/serg/tranquility/lib/com.metamx.java-util-0.27.9.jar:/home/serg/tranquility/lib/org.mozilla.rhino-1.7R5.jar:/home/serg/tranquility/lib/com.jayway.jsonpath.json-path-2.1.0.jar:/home/serg/tranquility/lib/com.metamx.http-client-1.0.4.jar:/home/serg/tranquility/lib/com.metamx.server-metrics-0.2.8.jar:/home/serg/tranquility/lib/com.ning.compress-lzf-1.0.3.jar:/home/serg/tranquility/lib/com.google.protobuf.protobuf-java-2.5.0.jar:/home/serg/tranquility/lib/com.ibm.icu.icu4j-4.8.1.jar:/home/serg/tranquility/lib/net.jpountz.lz4.lz4-1.3.0.jar:/home/serg/tranquility/lib/org.mapdb.mapdb-1.0.8.jar:/home/serg/tranquility/lib/io.druid.druid-aws-common-0.9.1.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-support-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-core-1.10.21.jar:/home/serg/tranquility/lib/org.apache.httpcomponents.httpclient-4.5.1.jar:/home/serg/tranquility/lib/org.apache.httpcomponents.httpcore-4.4.3.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-simpledb-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-simpleworkflow-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-storagegateway-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-route53-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-s3-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-kms-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-importexport-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-sts-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-sqs-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-rds-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-redshift-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-elasticbeanstalk-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-glacier-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-sns-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-iam-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-datapipeline-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-elasticloadbalancing-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-emr-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-elasticache-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-elastictranscoder-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-ec2-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-dynamodb-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-cloudtrail-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-cloudwatch-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-logs-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-cognitoidentity-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-cognitosync-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-directconnect-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-cloudformation-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-cloudfront-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-kinesis-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-opsworks-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-ses-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-autoscaling-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-cloudsearch-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-cloudwatchmetrics-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-swf-libraries-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-codedeploy-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-codepipeline-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-config-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-lambda-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-ecs-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-cloudhsm-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-ssm-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-workspaces-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-machinelearning-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-directory-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-efs-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-codecommit-1.10.21.jar:/home/serg/tranquility/lib/com.amazonaws.aws-java-sdk-devicefarm-1.10.21.jar:/home/serg/tranquility/lib/io.druid.druid-console-0.0.3.jar:/home/serg/tranquility/lib/commons-cli.commons-cli-1.2.jar:/home/serg/tranquility/lib/org.glassfish.javax.el-3.0.0.jar:/home/serg/tranquility/lib/org.apache.zookeeper.zookeeper-3.4.8.jar:/home/serg/tranquility/lib/org.apache.curator.curator-framework-2.10.0.jar:/home/serg/tranquility/lib/org.apache.curator.curator-client-2.10.0.jar:/home/serg/tranquility/lib/org.apache.curator.curator-x-discovery-2.10.0.jar:/home/serg/tranquility/lib/org.apache.curator.curator-recipes-2.10.0.jar:/home/serg/tranquility/lib/com.fasterxml.jackson.jaxrs.jackson-jaxrs-json-provider-2.4.6.jar:/home/serg/tranquility/lib/com.fasterxml.jackson.jaxrs.jackson-jaxrs-base-2.4.6.jar:/home/serg/tranquility/lib/com.fasterxml.jackson.module.jackson-module-jaxb-annotations-2.4.6.jar:/home/serg/tranquility/lib/com.fasterxml.jackson.jaxrs.jackson-jaxrs-smile-provider-2.4.6.jar:/home/serg/tranquility/lib/com.sun.jersey.jersey-server-1.19.jar:/home/serg/tranquility/lib/com.sun.jersey.jersey-core-1.19.jar:/home/serg/tranquility/lib/javax.ws.rs.jsr311-api-1.1.1.jar:/home/serg/tranquility/lib/com.sun.jersey.contribs.jersey-guice-1.19.jar:/home/serg/tranquility/lib/org.eclipse.jetty.jetty-server-9.2.5.v20141112.jar:/home/serg/tranquility/lib/javax.servlet.javax.servlet-api-3.1.0.jar:/home/serg/tranquility/lib/org.eclipse.jetty.jetty-http-9.2.5.v20141112.jar:/home/serg/tranquility/lib/org.eclipse.jetty.jetty-util-9.2.5.v20141112.jar:/home/serg/tranquility/lib/org.eclipse.jetty.jetty-io-9.2.5.v20141112.jar:/home/serg/tranquility/lib/org.eclipse.jetty.jetty-proxy-9.2.5.v20141112.jar:/home/serg/tranquility/lib/org.eclipse.jetty.jetty-client-9.2.5.v20141112.jar:/home/serg/tranquility/lib/io.tesla.aether.tesla-aether-0.0.5.jar:/home/serg/tranquility/lib/org.eclipse.aether.aether-api-0.9.0.M2.jar:/home/serg/tranquility/lib/org.eclipse.aether.aether-spi-0.9.0.M2.jar:/home/serg/tranquility/lib/org.eclipse.aether.aether-util-0.9.0.M2.jar:/home/serg/tranquility/lib/org.eclipse.aether.aether-impl-0.9.0.M2.jar:/home/serg/tranquility/lib/org.eclipse.aether.aether-connector-file-0.9.0.M2.jar:/home/serg/tranquility/lib/io.tesla.aether.aether-connector-okhttp-0.0.9.jar:/home/serg/tranquility/lib/com.squareup.okhttp.okhttp-1.0.2.jar:/home/serg/tranquility/lib/org.apache.maven.wagon.wagon-provider-api-2.4.jar:/home/serg/tranquility/lib/org.codehaus.plexus.plexus-utils-3.0.15.jar:/home/serg/tranquility/lib/org.apache.maven.maven-aether-provider-3.1.1.jar:/home/serg/tranquility/lib/org.apache.maven.maven-model-3.1.1.jar:/home/serg/tranquility/lib/org.apache.maven.maven-model-builder-3.1.1.jar:/home/serg/tranquility/lib/org.codehaus.plexus.plexus-interpolation-1.19.jar:/home/serg/tranquility/lib/org.apache.maven.maven-repository-metadata-3.1.1.jar:/home/serg/tranquility/lib/org.apache.maven.maven-settings-builder-3.1.1.jar:/home/serg/tranquility/lib/org.apache.maven.maven-settings-3.1.1.jar:/home/serg/tranquility/lib/net.spy.spymemcached-2.11.7.jar:/home/serg/tranquility/lib/org.eclipse.jetty.jetty-servlet-9.2.5.v20141112.jar:/home/serg/tranquility/lib/org.eclipse.jetty.jetty-security-9.2.5.v20141112.jar:/home/serg/tranquility/lib/org.eclipse.jetty.jetty-servlets-9.2.5.v20141112.jar:/home/serg/tranquility/lib/org.eclipse.jetty.jetty-continuation-9.2.5.v20141112.jar:/home/serg/tranquility/lib/com.ircclouds.irc.irc-api-1.0-0014.jar:/home/serg/tranquility/lib/com.maxmind.geoip2.geoip2-0.4.0.jar:/home/serg/tranquility/lib/com.maxmind.maxminddb.maxminddb-0.2.0.jar:/home/serg/tranquility/lib/com.google.http-client.google-http-client-jackson2-1.15.0-rc.jar:/home/serg/tranquility/lib/org.apache.derby.derby-10.11.1.1.jar:/home/serg/tranquility/lib/org.apache.derby.derbynet-10.11.1.1.jar:/home/serg/tranquility/lib/org.apache.derby.derbyclient-10.11.1.1.jar:/home/serg/tranquility/lib/org.apache.commons.commons-math3-3.6.1.jar:/home/serg/tranquility/lib/com.google.inject.guice-4.0.jar:/home/serg/tranquility/lib/com.google.inject.extensions.guice-servlet-4.0.jar:/home/serg/tranquility/lib/com.google.inject.extensions.guice-multibindings-4.0.jar:/home/serg/tranquility/lib/org.apache.kafka.kafka_2.11-0.8.2.2.jar:/home/serg/tranquility/lib/com.yammer.metrics.metrics-core-2.2.0.jar:/home/serg/tranquility/lib/net.sf.jopt-simple.jopt-simple-3.2.jar:/home/serg/tranquility/lib/com.101tec.zkclient-0.3.jar:/home/serg/tranquility/lib/org.apache.kafka.kafka-clients-0.8.2.2.jar:/home/serg/tranquility/lib/org.xerial.snappy.snappy-java-1.1.1.7.jar:/home/serg/tranquility/lib/ch.qos.logback.logback-core-1.1.2.jar:/home/serg/tranquility/lib/ch.qos.logback.logback-classic-1.1.2.jar:/home/serg/tranquility/lib/org.apache.logging.log4j.log4j-to-slf4j-2.4.jar:/home/serg/tranquility/lib/org.apache.logging.log4j.log4j-api-2.4.jar:/home/serg/tranquility/lib/org.slf4j.log4j-over-slf4j-1.7.12.jar:/home/serg/tranquility/lib/org.scalatra.scalatra_2.11-2.3.1.jar:/home/serg/tranquility/lib/org.scalatra.scalatra-common_2.11-2.3.1.jar:/home/serg/tranquility/lib/org.clapper.grizzled-slf4j_2.11-1.0.2.jar:/home/serg/tranquility/lib/org.scalatra.rl.rl_2.11-0.4.10.jar:/home/serg/tranquility/lib/com.googlecode.juniversalchardet.juniversalchardet-1.0.3.jar:/home/serg/tranquility/lib/eu.medsea.mimeutil.mime-util-2.1.3.jar:/home/serg/tranquility/lib/org.joda.joda-convert-1.7.jar:/home/serg/tranquility/lib/org.scala-lang.modules.scala-xml_2.11-1.0.3.jar

2018-10-17 22:30:22,106 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

2018-10-17 22:30:22,106 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp

2018-10-17 22:30:22,106 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=

2018-10-17 22:30:22,106 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux

2018-10-17 22:30:22,106 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64

2018-10-17 22:30:22,106 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=4.15.0-34-generic

2018-10-17 22:30:22,106 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=serg

2018-10-17 22:30:22,106 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=/home/serg

2018-10-17 22:30:22,106 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/home/serg/tranquility

2018-10-17 22:30:22,107 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=192.168.8.203 sessionTimeout=20000 watcher=org.apache.curator.ConnectionState@10f477e2

2018-10-17 22:30:22,133 [main-SendThread(192.168.8.203:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server 192.168.8.203/192.168.8.203:2181. Will not attempt to authenticate using SASL (unknown error)

2018-10-17 22:30:22,133 [main] INFO c.m.c.l.Lifecycle$AnnotationBasedHandler - Invoking start method[public void com.metamx.common.scala.net.curator.Disco.start()] on object[com.metamx.common.scala.net.curator.Disco@6fa02932].

2018-10-17 22:30:22,136 [main] INFO c.m.c.l.Lifecycle$AnnotationBasedHandler - Invoking start method[public void com.metamx.tranquility.tranquilizer.Tranquilizer.start()] on object[Tranquilizer(com.metamx.tranquility.beam.TransformingBeam@67fb5025)].

2018-10-17 22:30:22,139 [main] INFO org.eclipse.jetty.server.Server - jetty-9.2.5.v20141112

2018-10-17 22:30:22,143 [main-SendThread(192.168.8.203:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to 192.168.8.203/192.168.8.203:2181, initiating session

2018-10-17 22:30:22,154 [main-SendThread(192.168.8.203:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server 192.168.8.203/192.168.8.203:2181, sessionid = 0x16631f7472102f7, negotiated timeout = 20000

2018-10-17 22:30:22,161 [main-EventThread] INFO o.a.c.f.state.ConnectionStateManager - State change: CONNECTED

2018-10-17 22:30:22,207 [main] INFO o.e.jetty.server.ServerConnector - Started ServerConnector@5234b61a{HTTP/1.1}{0.0.0.0:8200}

2018-10-17 22:30:22,208 [main] INFO org.eclipse.jetty.server.Server - Started @4613ms

2018-10-17 22:31:10,301 [ClusteredBeam-ZkFuturePool-532e4d75-fd0a-4982-a6ff-25ac7f8b4ced] INFO c.m.tranquility.beam.ClusteredBeam - Creating new merged beam for identifier[druid:overlord/DataProduction] timestamp[2018-10-17T00:00:00.000Z] (target = 1, actual = 0)

2018-10-17 22:31:10,455 [ClusteredBeam-ZkFuturePool-532e4d75-fd0a-4982-a6ff-25ac7f8b4ced] INFO com.metamx.common.scala.control$ - Creating druid indexing task (service = druid:overlord): {

“type” : “index_realtime”,

“id” : “index_realtime_DataProduction_2018-10-17T00:00:00.000Z_0_0”,

“resource” : {

“availabilityGroup” : “DataProduction-2018-10-17T00:00:00.000Z-0000”,

“requiredCapacity” : 1

},

“spec” : {

“dataSchema” : {

“dataSource” : “DataProduction”,

“parser” : {

“type” : “map”,

“parseSpec” : {

“format” : “json”,

“timestampSpec” : {

“column” : “ts”,

“format” : “millis”,

“missingValue” : null

},

“dimensionsSpec” : {

“dimensions” : [ “dim1”, “dim2”, “dim3”, “dim4”, “dim5”, “dim6”, “dim7”, “dim8”, “dim9”, “dim10”, “dim11”, “dim12”, “dim13” ],

“spatialDimensions” :

}

}

},

“metricsSpec” : [ {

“type” : “count”,

“name” : “count”

}, {

“type” : “doubleMin”,

“name” : “min”,

“fieldName” : “metric_value”

}, {

“type” : “doubleMax”,

“name” : “max”,

“fieldName” : “metric_value”

}, {

“type” : “doubleSum”,

“name” : “sum”,

“fieldName” : “metric_value”

} ],

“granularitySpec” : {

“type” : “uniform”,

“segmentGranularity” : “DAY”,

“queryGranularity” : {

“type” : “duration”,

“duration” : 60000,

“origin” : “1970-01-01T00:00:00.000Z”

}

}

},

“ioConfig” : {

“type” : “realtime”,

“plumber” : null,

“firehose” : {

“type” : “clipped”,

“interval” : “2018-10-17T00:00:00.000Z/2018-10-18T00:00:00.000Z”,

“delegate” : {

“type” : “timed”,

“shutoffTime” : “2018-10-18T00:15:00.000Z”,

“delegate” : {

“type” : “receiver”,

“serviceName” : “firehose:druid:overlord:DataProduction-017-0000-0000”,

“bufferSize” : 100000

}

}

}

},

“tuningConfig” : {

“shardSpec” : {

“type” : “linear”,

“partitionNum” : 0

},

“rejectionPolicy” : {

“type” : “none”

},

“buildV9Directly” : false,

“maxPendingPersists” : 0,

“intermediatePersistPeriod” : “PT10M”,

“windowPeriod” : “PT10M”,

“type” : “realtime”,

“maxRowsInMemory” : “100000”

}

}

}

Oct 17, 2018 10:31:10 PM com.twitter.finagle.Init$$anonfun$1 apply$mcV$sp

INFO: Finagle version 6.31.0 (rev=50d3bb0eea5ad3ed332111d707184c80fed6a506) built at 20151203-164135

2018-10-17 22:31:10,798 [ClusteredBeam-ZkFuturePool-532e4d75-fd0a-4982-a6ff-25ac7f8b4ced] INFO c.m.c.s.net.finagle.DiscoResolver - Updating instances for service[druid:overlord] to Set(ServiceInstance{name=‘druid:overlord’, id=‘5452b7db-d60e-4d70-bc02-4a5ba0a65a3b’, address=‘druid-coordinator-vm.mycompanylocal’, port=8090, sslPort=-1, payload=null, registrationTimeUTC=1538436911532, serviceType=DYNAMIC, uriSpec=null})

2018-10-17 22:31:10,890 [ClusteredBeam-ZkFuturePool-532e4d75-fd0a-4982-a6ff-25ac7f8b4ced] INFO c.m.t.finagle.FinagleRegistry - Created client for service: disco!druid:overlord

2018-10-17 22:31:12,150 [finagle/netty3-1] ERROR com.metamx.common.scala.control$ - Non-retryable response for uri[/druid/indexer/v1/task] with content: {“error”:“Instantiation of [simple type, class io.druid.segment.indexing.RealtimeTuningConfig] value failed: Failed to create directory within 10000 attempts (tried 1539815471149-0 to 1539815471149-9999)”}

2018-10-17 22:31:12,164 [ClusteredBeam-ZkFuturePool-532e4d75-fd0a-4982-a6ff-25ac7f8b4ced] ERROR c.m.tranquility.beam.ClusteredBeam - Failed to update cluster state: druid:overlord/DataProduction

com.metamx.tranquility.druid.IndexServicePermanentException: Service[druid:overlord] call failed with status: 400 Bad Request

at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:158) ~[io.druid.tranquility-core-0.8.2.jar:0.8.2]

at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132) ~[io.druid.tranquility-core-0.8.2.jar:0.8.2]

at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.util.Try$.apply(Try.scala:13) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.util.Future$.apply(Future.scala:97) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.util.Promise$Transformer.k(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.util.Promise.runq(Promise.scala:350) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.util.Promise.update(Promise.scala:694) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.util.Promise.setValue(Promise.scala:670) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0]

at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:194) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na]

at com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0]

at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na]

at com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0]

at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) ~[io.netty.netty-3.10.5.Final.jar:na]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_181]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_181]

at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]

2018-10-17 22:31:12,176 [ClusteredBeam-ZkFuturePool-532e4d75-fd0a-4982-a6ff-25ac7f8b4ced] WARN c.m.tranquility.beam.ClusteredBeam - Emitting alert: [anomaly] Failed to create merged beam: druid:overlord/DataProduction

{ }

java.lang.IllegalStateException: Failed to save new beam for identifier[druid:overlord/DataProduction] timestamp[2018-10-17T00:00:00.000Z]

at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:289) ~[io.druid.tranquility-core-0.8.2.jar:0.8.2]

at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:286) ~[io.druid.tranquility-core-0.8.2.jar:0.8.2]

at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:924) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:922) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.util.Promise$Transformer.k(Promise.scala:112) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.util.Promise.runq(Promise.scala:350) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.util.ExecutorServiceFuturePool$$anon$2.run(FuturePool.scala:107) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_181]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_181]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_181]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_181]

at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]

Caused by: com.metamx.tranquility.druid.IndexServicePermanentException: Service[druid:overlord] call failed with status: 400 Bad Request

at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:158) ~[io.druid.tranquility-core-0.8.2.jar:0.8.2]

at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132) ~[io.druid.tranquility-core-0.8.2.jar:0.8.2]

at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.util.Try$.apply(Try.scala:13) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.util.Future$.apply(Future.scala:97) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.util.Promise$Transformer.k(Promise.scala:112) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.util.Promise.runq(Promise.scala:350) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.util.Promise.update(Promise.scala:694) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.util.Promise.setValue(Promise.scala:670) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0]

at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:194) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na]

at com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0]

at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na]

at com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0]

at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[io.netty.netty-3.10.5.Final.jar:na]

at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) ~[io.netty.netty-3.10.5.Final.jar:na]

… 3 common frames omitted

``

tranqulity.log (627 KB)

There are additional configurations that explicitly set the scratch directory for indexing tasks: druid.io/docs/latest/configuration/index.html#additional-peon-configuration

Can you check if you have ‘druid.indexer.task.baseDir’ or ‘druid.indexer.task.baseTaskDir’ set in your common.runtime.properties or MM’s runtime.properties file?

If you don’t, I would also check specifically that var/tmp/persistent/tasks, if it exists, is writable by the MM process’ user/group.

Thank you David!

In my MM conf file I have “druid.indexer.task.baseTaskDir=var/druid/task” and “Djava.io.tmpdir=var/tmp” in the jvm.config. But " var/tmp/persistent/tasks" doesn’t exists.

Anyhow I find the issue: the problem was that the overlord disk was full, and in the log a bit further down than the intial error, there was this:

Caused by: com.metamx.tranquility.druid.IndexServicePermanentException: Service[druid:overlord] call failed with status: 400 Bad Request
        at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:158) ~[io.druid.tranquility-core-0.8.2.jar:0.8.2]
        at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132) ~[io.druid.tranquility-core-0.8.2.jar:0.8.2]
        at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]
        at com.twitter.util.Try$.apply(Try.scala:13) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0]

``

I clean up the overlord disk and that fix it.

So why tranquility was still working even though that persist folder does not exist?

thanks for the support.