Messages from tranquility are not getting pushed to druid

We are using Druid 0.7 with indexer deployed in local mode ( No middle manager configured).
Using latest version of Tranquility_2.10 and is running in the storm as a storm bolt.

From the overlord console, it looks like task is getting created sucessfully. Status shows that it is running. There are no exceptions in the task logs.

On the tranquility side, there are no errors. We have logs to confirm that the messages were processed by tranquility bolt.

However, data source is not getting created and the data is not flowing through.

We have spent a last couple days trying to get past this issue but no luck. Any help is highly appreciated !!! Thanks in advance.

Please let me know if you need any other information and I will be happy to provide.

Please find the task logs and tranquility logs below:

2015-03-12T06:37:02,136 INFO [main] io.druid.guice.PropertiesModule - Loading properties from common.runtime.properties
2015-03-12T06:37:02,141 INFO [main] io.druid.guice.PropertiesModule - Loading properties from runtime.properties
2015-03-12T06:37:03,015 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, coordinates=[io.druid.extensions:druid-examples, io.druid.extensions:druid-kafka-eight, io.druid.extensions:mysql-metadata-storage], defaultVersion='0.7.0', localRepository='/root/.m2/repository', remoteRepositories=[[http://repo1.maven.org/maven2/](http://repo1.maven.org/maven2/), [https://metamx.artifactoryonline.com/metamx/pub-libs-releases-local]](https://metamx.artifactoryonline.com/metamx/pub-libs-releases-local%5D)}]
2015-03-12T06:37:03,242 INFO [main] io.druid.initialization.Initialization - Loading extension[io.druid.extensions:druid-examples] for class[io.druid.cli.CliCommandCreator]
2015-03-12T06:37:04,427 INFO [main] io.druid.initialization.Initialization - Added URL[file:/root/.m2/repository/io/druid/extensions/druid-examples/0.7.0/druid-examples-0.7.0.jar]
2015-03-12T06:37:04,428 INFO [main] io.druid.initialization.Initialization - Added URL[file:/root/.m2/repository/org/twitter4j/twitter4j-core/3.0.3/twitter4j-core-3.0.3.jar]
2015-03-12T06:37:04,428 INFO [main] io.druid.initialization.Initialization - Added URL[file:/root/.m2/repository/org/twitter4j/twitter4j-async/3.0.3/twitter4j-async-3.0.3.jar]
2015-03-12T06:37:04,428 INFO [main] io.druid.initialization.Initialization - Added URL[file:/root/.m2/repository/org/twitter4j/twitter4j-stream/3.0.3/twitter4j-stream-3.0.3.jar]
2015-03-12T06:37:04,428 INFO [main] io.druid.initialization.Initialization - Added URL[file:/root/.m2/repository/commons-validator/commons-validator/1.4.0/commons-validator-1.4.0.jar]
2015-03-12T06:37:04,428 INFO [main] io.druid.initialization.Initialization - Added URL[file:/root/.m2/repository/commons-beanutils/commons-beanutils/1.8.3/commons-beanutils-1.8.3.jar]
2015-03-12T06:37:04,428 INFO [main] io.druid.initialization.Initialization - Added URL[file:/root/.m2/repository/commons-digester/commons-digester/1.8/commons-digester-1.8.jar]
2015-03-12T06:37:04,429 INFO [main] io.druid.initialization.Initialization - Added URL[file:/root/.m2/repository/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar]
2015-03-12T06:37:04,431 INFO [main] io.druid.initialization.Initialization - Loading extension[io.druid.extensions:druid-kafka-eight] for class[io.druid.cli.CliCommandCreator]
2015-03-12T06:37:04,563 INFO [main] io.druid.initialization.Initialization - Added URL[file:/root/.m2/repository/io/druid/extensions/druid-kafka-eight/0.7.0/druid-kafka-eight-0.7.0.jar]
2015-03-12T06:37:04,564 INFO [main] io.druid.initialization.Initialization - Added URL[file:/root/.m2/repository/org/apache/kafka/kafka_2.10/[0.8.2.0/kafka_2.10-0.8.2.0.jar](http://0.8.2.0/kafka_2.10-0.8.2.0.jar)]
2015-03-12T06:37:04,564 INFO [main] io.druid.initialization.Initialization - Added URL[file:/root/.m2/repository/org/apache/kafka/kafka-clients/[0.8.2.0/kafka-clients-0.8.2.0.jar](http://0.8.2.0/kafka-clients-0.8.2.0.jar)]
2015-03-12T06:37:04,564 INFO [main] io.druid.initialization.Initialization - Added URL[file:/root/.m2/repository/net/jpountz/lz4/lz4/1.3.0/lz4-1.3.0.jar]
2015-03-12T06:37:04,564 INFO [main] io.druid.initialization.Initialization - Added URL[file:/root/.m2/repository/org/xerial/snappy/snappy-java/[1.1.1.6/snappy-java-1.1.1.6.jar](http://1.1.1.6/snappy-java-1.1.1.6.jar)]
2015-03-12T06:37:04,564 INFO [main] io.druid.initialization.Initialization - Added URL[file:/root/.m2/repository/com/yammer/metrics/metrics-core/2.2.0/metrics-core-2.2.0.jar]
2015-03-12T06:37:04,564 INFO [main] io.druid.initialization.Initialization - Added URL[file:/root/.m2/repository/org/scala-lang/scala-library/2.10.4/scala-library-2.10.4.jar]
2015-03-12T06:37:04,564 INFO [main] io.druid.initialization.Initialization - Added URL[file:/root/.m2/repository/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6.jar]
2015-03-12T06:37:04,565 INFO [main] io.druid.initialization.Initialization - Added URL[file:/root/.m2/repository/org/slf4j/slf4j-log4j12/1.6.1/slf4j-log4j12-1.6.1.jar]
2015-03-12T06:37:04,565 INFO [main] io.druid.initialization.Initialization - Added URL[file:/root/.m2/repository/log4j/log4j/1.2.16/log4j-1.2.16.jar]
2015-03-12T06:37:04,565 INFO [main] io.druid.initialization.Initialization - Added URL[file:/root/.m2/repository/jline/jline/0.9.94/jline-0.9.94.jar]
2015-03-12T06:37:04,565 INFO [main] io.druid.initialization.Initialization - Added URL[file:/root/.m2/repository/io/netty/netty/3.7.0.Final/netty-3.7.0.Final.jar]
2015-03-12T06:37:04,565 INFO [main] io.druid.initialization.Initialization - Added URL[file:/root/.m2/repository/net/sf/jopt-simple/jopt-simple/3.2/jopt-simple-3.2.jar]
2015-03-12T06:37:04,565 INFO [main] io.druid.initialization.Initialization - Added URL[file:/root/.m2/repository/com/101tec/zkclient/0.3/zkclient-0.3.jar]
2015-03-12T06:37:04,570 INFO [main] io.druid.initialization.Initialization - Loading extension[io.druid.extensions:mysql-metadata-storage] for class[io.druid.cli.CliCommandCreator]
2015-03-12T06:37:04,999 INFO [main] io.druid.initialization.Initialization - Added URL[file:/root/.m2/repository/io/druid/extensions/mysql-metadata-storage/0.7.0/mysql-metadata-storage-0.7.0.jar]
2015-03-12T06:37:05,000 INFO [main] io.druid.initialization.Initialization - Added URL[file:/root/.m2/repository/mysql/mysql-connector-java/5.1.33/mysql-connector-java-5.1.33.jar]
2015-03-12T06:37:05,000 INFO [main] io.druid.initialization.Initialization - Added URL[file:/root/.m2/repository/org/jdbi/jdbi/2.32/jdbi-2.32.jar]
2015-03-12T06:37:05,206 INFO [main] io.druid.initialization.Initialization - Loading extension[io.druid.extensions:druid-examples] for class[io.druid.initialization.DruidModule]
2015-03-12T06:37:05,207 INFO [main] io.druid.initialization.Initialization - Adding extension module[class io.druid.examples.ExamplesDruidModule] for class[io.druid.initialization.DruidModule]
2015-03-12T06:37:05,208 INFO [main] io.druid.initialization.Initialization - Loading extension[io.druid.extensions:druid-kafka-eight] for class[io.druid.initialization.DruidModule]
2015-03-12T06:37:05,209 INFO [main] io.druid.initialization.Initialization - Adding extension module[class io.druid.firehose.kafka.KafkaEightDruidModule] for class[io.druid.initialization.DruidModule]
2015-03-12T06:37:05,209 INFO [main] io.druid.initialization.Initialization - Loading extension[io.druid.extensions:mysql-metadata-storage] for class[io.druid.initialization.DruidModule]
2015-03-12T06:37:05,210 INFO [main] io.druid.initialization.Initialization - Adding extension module[class io.druid.metadata.storage.mysql.MySQLMetadataStorageModule] for class[io.druid.initialization.DruidModule]
2015-03-12T06:37:05,994 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.metrics.DruidMonitorSchedulerConfig] from props[druid.monitoring.] as [io.druid.server.metrics.DruidMonitorSchedulerConfig@60e284f5]
2015-03-12T06:37:06,008 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.metrics.MonitorsConfig] from props[druid.monitoring.] as [MonitorsConfig{monitors=[]}]
2015-03-12T06:37:06,035 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.DruidNode] from props[druid.] as [DruidNode{serviceName='overlord', host='10.65.204.10', port=8090}]
2015-03-12T06:37:06,063 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.initialization.ServerConfig] from props[druid.server.http.] as [ServerConfig{numThreads=40, maxIdleTime=PT5M}]
2015-03-12T06:37:06,077 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.indexing.common.config.TaskConfig] from props[druid.indexer.task.] as [io.druid.indexing.common.config.TaskConfig@71385a8]
2015-03-12T06:37:06,085 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.http.DruidHttpClientConfig] from props[druid.global.http.] as [io.druid.guice.http.DruidHttpClientConfig@130e02d2]
2015-03-12T06:37:06,193 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.client.indexing.IndexingServiceSelectorConfig] from props[druid.selectors.indexing.] as [io.druid.client.indexing.IndexingServiceSelectorConfig@24a59823]
2015-03-12T06:37:06,200 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.curator.CuratorConfig] from props[druid.zk.service.] as [io.druid.curator.CuratorConfig@46728703]
2015-03-12T06:37:06,207 WARN [main] org.apache.curator.retry.ExponentialBackoffRetry - maxRetries too large (30). Pinning to 29
2015-03-12T06:37:06,245 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.initialization.CuratorDiscoveryConfig] from props[druid.discovery.curator.] as [io.druid.server.initialization.CuratorDiscoveryConfig@1f3e29be]
2015-03-12T06:37:06,498 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.indexing.common.RetryPolicyConfig] from props[druid.peon.taskActionClient.retry.] as [io.druid.indexing.common.RetryPolicyConfig@760e2359]
2015-03-12T06:37:06,503 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.segment.loading.LocalDataSegmentPusherConfig] from props[druid.storage.] as [io.druid.segment.loading.LocalDataSegmentPusherConfig@4f9c0948]
2015-03-12T06:37:06,527 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.client.DruidServerConfig] from props[druid.server.] as [io.druid.client.DruidServerConfig@2f6b7827]
2015-03-12T06:37:06,533 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.initialization.BatchDataSegmentAnnouncerConfig] from props[druid.announcer.] as [io.druid.server.initialization.BatchDataSegmentAnnouncerConfig@1553f9b2]
2015-03-12T06:37:06,552 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.initialization.ZkPathsConfig] from props[druid.zk.paths.] as [io.druid.server.initialization.ZkPathsConfig@58d3f4be]
2015-03-12T06:37:06,560 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[interface io.druid.server.coordination.DataSegmentAnnouncerProvider] from props[druid.announcer.] as [io.druid.server.coordination.BatchDataSegmentAnnouncerProvider@221614c2]
2015-03-12T06:37:06,568 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[interface io.druid.client.FilteredServerViewProvider] from props[druid.announcer.] as [io.druid.client.FilteredBatchServerViewProvider@5f717b35]
2015-03-12T06:37:06,583 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.QueryConfig] from props[druid.query.] as [io.druid.query.QueryConfig@727721e6]
2015-03-12T06:37:06,602 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.search.search.SearchQueryConfig] from props[druid.query.search.] as [io.druid.query.search.search.SearchQueryConfig@19b18458]
2015-03-12T06:37:06,611 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.groupby.GroupByQueryConfig] from props[druid.query.groupBy.] as [io.druid.query.groupby.GroupByQueryConfig@5dbe43af]
2015-03-12T06:37:06,615 INFO [main] org.skife.config.ConfigurationObjectFactory - Assigning value [100000000] for [druid.computation.buffer.size] on [io.druid.query.DruidProcessingConfig#intermediateComputeSizeBytes()]
2015-03-12T06:37:06,619 INFO [main] org.skife.config.ConfigurationObjectFactory - Assigning value [1] for [druid.processing.numThreads] on [io.druid.query.DruidProcessingConfig#getNumThreads()]
2015-03-12T06:37:06,619 INFO [main] org.skife.config.ConfigurationObjectFactory - Using method itself for [${base_path}.columnCache.sizeBytes] on [io.druid.query.DruidProcessingConfig#columnCacheSizeBytes()]
2015-03-12T06:37:06,621 INFO [main] org.skife.config.ConfigurationObjectFactory - Assigning default value [processing-%s] for [${base_path}.formatString] on [com.metamx.common.concurrent.ExecutorServiceConfig#getFormatString()]
2015-03-12T06:37:06,735 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.topn.TopNQueryConfig] from props[druid.query.topN.] as [io.druid.query.topn.TopNQueryConfig@5c1d3ee2]
2015-03-12T06:37:06,754 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[interface io.druid.server.log.RequestLoggerProvider] from props[druid.request.logging.] as [io.druid.server.log.NoopRequestLoggerProvider@6f65b863]
2015-03-12T06:37:06,766 INFO [main] org.eclipse.jetty.util.log - Logging initialized @5480ms
2015-03-12T06:37:06,859 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void com.metamx.emitter.service.ServiceEmitter.start()] on object[com.metamx.emitter.service.ServiceEmitter@21b953c3].
2015-03-12T06:37:06,859 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void com.metamx.metrics.MonitorScheduler.start()] on object[com.metamx.metrics.MonitorScheduler@e2a3881].
2015-03-12T06:37:06,862 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void com.metamx.http.client.NettyHttpClient.start()] on object[com.metamx.http.client.NettyHttpClient@2034da9].
2015-03-12T06:37:06,862 INFO [main] io.druid.curator.CuratorModule - Starting Curator
2015-03-12T06:37:06,862 INFO [main] org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
2015-03-12T06:37:06,873 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2015-03-12T06:37:06,873 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:[host.name](http://host.name/)=localhost
2015-03-12T06:37:06,873 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.7.0_75
2015-03-12T06:37:06,874 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation
2015-03-12T06:37:06,874 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/jdk1.7.0/jre
2015-03-12T06:37:06,874 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=config/_common:config/overlord:lib/curator-framework-2.7.0.jar:lib/opencsv-2.3.jar:lib/activation-1.1.1.jar:lib/validation-api-1.1.0.Final.jar:lib/jdbi-2.32.jar:lib/jersey-servlet-1.17.1.jar:lib/guice-4.0-beta.jar:lib/maven-model-builder-3.1.1.jar:lib/disruptor-3.3.0.jar:lib/protobuf-java-2.5.0.jar:lib/compress-lzf-1.0.3.jar:lib/druid-common-0.7.0.jar:lib/okhttp-1.0.2.jar:lib/jackson-module-jaxb-annotations-2.4.4.

package com.paypal.pxpinsights;

import backtype.storm.task.IMetricsContext;

import com.google.common.base.Throwables;

import com.google.common.collect.ImmutableList;

import com.metamx.common.Granularity;

import com.metamx.tranquility.beam.Beam;

import com.metamx.tranquility.beam.ClusteredBeamTuning;

import com.metamx.tranquility.druid.DruidBeams;

import com.metamx.tranquility.druid.DruidDimensions;

import com.metamx.tranquility.druid.DruidEnvironment;

import com.metamx.tranquility.druid.DruidLocation;

import com.metamx.tranquility.druid.DruidRollup;

import com.metamx.tranquility.storm.BeamFactory;

import com.metamx.tranquility.typeclass.Timestamper;

import io.druid.data.input.impl.TimestampSpec;

import io.druid.granularity.QueryGranularity;

import io.druid.query.aggregation.AggregatorFactory;

import io.druid.query.aggregation.CountAggregatorFactory;

import io.druid.query.aggregation.MinAggregatorFactory;

import org.apache.curator.framework.CuratorFramework;

import org.apache.curator.framework.CuratorFrameworkFactory;

import org.apache.curator.retry.RetryOneTime;

import org.joda.time.DateTime;

import org.joda.time.Period;

import java.util.List;

import java.util.Map;

public class DruidBeamFactory implements BeamFactory<Map<String, Object>> {

//private static final String ZOOKEEPER_CONNECTION_STRING = “10.65.204.111:2181”;

private static final String ZOOKEEPER_CONNECTION_STRING = “10.65.204.172:2181”;

private static final int BASE_SLEEP_TIME = 100,

MAX_SLEEP_TIME_MILLIS = 1000,

MAX_RETRIES = 5;

public Beam<Map<String, Object>> makeBeam(Map<?, ?> map, IMetricsContext imc) {

try {

// final CuratorFramework curator = CuratorFrameworkFactory.newClient(ZOOKEEPER_CONNECTION_STRING,

// new BoundedExponentialBackoffRetry(

// BASE_SLEEP_TIME,

// MAX_SLEEP_TIME_MILLIS,

// MAX_RETRIES)

// );

final CuratorFramework curator = CuratorFrameworkFactory

.builder()

.connectString(ZOOKEEPER_CONNECTION_STRING)

.retryPolicy(new RetryOneTime(MAX_SLEEP_TIME_MILLIS))

.sessionTimeoutMs(30 * 1000)

.build();

curator.start();

// TODO: Set the dimensions correctly

final List dimensions = ImmutableList.of(PxpEventDeserializeScheme.NAME,

PxpEventDeserializeScheme.EXPT_ID,

PxpEventDeserializeScheme.TR_ID,

“xmg1”);

final String dataSource = “pxpsoa1234”,

indexService = “overlord”,

fireHosePattern = “druid:firehose:%s”,

discoveryPath = “/druid/discovery”;

final List aggregators = ImmutableList.of(

new CountAggregatorFactory(“cnt”),

new MinAggregatorFactory(“max_value”, “value”)

);

/* final DruidBeams.Builder<Map<String, Object>> builder = DruidBeams

.builder(

new Timestamper<Map<String, Object>>() {

@Override

public DateTime timestamp(Map<String, Object> theMap) {

System.out.println(" In Tranquil times " + theMap.get(PxpEventDeserializeScheme.TIME_STAMP));

return new DateTime(theMap.get(PxpEventDeserializeScheme.TIME_STAMP));

}

}

)

.curator(curator)

.discoveryPath(discoveryPath)

.timestampSpec(new TimestampSpec(“timestamp”,“auto”))

.location(

// DruidLocation.create(

// indexService, fireHosePattern, dataSource

// )

new DruidLocation(new DruidEnvironment(indexService, fireHosePattern), dataSource)

)

.rollup(DruidRollup.create(DruidDimensions.specific(dimensions), aggregators, QueryGranularity.MINUTE))

// Duration of 10 months from current…

.tuning(ClusteredBeamTuning

.builder()

.segmentGranularity(Granularity.MINUTE)

.windowPeriod(new Period(“PT10M”))

.partitions(1)

.replicants(1)

.build()

);

return builder.buildBeam();*/

final DruidBeams.Builder<Map<String, Object>> builder = DruidBeams

.builder(

new Timestamper<Map<String, Object>>() {

@Override

public DateTime timestamp(Map<String, Object> theMap) {

System.out.println(" In Tranquil times " + theMap.get(PxpEventDeserializeScheme.TIME_STAMP));

return new DateTime(theMap.get(PxpEventDeserializeScheme.TIME_STAMP));

}

}

)

.curator(curator)

.discoveryPath(discoveryPath)

.timestampSpec(new TimestampSpec(“timestamp”,“auto”))

.location(

// DruidLocation.create(

// indexService, fireHosePattern, dataSource

// )

new DruidLocation(new DruidEnvironment(indexService, fireHosePattern), dataSource)

)

.rollup(DruidRollup.create(DruidDimensions.specific(dimensions), aggregators, QueryGranularity.MINUTE))

// Duration of 10 months from current…

.tuning(ClusteredBeamTuning

.builder()

.segmentGranularity(Granularity.DAY)

.windowPeriod(new Period(“P25D”))

//.windowPeriod(new Period(“PT10M”))

//.windowPeriod(new Period(“PT1M”))

.partitions(1)

// .replicants(1)

.replicants(1)

.build()

);

return builder.buildBeam();

} catch (Exception e) {

throw Throwables.propagate(e);

}

}

}

``

Noting that I believe this is the same issue as this other thread: https://groups.google.com/forum/#!topic/druid-user/eRyWQFZleOU

Hi Gopi, that config looks reasonable enough to me. I wonder, if you restart everything in UTC, does it still get stuck? And if so, what does it get stuck on?

Hi Gopi, have you had a chance to give the UTC thing a try and see where things get stuck?

Thanks Gian for checking back. After setting the timzone to UTC on all Druid components, Kafka and storm most of the problems went away. We had other issues that Dhawan ( my colleague ) sorted it out after receiving some guidance from you and Fangjin.
Thank you very much.

Great to hear!

I know it’s an old post, but I’m hitting a similar issue, using tranquility-core_2.10
I’m unable to create new datasource (dynamically) and the only error I can find is in the zk log :

org.apache.zookeeper.server.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x15926226a82073f type:create cx

id:0x14 zxid:0x91708 txntype:-1 reqpath:n/a Error Path:/tranquility/beams/druid:overlord/LuckyFish_prod/mutex/leases Error:KeeperErrorCode = NoNode for /tranquility/beams/druid:overlord/LuckyFish_prod/mut

ex/leases

When sending the same message on an existing datasource it works.

any help will be appreciated.

Hey Yair,

This looks like it’s due to a Curator bug that gets triggered when you lose your ZK session. See more here: https://github.com/druid-io/tranquility/issues/205

You should be able to work around this by setting up your system to lose ZK sessions less often. Possible approaches here are improving network connectivity, giving more hardware to ZK (if it’s under heavy load), or similar strategies. When the fixed Curator is released we will upgrade.