Problem in ingestion with Tranquility

I try to ingest data with Tranquility from Kafka. But work can’t start,below is my json configuration and log for Tranquility.

{

*** “dataSources” : [***

*** {***

*** “spec” : {***

*** “dataSchema” : {***

*** “parser” : {***

*** “type” : “string”,***

*** “parseSpec” : {***

*** “timestampSpec” : {***

*** “format” : “auto”,***

*** “column” : “timestamp”***

*** },***

*** “dimensionsSpec” : {***

*** “spatialDimensions” : [***

*** { }***

*** ],***

*** “dimensions” : [***

*** “page”,“lag”,“lagtype”***

*** ]***

*** },***

*** “format” : “json”***

*** }***

*** },***

*** “dataSource” : “api-tranquility”,***

*** “granularitySpec” : {***

*** “segmentGranularity” : “hour”,***

*** “type” : “uniform”,***

*** “queryGranularity” : “none”***

*** },***

*** “metricsSpec” : [***

*** {***

*** “type” : “count”,***

*** “name” : “count”***

*** }***

*** ]***

*** },***

*** “tuningConfig” : {***

*** “maxRowsInMemory” : “100000”,***

*** “type” : “realtime”,***

*** “windowPeriod” : “PT10M”,***

*** “intermediatePersistPeriod” : “PT10M”***

*** }***

*** },***

*** “properties” : {***

*** “topicPattern” : “api-druid”,***

*** “topicPattern.priority” : “1”,***

*** }***

*** }***

*** ],***

*** “properties” : {***

*** “zookeeper.connect” : “10.10.10.10:2181”,***

*** “zookeeper.timeout” : “PT15S”,***

*** “druid.selectors.indexing.serviceName” : “overlord”,***

*** “druid.discovery.curator.path” : “/data/druid/realtime/tranquility”,***

*** “kafka.zookeeper.connect” : “10.10.10.11:2181”,***

*** “kafka.group.id” : “tranquility-api”,***

*** “consumer.numThreads” : “15”,***

*** “commit.periodMillis” : “500”,***

*** “reportDropsAsExceptions” : “false”***

*** }***

}

below is log:

2016-02-24 08:28:20,625 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [2] for [consumer.numThreads] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getConsumerNumThreads()]

2016-02-24 08:28:20,636 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning default value [(?!)] for [topicPattern] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getTopicPattern()]

2016-02-24 08:28:20,637 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [tranquility-kafka] for [kafka.group.id] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getKafkaGroupId()]

2016-02-24 08:28:20,637 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [localhost:2181] for [kafka.zookeeper.connect] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getKafkaZookeeperConnect()]

2016-02-24 08:28:20,637 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning default value [false] for [useTopicAsDataSource] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#useTopicAsDataSource()]

2016-02-24 08:28:20,638 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning default value [1] for [topicPattern.priority] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getTopicPatternPriority()]

2016-02-24 08:28:20,638 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [15000] for [commit.periodMillis] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getCommitPeriodMillis()]

2016-02-24 08:28:20,639 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [false] for [reportDropsAsExceptions] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#reportDropsAsExceptions()]

2016-02-24 08:28:20,639 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [PT20S] for [zookeeper.timeout] on [com.metamx.tranquility.config.PropertiesBasedConfig#zookeeperTimeout()]

2016-02-24 08:28:20,726 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [druid/overlord] for [druid.selectors.indexing.serviceName] on [com.metamx.tranquility.config.PropertiesBasedConfig#druidIndexingServiceName()]

2016-02-24 08:28:20,726 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [task.partitions] on [com.metamx.tranquility.config.PropertiesBasedConfig#taskPartitions()]

2016-02-24 08:28:20,727 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [task.replicants] on [com.metamx.tranquility.config.PropertiesBasedConfig#taskReplicants()]

2016-02-24 08:28:20,728 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [task.warmingPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#taskWarmingPeriod()]

2016-02-24 08:28:20,728 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [localhost:2181] for [zookeeper.connect] on [com.metamx.tranquility.config.PropertiesBasedConfig#zookeeperConnect()]

2016-02-24 08:28:20,728 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [tranquility.maxBatchSize] on [com.metamx.tranquility.config.PropertiesBasedConfig#tranquilityMaxBatchSize()]

2016-02-24 08:28:20,732 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [tranquility.maxPendingBatches] on [com.metamx.tranquility.config.PropertiesBasedConfig#tranquilityMaxPendingBatches()]

2016-02-24 08:28:20,733 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [tranquility.lingerMillis] on [com.metamx.tranquility.config.PropertiesBasedConfig#tranquilityLingerMillis()]

2016-02-24 08:28:20,733 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [/druid/discovery] for [druid.discovery.curator.path] on [com.metamx.tranquility.config.PropertiesBasedConfig#discoPath()]

2016-02-24 08:28:20,733 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseGracePeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseGracePeriod()]

2016-02-24 08:28:20,733 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseQuietPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseQuietPeriod()]

2016-02-24 08:28:20,734 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseRetryPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseRetryPeriod()]

2016-02-24 08:28:20,734 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseChunkSize] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseChunkSize()]

2016-02-24 08:28:20,734 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.randomizeTaskId] on [com.metamx.tranquility.config.PropertiesBasedConfig#randomizeTaskId()]

2016-02-24 08:28:20,735 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.indexRetryPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#indexRetryPeriod()]

2016-02-24 08:28:20,735 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseBufferSize] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseBufferSize()]

2016-02-24 08:28:20,925 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [2] for [consumer.numThreads] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getConsumerNumThreads()]

2016-02-24 08:28:20,926 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [twitter] for [topicPattern] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getTopicPattern()]

2016-02-24 08:28:20,926 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [tranquility-kafka] for [kafka.group.id] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getKafkaGroupId()]

2016-02-24 08:28:20,926 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [localhost:2181] for [kafka.zookeeper.connect] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getKafkaZookeeperConnect()]

2016-02-24 08:28:20,927 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning default value [false] for [useTopicAsDataSource] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#useTopicAsDataSource()]

2016-02-24 08:28:20,927 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [1] for [topicPattern.priority] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getTopicPatternPriority()]

2016-02-24 08:28:20,928 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [15000] for [commit.periodMillis] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getCommitPeriodMillis()]

2016-02-24 08:28:20,928 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [false] for [reportDropsAsExceptions] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#reportDropsAsExceptions()]

2016-02-24 08:28:20,928 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [PT20S] for [zookeeper.timeout] on [com.metamx.tranquility.config.PropertiesBasedConfig#zookeeperTimeout()]

2016-02-24 08:28:20,929 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [druid/overlord] for [druid.selectors.indexing.serviceName] on [com.metamx.tranquility.config.PropertiesBasedConfig#druidIndexingServiceName()]

2016-02-24 08:28:20,929 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [task.partitions] on [com.metamx.tranquility.config.PropertiesBasedConfig#taskPartitions()]

2016-02-24 08:28:20,929 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [task.replicants] on [com.metamx.tranquility.config.PropertiesBasedConfig#taskReplicants()]

2016-02-24 08:28:20,930 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [task.warmingPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#taskWarmingPeriod()]

2016-02-24 08:28:20,930 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [localhost:2181] for [zookeeper.connect] on [com.metamx.tranquility.config.PropertiesBasedConfig#zookeeperConnect()]

2016-02-24 08:28:20,930 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [tranquility.maxBatchSize] on [com.metamx.tranquility.config.PropertiesBasedConfig#tranquilityMaxBatchSize()]

2016-02-24 08:28:20,930 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [tranquility.maxPendingBatches] on [com.metamx.tranquility.config.PropertiesBasedConfig#tranquilityMaxPendingBatches()]

2016-02-24 08:28:20,931 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [tranquility.lingerMillis] on [com.metamx.tranquility.config.PropertiesBasedConfig#tranquilityLingerMillis()]

2016-02-24 08:28:20,931 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [/druid/discovery] for [druid.discovery.curator.path] on [com.metamx.tranquility.config.PropertiesBasedConfig#discoPath()]

2016-02-24 08:28:20,931 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseGracePeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseGracePeriod()]

2016-02-24 08:28:20,932 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseQuietPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseQuietPeriod()]

2016-02-24 08:28:20,932 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseRetryPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseRetryPeriod()]

2016-02-24 08:28:20,932 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseChunkSize] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseChunkSize()]

2016-02-24 08:28:20,932 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.randomizeTaskId] on [com.metamx.tranquility.config.PropertiesBasedConfig#randomizeTaskId()]

2016-02-24 08:28:20,933 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.indexRetryPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#indexRetryPeriod()]

2016-02-24 08:28:20,933 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseBufferSize] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseBufferSize()]

2016-02-24 08:28:20,937 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [2] for [consumer.numThreads] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getConsumerNumThreads()]

2016-02-24 08:28:20,938 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [wikipedia.*] for [topicPattern] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getTopicPattern()]

2016-02-24 08:28:20,939 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [tranquility-kafka] for [kafka.group.id] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getKafkaGroupId()]

2016-02-24 08:28:20,939 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [localhost:2181] for [kafka.zookeeper.connect] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getKafkaZookeeperConnect()]

2016-02-24 08:28:20,939 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning default value [false] for [useTopicAsDataSource] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#useTopicAsDataSource()]

2016-02-24 08:28:20,940 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [1] for [topicPattern.priority] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getTopicPatternPriority()]

2016-02-24 08:28:20,940 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [15000] for [commit.periodMillis] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getCommitPeriodMillis()]

2016-02-24 08:28:20,941 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [false] for [reportDropsAsExceptions] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#reportDropsAsExceptions()]

2016-02-24 08:28:20,941 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [PT20S] for [zookeeper.timeout] on [com.metamx.tranquility.config.PropertiesBasedConfig#zookeeperTimeout()]

2016-02-24 08:28:20,941 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [druid/overlord] for [druid.selectors.indexing.serviceName] on [com.metamx.tranquility.config.PropertiesBasedConfig#druidIndexingServiceName()]

2016-02-24 08:28:20,942 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [2] for [task.partitions] on [com.metamx.tranquility.config.PropertiesBasedConfig#taskPartitions()]

2016-02-24 08:28:20,942 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [2] for [task.replicants] on [com.metamx.tranquility.config.PropertiesBasedConfig#taskReplicants()]

2016-02-24 08:28:20,942 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [task.warmingPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#taskWarmingPeriod()]

2016-02-24 08:28:20,942 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [localhost:2181] for [zookeeper.connect] on [com.metamx.tranquility.config.PropertiesBasedConfig#zookeeperConnect()]

2016-02-24 08:28:20,943 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [tranquility.maxBatchSize] on [com.metamx.tranquility.config.PropertiesBasedConfig#tranquilityMaxBatchSize()]

2016-02-24 08:28:20,943 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [tranquility.maxPendingBatches] on [com.metamx.tranquility.config.PropertiesBasedConfig#tranquilityMaxPendingBatches()]

2016-02-24 08:28:20,943 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [tranquility.lingerMillis] on [com.metamx.tranquility.config.PropertiesBasedConfig#tranquilityLingerMillis()]

2016-02-24 08:28:20,943 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [/druid/discovery] for [druid.discovery.curator.path] on [com.metamx.tranquility.config.PropertiesBasedConfig#discoPath()]

2016-02-24 08:28:20,944 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseGracePeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseGracePeriod()]

2016-02-24 08:28:20,944 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseQuietPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseQuietPeriod()]

2016-02-24 08:28:20,945 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseRetryPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseRetryPeriod()]

2016-02-24 08:28:20,945 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseChunkSize] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseChunkSize()]

2016-02-24 08:28:20,946 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.randomizeTaskId] on [com.metamx.tranquility.config.PropertiesBasedConfig#randomizeTaskId()]

2016-02-24 08:28:20,946 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.indexRetryPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#indexRetryPeriod()]

2016-02-24 08:28:20,946 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseBufferSize] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseBufferSize()]

2016-02-24 08:28:20,967 [main] WARN o.a.c.retry.ExponentialBackoffRetry - maxRetries too large (500). Pinning to 29

Exception in thread “main” java.lang.NoSuchMethodError: java.util.List.sort(Ljava/util/Comparator;)V

*** at com.metamx.tranquility.kafka.writer.WriterController.(WriterController.java:58)***

*** at com.metamx.tranquility.kafka.KafkaMain.run(KafkaMain.java:122)***

*** at com.metamx.tranquility.kafka.KafkaMain.main(KafkaMain.java:71)***

*** at com.metamx.tranquility.distribution.DistributionMain$.main(DistributionMain.scala:37)***

*** at com.metamx.tranquility.distribution.DistributionMain.main(DistributionMain.scala)***

It is the same problem with example test.

Hi,

I resolved the same exception by upgrading java7 to java-8.

Sorry, yes, this was a mistake in the original release of tranquility-kafka. It was intended to support java 7 too. The next release will support both java 7 and 8 (master already does). Sorry for the inconvenience.

Hi, Gian:

Could you help to view this thread?https://groups.google.com/forum/#!topic/druid-user/qNvGG2ogH2A

There are some exception when spark streaming ingest data with tranquility.

在 2016年2月25日星期四 UTC+8上午9:52:15,Gian Merlino写道:

yep, was just commenting there.