Tranquility JavaExample doesn't send any data to druid, doesn't show any errors

Hi ,

I tried the example program at : https://github.com/druid-io/tranquility/blob/master/core/src/test/java/com/metamx/tranquility/example/JavaExample.java

My tranquility configuration file is :

{

“dataSources” : [

{

“spec” : {

“dataSchema” : {

“dataSource”:“pageviews_1”,

“parser”:{

“type”:“string”,

“parseSpec”:{

“timestampSpec”:{

“column”:“time”,

“format”:“auto”

},

“dimensionsSpec”:{

“dimensions”:[

“url”,

“user”,

“os”

],

“dimensionExclusions”:[

“time”

]

},

“format”:“json”

}

},

“granularitySpec”:{

“type”:“uniform”,

“segmentGranularity”:“fifteen_minute”,

“queryGranularity”:“fifteen_minute”

},

“metricsSpec”:[

{

“name”:“views”,

“type”:“count”

}

]

},

“tuningConfig” : {

“type” : “realtime”,

“windowPeriod” : “PT10M”,

“intermediatePersistPeriod” : “PT10M”,

“maxRowsInMemory” : “100000”

}

},

“properties” : {

“task.partitions” : “1”,

“task.replicants” : “1”

}

}

],

“properties” : {

“zookeeper.connect” : “localhost”

}

}

``

JavaExample.java

package com.example;

import java.io.InputStream;

import java.util.Map;

import com.google.common.collect.ImmutableMap;

import com.metamx.common.logger.Logger;

import com.metamx.tranquility.config.DataSourceConfig;

import com.metamx.tranquility.config.PropertiesBasedConfig;

import com.metamx.tranquility.config.TranquilityConfig;

import com.metamx.tranquility.druid.DruidBeams;

import com.metamx.tranquility.tranquilizer.MessageDroppedException;

import com.metamx.tranquility.tranquilizer.Tranquilizer;

import com.twitter.util.FutureEventListener;

import scala.runtime.BoxedUnit;

public class JavaExample {

private static final Logger log = new Logger(JavaExample.class);

public static void main(String args)

{

// Read config from “example.json” on the classpath.

final InputStream configStream = JavaExample.class.getResourceAsStream(“example.json”);

final TranquilityConfig config = TranquilityConfig.read(configStream);

final DataSourceConfig wikipediaConfig = config.getDataSource(“pageviews_1”);

final Tranquilizer<Map<String, Object>> sender = DruidBeams.fromConfig(wikipediaConfig)

.buildTranquilizer(wikipediaConfig.tranquilizerBuilder());

sender.start();

try {

// Send 10000 objects

for (int i = 0; i < 10; i++) {

// Build a sample event to send; make sure we use a current date

final Map<String, Object> obj = ImmutableMap.<String, Object>of(

“time”, “2018-11-07T06:13:43+00:00”,

“url”, “foo”,

“user”, “tintin”

);

// Asynchronously send event to Druid:

sender.send(obj).addEventListener(

new FutureEventListener()

{

@Override

public void onSuccess(BoxedUnit value)

{

log.info(“Sent message: %s”, obj);

}

@Override

public void onFailure(Throwable e)

{

if (e instanceof MessageDroppedException) {

log.warn(e, “Dropped message: %s”, obj);

} else {

log.error(e, “Failed to send message: %s”, obj);

}

}

}

);

}

}

finally {

sender.flush();

sender.stop();

}

}

}

``

I ensure to put a recent timestamp while executing the program. But i couldn’t see any realtime task created in the coordinator console. Neither any error in shown after executing the program. the program just keeps running as shown below.

My druid setup is running on a VM which is accessible from my machine on which i am executing the program. I can send data over HTTP from my machine to the tranquility server for which realtime tasks are created successfully. But the example code doesn’t work. I also tried setting zookeeper.connect to IP of VM where druid is running, still no luck.

Am i missing something here? does any of the configs look incorrect?

Thanks,

Prathamesh

I tried the sample program again and added following properties to the config file:

“druid.discovery.curator.path” : “/druid/discovery”,

“druid.selectors.indexing.serviceName” : “druid/overlord”,

“http.port” : “8200”,

``

This time i got:

Failed to create merged beam: druid:overlord/pageviews

``

If the datasource exists i.e. created by sending couple of events directly over HTTP, can’t we use tranquility library to send additional data?

I already have “pageviews” datasource. How do i send additional data into this datasource using Tranquility library?

I found the solution to this.

We have to set druid.host in the properties file ( *conf/druid/_common/common.runtime.properties *) to machine’s reachable IP address.

I have noticed that, this is however not required when using tranquility server directly. But when one wants to use tranquility core module (tranquility-core_2.11) this change would be required.

Thanks,

Prathamesh