Error when sending data to Druid through Tranquility

Hey !

I am new to Druid and still in the process of discovering the technologies…
So far, I have completed the Getting Started section, and in particular the loading stream data section. This works so far !

Now I would like to use Tranquility so that I can also stream data from my own application. I have been trying to use the sample code and reusing the same server configuration json file as in the getting started example.

server.json

{
“dataSources” : {
“pageviews” : {
“spec” : {
“dataSchema” : {
“dataSource” : “pageviews”,
“parser” : {
“type” : “string”,
“parseSpec” : {
“timestampSpec” : {
“column” : “time”,
“format” : “auto”
},
“dimensionsSpec” : {
“dimensions” : [“url”, “user”]
},
“format” : “json”
}
},
“granularitySpec” : {
“type” : “uniform”,
“segmentGranularity” : “hour”,
“queryGranularity” : “none”
},
“metricsSpec” : [
{“name”: “views”, “type”: “count”},
{“name”: “latencyMs”, “type”: “doubleSum”, “fieldName”: “latencyMs”}
]
},
“ioConfig” : {
“type” : “realtime”
},
“tuningConfig” : {
“type” : “realtime”,
“maxRowsInMemory” : “100000”,
“intermediatePersistPeriod” : “PT10M”,
“windowPeriod” : “PT10M”
}
},
“properties” : {
“task.partitions” : “1”,
“task.replicants” : “1”
}
}
},
“properties” : {
“zookeeper.connect” : “localhost”,
“druid.discovery.curator.path” : “/druid/discovery”,
“druid.selectors.indexing.serviceName” : “druid/overlord”,
“http.port” : “8200”,
“http.threads” : “8”
}
}

The test code:

public class Main {
private static final Logger log = new Logger(Main.class);

public static void main(String[] args) {

    // Read config from "server.json" on the classpath.
    final InputStream configStream = Main.class.getClassLoader().getResourceAsStream("server.json");
    final TranquilityConfig<PropertiesBasedConfig> config = TranquilityConfig.read(configStream);
    final DataSourceConfig<PropertiesBasedConfig> pageViewConf = config.getDataSource("pageviews");
    final Tranquilizer<Map<String, Object>> sender = DruidBeams.fromConfig(pageViewConf)
            .buildTranquilizer(pageViewConf.tranquilizerBuilder());

    sender.start();

    try {
        // Send 10000 objects
        for (int i = 0; i < 10000; i++) {
            // Build a sample event to send; make sure we use a current date
            final Map<String, Object> obj = ImmutableMap.<String, Object>of(
                    "time", new DateTime().toString(),
                    "url", "/foo/bar",
                    "user", "alice",
                    "latencyMs", i
            );

            // Asynchronously send event to Druid:
            sender.send(obj).addEventListener(
                    new FutureEventListener<BoxedUnit>() {

                        public void onSuccess(BoxedUnit value) {
                            log.info("Sent message: %s", obj);
                        }

                        public void onFailure(Throwable e) {
                            if (e instanceof MessageDroppedException) {
                                log.warn(e, "Dropped message: %s", obj);
                            } else {
                                log.error(e, "Failed to send message: %s", obj);
                            }
                        }
                    }
            );
        }
    } finally {
        sender.flush();
        sender.stop();
    }
}

}

I have started all 5 nodes (historical, broker, coordinator, overlord and middleManager) on local (I am still doing everything local so far) and when starting the code, I get the following error:

Exception in thread “main” java.lang.IllegalArgumentException: Instantiation of [simple type, class io.druid.data.input.impl.StringInputRowParser] value failed: com.metamx.common.parsers.JSONPathParser
at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:2774)
at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:2700)

Caused by: com.fasterxml.jackson.databind.JsonMappingException: Instantiation of [simple type, class io.druid.data.input.impl.StringInputRowParser] value failed: com.metamx.common.parsers.JSONPathParser
at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.wrapException(StdValueInstantiator.java:405)
at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith(StdValueInstantiator.java:234)


Caused by: java.lang.ClassNotFoundException: com.metamx.common.parsers.JSONPathParser
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

From the error I suspect it is because the compiler does not find com.metamx.common.parsers.JSONPathParser? If this is the case, how can I include this in my project?

Thank for your help !

Pascal

Okay,

After manually adding the missing packages to my classpath as well as multiple other ones, it worked :).
I think it would be nice to update the documentation about that or to provide a starting project with the dependencies resolved !

Thanks for your work anyway !

Hi Pascal, great to see you’ve resolved your issue. Tranquility docs are located at https://github.com/druid-io/tranquility/tree/master/docs. It would be great if you could help contribute an update around your workarounds.

Hi Fangjin !

Thanks for your reply. Yes, I read the whole docs about Tranquility and did not find the solution to my problem, unfortunately… As I said, I solved it by importing all the missing dependencies, manually. As a starting point, I took the Java Example code and imported the Core API. However, it appears that this is not enough to run the example code because I was still missing JSONPathParser, which can be found here. I made a jar out of it and included it on my classpath (did not find it on maven…). In my specific case, I also needed jsonpath from jayway, which can be found here. Adding these dependencies was enough for me! :slight_smile:

Anyway, thanks for your work, keep up !

Hey,

I am having similar issue with Tranquility core API. Issue started for me since tranquility 0.8.0. Adding JSONPathParser.java to your source and including json-path artifact it started working again. Current workaround seems hackish and I am not sure how good idea it would be to update docs to use that.

Hey Rasmus,

This should be getting pulled in automatically as a dependency, especially if you are using Maven (or similar) to incorporate tranquility-core into your project. Is that not happening?

Hey Gian,

Yes, it looks like it is not imported correctly.

Here is the pom.xml i’m using:

<project xmlns=“http://maven.apache.org/POM/4.0.0” xmlns:xsi=“http://www.w3.org/2001/XMLSchema-instance

xsi:schemaLocation=“http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd”>

4.0.0

com.mycompany.app

my-app

jar

1.0-SNAPSHOT

my-app

http://maven.apache.org

junit

junit

3.8.1

test

io.druid

tranquility-core_2.11

0.8.2

<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>

<maven.compiler.source>1.8</maven.compiler.source>

<maven.compiler.target>1.8</maven.compiler.target>

org.apache.maven.plugins

maven-dependency-plugin

2.10

copy-dependencies

package

copy-dependencies

${project.build.directory}/lib

``

And here is some test code:

package com.mycompany.app;

import java.io.InputStream;

import java.util.Map;

import com.metamx.tranquility.config.DataSourceConfig;

import com.metamx.tranquility.config.PropertiesBasedConfig;

import com.metamx.tranquility.config.TranquilityConfig;

import com.metamx.tranquility.druid.DruidBeams;

import com.metamx.tranquility.tranquilizer.Tranquilizer;

public class App {

public static void main( String args ) throws InterruptedException {

final InputStream configStream = App.class.getClassLoader().getResourceAsStream(“indexing.yml”);

final TranquilityConfig druidConfig = TranquilityConfig.read(configStream);

String dataSource = druidConfig.globalConfig().properties().getProperty(“datasource”);

final DataSourceConfig dataSourceConfig = druidConfig.getDataSource(dataSource);

Tranquilizer<Map<String, Object>> service = DruidBeams.fromConfig(dataSourceConfig).buildTranquilizer(

dataSourceConfig.tranquilizerBuilder()

);

service.start();

}

}

``

Running this will crash with output:

Exception in thread “main” java.lang.IllegalArgumentException: Instantiation of [simple type, class io.druid.data.input.impl.StringInputRowParser] value failed: com.metamx.common.parsers.JSONPathParser

at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:2774)

at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:2700)

at io.druid.segment.indexing.DataSchema.getParser(DataSchema.java:101)

at com.metamx.tranquility.druid.DruidBeams$.fromConfigInternal(DruidBeams.scala:301)

at com.metamx.tranquility.druid.DruidBeams$.fromConfig(DruidBeams.scala:204)

at com.metamx.tranquility.druid.DruidBeams$.fromConfig(DruidBeams.scala:123)

at com.metamx.tranquility.druid.DruidBeams.fromConfig(DruidBeams.scala)

at com.mycompany.app.App.main(App.java:32)

Caused by: com.fasterxml.jackson.databind.JsonMappingException: Instantiation of [simple type, class io.druid.data.input.impl.StringInputRowParser] value failed: com.metamx.common.parsers.JSONPathParser

at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.wrapException(StdValueInstantiator.java:405)

at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith(StdValueInstantiator.java:234)

at com.fasterxml.jackson.databind.deser.impl.PropertyBasedCreator.build(PropertyBasedCreator.java:167)

at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:398)

at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1064)

at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:264)

at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:156)

at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:126)

at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:113)

at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:84)

at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:132)

at com.fasterxml.jackson.databind.deser.impl.TypeWrappedDeserializer.deserialize(TypeWrappedDeserializer.java:41)

at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:2769)

… 7 more

Caused by: java.lang.ClassNotFoundException: com.metamx.common.parsers.JSONPathParser

at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

at io.druid.data.input.impl.JSONParseSpec.makeParser(JSONParseSpec.java:72)

at io.druid.data.input.impl.StringInputRowParser.(StringInputRowParser.java:56)

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

at com.fasterxml.jackson.databind.introspect.AnnotatedConstructor.call(AnnotatedConstructor.java:125)

at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith(StdValueInstantiator.java:230)

… 18 more

``

<!-- https://mvnrepository.com/artifact/com.metamx/java-util -->
<dependency>
    <groupId>com.metamx</groupId>
    <artifactId>java-util</artifactId>
    <version>0.27.9</version>
</dependency>
<!-- https://mvnrepository.com/artifact/joda-time/joda-time -->
<dependency>
    <groupId>joda-time</groupId>
    <artifactId>joda-time</artifactId>
    <version>2.9.4</version>
</dependency>


在 2016年5月18日星期三 UTC+8下午4:53:27,Pascal写道: