Jackson problem: Spark and Tranquility

Hi, list.

We are working on a spark application that sends messages to Druid. For that, we’re using Tranquility core. In my local test, I’m using the “spark-1.6.1-bin-hadoop2.6” distribution and the following dependencies in my app:

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming_2.10</artifactId>
    <version>1.6.1</version>
    <scope>provided</scope>
</dependency>
<dependency>
    <groupId>io.druid</groupId>
    <artifactId>tranquility-core_2.10</artifactId>
    <version>0.7.4</version>
</dependency>

But i getting the error down below when Tranquility tries to create Tranquilizer object:

tranquilizer = DruidBeams.fromConfig(dataSourceConfig).buildTranquilizer(tranquilizerBuider);

The stacktrace is down below:

java.lang.IllegalAccessError: tried to access method com.fasterxml.jackson.databind.introspect.AnnotatedMember.getAllAnnotations()Lcom/fasterxml/jackson/databind/introspect/AnnotationMap; from class com.fasterxml.jackson.databind.introspect.GuiceAnnotationIntrospector
at com.fasterxml.jackson.databind.introspect.GuiceAnnotationIntrospector.findInjectableValueId(GuiceAnnotationIntrospector.java:39)
at com.fasterxml.jackson.databind.introspect.AnnotationIntrospectorPair.findInjectableValueId(AnnotationIntrospectorPair.java:269)
at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory._addDeserializerConstructors(BasicDeserializerFactory.java:433)
at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory._constructDefaultValueInstantiator(BasicDeserializerFactory.java:325)
at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.findValueInstantiator(BasicDeserializerFactory.java:266)
at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.buildBeanDeserializer(BeanDeserializerFactory.java:266)
at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.createBeanDeserializer(BeanDeserializerFactory.java:168)
at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:399)
at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:348)
at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:261)
at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:241)
at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)
at com.fasterxml.jackson.databind.DeserializationContext.findContextualValueDeserializer(DeserializationContext.java:380)
at com.fasterxml.jackson.databind.deser.impl.PropertyBasedCreator.construct(PropertyBasedCreator.java:96)
at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.resolve(BeanDeserializerBase.java:413)
at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:292)
at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:241)
at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)
at com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:394)
at com.fasterxml.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:3169)
at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:2767)
at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:2700)
at com.metamx.tranquility.druid.DruidBeams$.fromConfigInternal(DruidBeams.scala:192)
at com.metamx.tranquility.druid.DruidBeams$.fromConfig(DruidBeams.scala:119)
at com.metamx.tranquility.druid.DruidBeams.fromConfig(DruidBeams.scala)

Does someone faced that problem too?

Hey Marcelo,

This usually happens if you have mismatched versions of the jackson-* jars on your classpath. It might help to do a mvn dependency:list to find all of your jackson- dependencies and then pin them all to a specific version in your pom.xml.

Hi, Gian.

Thanks for the tip. I ran mvn dependency:list and get:

[INFO] com.fasterxml.jackson.module:jackson-module-scala_2.10:jar:2.4.5:compile
[INFO] com.fasterxml.jackson.core:jackson-databind:jar:2.4.6:compile
[INFO] com.fasterxml.jackson.datatype:jackson-datatype-joda:jar:2.4.6:compile
[INFO] com.fasterxml.jackson.dataformat:jackson-dataformat-smile:jar:2.4.6:compile
[INFO] com.fasterxml.jackson.jaxrs:jackson-jaxrs-smile-provider:jar:2.4.6:compile
[INFO] com.fasterxml.jackson.jaxrs:jackson-jaxrs-base:jar:2.4.6:compile
[INFO] com.fasterxml.jackson.module:jackson-module-jaxb-annotations:jar:2.4.6:compile
[INFO] com.fasterxml.jackson.core:jackson-core:jar:2.4.6:compile
[INFO] com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:jar:2.4.6:compile
[INFO] com.fasterxml.jackson.datatype:jackson-datatype-guava:jar:2.4.6:compile
[INFO] com.fasterxml.jackson.core:jackson-annotations:jar:2.4.6:compile
[INFO] org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
[INFO] org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO] com.google.http-client:google-http-client-jackson2:jar:1.15.0-rc:compile
[INFO] org.json4s:json4s-jackson_2.10:jar:3.2.10:provided
[INFO] org.codehaus.jackson:jackson-xc:jar:1.8.3:provided
[INFO] org.codehaus.jackson:jackson-jaxrs:jar:1.8.3:provided

Some of those are used by Tranquility and the last 3 were provided by spark-streaming dependency. I used assembly plugin to build a jar with dependencies but i dont get exactly how spark run my app. Does it run using jackson inside my jar-with-dependencies or if it uses the jackson provided by spark. What is missing?

The downgrade for jackson on Tranquility was made for compabilities with Spark, is that right?

Hi, Gian and List.

Just to close the thread. Unfortunately, I didnt solve the jackson lib problem but I did a workaround that works fine for me. Perhaps this help
another one.

The problem raised from this line when I try to create tranquilizer object (used to connect to Druid) using this utility fromConfig.

tranquilizer = DruidBeams.fromConfig(dataSourceConfig).buildTranquilizer(tranquilizerBuider);

Instead, I create the tranquilizer object-by-object as showed below:

DruidDimensions dimentions = getDimensions(props);
List<AggregatorFactory> aggregators = getAggregations(props);

TimestampSpec timestampSpec = new TimestampSpec("timestamp", "auto", null);
Timestamper<Map<String, Object>> timestamper = map -> new DateTime(map.get("timestamp"));
DruidLocation druidLocation = DruidLocation.create("overlord", "druid:firehose:%s", dataSource);
DruidRollup druidRollup = DruidRollup.create(dimentions, aggregators, QueryGranularity.ALL);

ClusteredBeamTuning clusteredBeamTuning = ClusteredBeamTuning.builder()
                                                    .segmentGranularity(Granularity.FIVE_MINUTE)
                                                    .windowPeriod(new Period("PT60m"))
                                                    .partitions(1)
                                                    .replicants(1)
                                                    .build();

tranquilizer = DruidBeams.builder(timestamper)
               .curator(buildCurator(props))
               .discoveryPath("/druid/discovery")
               .location(druidLocation)
               .timestampSpec(timestampSpec)
               .rollup(druidRollup)
               .tuning(clusteredBeamTuning)
               .buildTranquilizer();

tranquilizer.start();

That worked for me.
Thank you, Gian.