Tranquility and cardinality metrics

I tried to add a cardinality metric to a tranquility server and got a NullPointerException at startup.

After some spelunking I found that tranquility (even on master) is built against older versions of druid which have slightly different ideas about how a metricSpec of type cardinality should look.

(Note: I’m finding this somewhat concerning as I would think that since cardinality is so prominently featured in the druid docs and tranquility is the most common way to do real-time ingest into druid, I shouldn’t be discovering this.)

My goal at the moment is to generate indexing and tranquility ingest configurations from a set of common sources, and partition these sources into their separate ops and dev portions. I can work around tranquility’s and druid’s version mismatch by including both fieldNames, and fields in the metricSpec.

So a couple of questions:

Am I just the first to run across an annoying bug?

Am I using tranquility in a very different way from the rest of the community for no good reason?

Did I make a brown paper bag error? (I’m open to this.)

I found that I was running tranquility-0.8.0 and upgraded to 0.8.2, but I still get the NullPointerException. Here is more info:

The offending metricSpec:

{

“name” : “thing”,

“type” : “cardinality”,

“fields” : [“guild1”, “guid2”, “id3”],

“byRow”: false

},

I can work around this by providing “fieldNames” with fields. But the docs don’t indicate that; I think I had to read code to figure it out: http://druid.io/docs/0.9.2/querying/aggregations.html And I probably would have had problems later if I tried to use the same metricSpec in a batch task.

Tranquility 0.8.2 ships with druid-processing 0.9.1. If I’m reading the code right the change was introduced into 0.9.2 here:

Rolling a new release of tranquility with druid-0.9.2 would solve this in theory.

The exception:

2017-02-02 21:39:49,276 [main] INFO io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, directory=‘extensions’, hadoopDependenciesDir=‘hadoop-dependencies’, hadoopContainerDruidClasspath=‘null’, loadList=null}]

java.lang.NullPointerException

at java.util.AbstractCollection.addAll(AbstractCollection.java:341)

at io.druid.segment.indexing.DataSchema.getParser(DataSchema.java:105)

at com.metamx.tranquility.druid.DruidBeams$.fromConfigInternal(DruidBeams.scala:301)

at com.metamx.tranquility.druid.DruidBeams$.fromConfig(DruidBeams.scala:204)

at com.metamx.tranquility.server.http.ServerMain$$anonfun$2.apply(ServerMain.scala:118)

at com.metamx.tranquility.server.http.ServerMain$$anonfun$2.apply(ServerMain.scala:98)

at com.metamx.common.scala.collection.package$MapLikeOps$$anonfun$strictMapValues$1.apply(package.scala:143)

at com.metamx.common.scala.collection.package$MapLikeOps$$anonfun$strictMapValues$1.apply(package.scala:143)

at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)

at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)

at scala.collection.immutable.Map$Map1.foreach(Map.scala:116)

at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)

at scala.collection.AbstractTraversable.map(Traversable.scala:104)

at com.metamx.common.scala.collection.package$MapLikeOps.strictMapValues(package.scala:143)

at com.metamx.tranquility.server.http.ServerMain$.createServlet(ServerMain.scala:98)

at com.metamx.tranquility.server.http.ServerMain$.main(ServerMain.scala:77)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at com.twitter.app.App$$anonfun$nonExitingMain$3.apply(App.scala:168)

at com.twitter.app.App$$anonfun$nonExitingMain$3.apply(App.scala:167)

at scala.Option.foreach(Option.scala:257)

at com.twitter.app.App$class.nonExitingMain(App.scala:167)

at com.metamx.tranquility.server.http.ServerMain$.nonExitingMain(ServerMain.scala:49)

at com.twitter.app.App$class.main(App.scala:133)

at com.metamx.tranquility.server.http.ServerMain$.main(ServerMain.scala:49)

at com.metamx.tranquility.distribution.DistributionMain$.main(DistributionMain.scala:34)

at com.metamx.tranquility.distribution.DistributionMain.main(DistributionMain.scala)

Exception thrown in main on startup

Hey Darrin,

You’re right, “fields” is newer, introduced in Druid 0.9.2. Tranquility is built against the Druid 0.9.1 API so it expects “fieldNames” rather than “fields” for the cardinality aggregator. The 0.9.1 docs are available here: http://druid.io/docs/0.9.1/querying/aggregations.html and I’d suggest referring to those when writing Tranquility specs.

We’ll probably skip Druid 0.9.2 for Tranquility, since Druid 0.10.0 is coming soon and we are focused on making sure Tranquility works with that API.

Hi Gian, Do you know when 0.10 Release is planned to be release ? Any tentative dates decided for the release ?

Regards,
Arpan Khagram

+91 8308993200