Anyone has theta data sketch column working with Tranquility?

Hi,

I am trying to setup my tranquility sever to work with theta sketch for my metric column. Anyone has any experience on this that can share? Thanks!

My spec looks something like this:

{
"spec": {
   "dataSchema": {
      "dataSource": "event_stats",
      "metricsSpec": [{
         "type": "count",
         "name": "count"
      },{
        "type" : "thetaSketch",
        "name" : "unique_ep",
        "fieldName" : "endpointId"
      }],
      "granularitySpec": {
         "segmentGranularity": "hour",
         "queryGranularity": "hour",
         "type": "uniform",
         "rollup": true
      },
      "parser": {
         "type": "string",
         "parseSpec": {
            "format": "json",
            "timestampSpec": {
               "column": "eventTime",
               "format": "auto"
            },
            "dimensionsSpec": {
               "dimensions": ["eventName",
                                 "carrierId",
                           "eventType",
                           "tapType",
                           "fieldName",
                           "fieldTitle",
                           "screenName",
                           "fieldAction",
                           "source",
                           "storeName",
                           "project",
                           "success",
                           "requestType",
                           "pushNotificationsEnabled",
                           "screenReaderEnabled",
                           "switchControlEnabled",
                           "speakScreenEnabled",
                           "homeTier",
                           "newHomeTier",
                           "oldHomeTier",
                           "deviceModel",
                           "deviceVendor",
                           "endpointId"]
            }
         }
      }
   }
......

``

My tranquility startup line looks like this:

!p95 tranquility-server tranquility/bin/tranquility server -configFile conf/tranquility/server_theta.json -Ddruid.extensions.directory=extensions

``

At the tranqulity-server.log I am seeing the all extensions are being loaded under extensions directory of druid including druid-datasketches extension. However the tranquility server is throwing errors:

java.lang.IllegalArgumentException: Could not resolve type id ‘thetaSketch’ into a subtype of [simple type, class io.druid.query.aggregation.AggregatorFactory]

at [Source: N/A; line: -1, column: -1] (through reference chain: Object[1])

    at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:2774)

    at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:2700)

    at com.metamx.tranquility.druid.DruidBeams$.makeFireDepartment(DruidBeams.scala:433)

    at com.metamx.tranquility.druid.DruidBeams$.fromConfigInternal(DruidBeams.scala:299)

    at com.metamx.tranquility.druid.DruidBeams$.fromConfig(DruidBeams.scala:204)

    at com.metamx.tranquility.server.http.ServerMain$$anonfun$2.apply(ServerMain.scala:118)

    at com.metamx.tranquility.server.http.ServerMain$$anonfun$2.apply(ServerMain.scala:98)

    at com.metamx.common.scala.collection.package$MapLikeOps$$anonfun$strictMapValues$1.apply(package.scala:143)

    at com.metamx.common.scala.collection.package$MapLikeOps$$anonfun$strictMapValues$1.apply(package.scala:143)

    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)

    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)

    at scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:221)

    at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428)

    at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)

    at scala.collection.AbstractTraversable.map(Traversable.scala:104)

    at com.metamx.common.scala.collection.package$MapLikeOps.strictMapValues(package.scala:143)

    at com.metamx.tranquility.server.http.ServerMain$.createServlet(ServerMain.scala:98)

    at com.metamx.tranquility.server.http.ServerMain$.main(ServerMain.scala:77)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

    at java.lang.reflect.Method.invoke(Method.java:498)

    at com.twitter.app.App$$anonfun$nonExitingMain$3.apply(App.scala:168)

    at com.twitter.app.App$$anonfun$nonExitingMain$3.apply(App.scala:167)

    at scala.Option.foreach(Option.scala:257)

    at com.twitter.app.App$class.nonExitingMain(App.scala:167)

    at com.metamx.tranquility.server.http.ServerMain$.nonExitingMain(ServerMain.scala:49)

    at com.twitter.app.App$class.main(App.scala:133)

    at com.metamx.tranquility.server.http.ServerMain$.main(ServerMain.scala:49)

    at com.metamx.tranquility.distribution.DistributionMain$.main(DistributionMain.scala:34)

    at com.metamx.tranquility.distribution.DistributionMain.main(DistributionMain.scala)

``

BTW, on the druid common.runtime.properties. I have already specify the datasketches on the load list like this.

druid.extensions.loadList=[“postgresql-metadata-storage”,“druid-s3-extensions”,“druid-histogram”, “druid-datasketches”, “druid-lookups-cached-global”]

``

Mystery solved. There is a mismatch between druid aggregator type name and tranquility.

https://groups.google.com/forum/#!topic/druid-user/XRw6CkzPV34

When I use hyperUnique as the aggregator type, it works.

Those are different aggregators: hyperUnique is built into Druid and thetaSketch is part of the datasketches extension.

Tranquillity is somewhat out of date these days as it hasn’t been rebuilt in a while. It might be missing support for some newer Druid aggregator types. If using direct Kafka or Kinesis ingestion makes sense for your use case, please consider it, as they are builtin and totally up to date.