thetaSketch error

Hi
I m trying thetaSketch and running into some error

{"version":"0.8.3","modules":[{"name":"io.druid.examples.ExamplesDruidModule","artifact":"druid-examples","version":"0.8.3"},{"name":"io.druid.firehose.kafka.KafkaEightDruidModule","artifact":"druid-kafka-eight","version":"0.8.3"}],"memory":{"maxMemory":5726797824,"totalMemory":442499072,"freeMemory":381018376,"usedMemory":61480696}}

My index spec is attached
Error :

<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 </title>
</head>
<body>
<h2>HTTP ERROR: 500</h2>
<p>Problem accessing /druid/indexer/v1/task. Reason:
<pre>    javax.servlet.ServletException: com.fasterxml.jackson.databind.JsonMappingException: Could not resolve type id 'thetaSketch' into a subtype of [simple type, class io.druid.query.aggregation.AggregatorFactory]
 at [Source: HttpInputOverHTTP@b11c130; line: 1, column: 652] (through reference chain: Object[][3])</pre></p>
<hr /><i><small>Powered by Jetty://</small></i>
</body>
</html>


What could be wrong?

Thanks and Regards

Manohar

thetatest.json (1.34 KB)

You need to include the theta sketches extension.

Thanks ,

Sorry , but do you have any link on how to add the extension? .I am pretty new to druid.

Thanks and Regards

Manohar

It goes in the druid.extensions.coordinates of your common.runtime.properties config for all your druid nodes. Should look something like this (give or take the other extensions):

druid.extensions.coordinates=[“io.druid.extensions:druid-datasketches”,“io.druid.extensions:druid-kafka-eight”,“io.druid.extensions:mysql-metadata-storage”]

Michael

Great !
Thanks

Hi Fangjin,

We are also facing same issue while using thetaSkech.

We have sketches extensions added in common property file.

druid.extensions.loadList=[“druid-datasketches”,“druid-kafka-indexing-service”]

All druid services are loading this extension while getting up but when we start tranquility and ingest event it gets down with below error. Seems like tranquility is not getting this extension.

2016-06-30 12:09:42,767 [KafkaConsumer-0] INFO io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, directory=‘extensions’, hadoopDependenciesDir=‘hadoop-dependencies’, loadList=null}]
2016-06-30 12:09:42,910 [KafkaConsumer-0] ERROR c.m.tranquility.kafka.KafkaConsumer - Exception:
java.lang.IllegalArgumentException: Could not resolve type id ‘thetaSketch’ into a subtype of [simple type, class io.druid.query.aggregation.AggregatorFactory]
at [Source: N/A; line: -1, column: -1] (through reference chain: Object[1])
at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:2774) ~[com.fasterxml.jackson.core.jackson-databind-2.4.6.jar:2.4.6]
at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:2700) ~[com.fasterxml.jackson.core.jackson-databind-2.4.6.jar:2.4.6]
at com.metamx.tranquility.druid.DruidBeams$.makeFireDepartment(DruidBeams.scala:406) ~[io.druid.tranquility-core-0.8.0.jar:0.8.0]
at com.metamx.tranquility.druid.DruidBeams$.fromConfigInternal(DruidBeams.scala:291) ~[io.druid.tranquility-core-0.8.0.jar:0.8.0]
at com.metamx.tranquility.druid.DruidBeams$.fromConfig(DruidBeams.scala:199) ~[io.druid.tranquility-core-0.8.0.jar:0.8.0]
at com.metamx.tranquility.kafka.KafkaBeamUtils$.createTranquilizer(KafkaBeamUtils.scala:40) ~[io.druid.tranquility-kafka-0.8.0.jar:0.8.0]
at com.metamx.tranquility.kafka.KafkaBeamUtils.createTranquilizer(KafkaBeamUtils.scala) ~[io.druid.tranquility-kafka-0.8.0.jar:0.8.0]
at com.metamx.tranquility.kafka.writer.TranquilityEventWriter.(TranquilityEventWriter.java:64) ~[io.druid.tranquility-kafka-0.8.0.jar:0.8.0]
at com.metamx.tranquility.kafka.writer.WriterController.createWriter(WriterController.java:171) ~[io.druid.tranquility-kafka-0.8.0.jar:0.8.0]
at com.metamx.tranquility.kafka.writer.WriterController.getWriter(WriterController.java:98) ~[io.druid.tranquility-kafka-0.8.0.jar:0.8.0]
at com.metamx.tranquility.kafka.KafkaConsumer$2.run(KafkaConsumer.java:231) ~[io.druid.tranquility-kafka-0.8.0.jar:0.8.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_65]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_65]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_65]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_65]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]

Is there anything which we need to configure for tranquility as well.

Here is the historical cluster log which shows that extension is added in common config file.

2016-06-30T11:25:44,248 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, directory=‘extensions’, hadoopDependenciesDir=‘hadoop-dependencies’, hadoopContainerDruidClasspath=‘null’, loadList=[druid-datasketches, druid-kafka-indexing-service]}]

Waiting for your response.

Thanks,

Hi try adding -Ddruid.extensions.loadList=’[“druid-datasketches”]’ -Ddruid.extensions.directory=/druid/path/extensions to your tranquility JVM properties. https://github.com/druid-io/tranquility/blob/master/docs/configuration.md

Thanks Mcap, Its woking

I tried the same. It’s not working for me. Where exactly did you add these options? I added them in bin/tranquility shell file. Added those arguments like
addJava “-Dlogback.configurationFile=${app_home}/…/conf/logback.xml -Ddruid.extensions.loadList=’[“druid-datasketches”]’ -Ddruid.extensions.directory=dist/druid/extensions”

``

``

I am still getting the same error

Shantanu,

Add druid.extensions.loadList=[“druid-datasketches”,“druid-kafka-indexing-service”] in your common property file and pass same argument while running tranquility.

Eg:

bin/tranquility kafka -configFile conf/kafka.json -Ddruid.extensions.loadList=’[“druid-datasketches”]’ -Ddruid.extensions.directory=/opt/druid/extensions

druid-datasketches jar should be in your extension directory path.

Hey,
Thanks a lot. That helped! :slight_smile:

Hi , I am trying to include a community extension “druid-distinctcount”. I have downoaded the extension by using pull-deps and included it in common.runtime.properties.
But i am getting error ;

{“error”:“Could not resolve type id ‘distinctCount’ into a subtype of [simple type, class io.druid.query.aggregation.AggregatorFactory]\n at [Source: HttpInputOverHTTP@30582c42[c=5289,q=1,[0]=EOF,s=STREAM]; line: 1, column: 4882] (through reference chain: Object[3])”}

I have included druid.extensions.loadList=[“druid-distinctcount”] in common.runtime.properties. I have also restarted all druid nodes after this.

Plz help me solve this issue.

Thank You

Ravi Bhatta