thetaSketch error

Manohar_Rao · January 29, 2016, 7:05am

Hi
I m trying thetaSketch and running into some error

{"version":"0.8.3","modules":[{"name":"io.druid.examples.ExamplesDruidModule","artifact":"druid-examples","version":"0.8.3"},{"name":"io.druid.firehose.kafka.KafkaEightDruidModule","artifact":"druid-kafka-eight","version":"0.8.3"}],"memory":{"maxMemory":5726797824,"totalMemory":442499072,"freeMemory":381018376,"usedMemory":61480696}}

My index spec is attached

Error :

<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 </title>
</head>
<body>
<h2>HTTP ERROR: 500</h2>
<p>Problem accessing /druid/indexer/v1/task. Reason:
<pre>    javax.servlet.ServletException: com.fasterxml.jackson.databind.JsonMappingException: Could not resolve type id 'thetaSketch' into a subtype of [simple type, class io.druid.query.aggregation.AggregatorFactory]
 at [Source: HttpInputOverHTTP@b11c130; line: 1, column: 652] (through reference chain: Object[][3])</pre></p>
<hr /><i><small>Powered by Jetty://</small></i>
</body>
</html>

What could be wrong?

Thanks and Regards

Manohar

thetatest.json (1.34 KB)

Fangjin · January 29, 2016, 7:21am

You need to include the theta sketches extension.

Manohar_Rao · January 29, 2016, 7:33am

Thanks ,

Sorry , but do you have any link on how to add the extension? .I am pretty new to druid.

Thanks and Regards

Manohar

mcapeloto · January 29, 2016, 7:57am

It goes in the druid.extensions.coordinates of your common.runtime.properties config for all your druid nodes. Should look something like this (give or take the other extensions):

druid.extensions.coordinates=[“io.druid.extensions:druid-datasketches”,“io.druid.extensions:druid-kafka-eight”,“io.druid.extensions:mysql-metadata-storage”]

Michael

Manohar_Rao · January 29, 2016, 9:00am

Great !
Thanks

Lovenish_Goyal · June 30, 2016, 2:25pm

Hi Fangjin,

We are also facing same issue while using thetaSkech.

We have sketches extensions added in common property file.

druid.extensions.loadList=[“druid-datasketches”,“druid-kafka-indexing-service”]

All druid services are loading this extension while getting up but when we start tranquility and ingest event it gets down with below error. Seems like tranquility is not getting this extension.

2016-06-30 12:09:42,767 [KafkaConsumer-0] INFO io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, directory=‘extensions’, hadoopDependenciesDir=‘hadoop-dependencies’, loadList=null}]
2016-06-30 12:09:42,910 [KafkaConsumer-0] ERROR c.m.tranquility.kafka.KafkaConsumer - Exception:
java.lang.IllegalArgumentException: Could not resolve type id ‘thetaSketch’ into a subtype of [simple type, class io.druid.query.aggregation.AggregatorFactory]
at [Source: N/A; line: -1, column: -1] (through reference chain: Object[1])
at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:2774) ~[com.fasterxml.jackson.core.jackson-databind-2.4.6.jar:2.4.6]
at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:2700) ~[com.fasterxml.jackson.core.jackson-databind-2.4.6.jar:2.4.6]
at com.metamx.tranquility.druid.DruidBeams$.makeFireDepartment(DruidBeams.scala:406) ~[io.druid.tranquility-core-0.8.0.jar:0.8.0]
at com.metamx.tranquility.druid.DruidBeams$.fromConfigInternal(DruidBeams.scala:291) ~[io.druid.tranquility-core-0.8.0.jar:0.8.0]
at com.metamx.tranquility.druid.DruidBeams$.fromConfig(DruidBeams.scala:199) ~[io.druid.tranquility-core-0.8.0.jar:0.8.0]
at com.metamx.tranquility.kafka.KafkaBeamUtils$.createTranquilizer(KafkaBeamUtils.scala:40) ~[io.druid.tranquility-kafka-0.8.0.jar:0.8.0]
at com.metamx.tranquility.kafka.KafkaBeamUtils.createTranquilizer(KafkaBeamUtils.scala) ~[io.druid.tranquility-kafka-0.8.0.jar:0.8.0]
at com.metamx.tranquility.kafka.writer.TranquilityEventWriter.(TranquilityEventWriter.java:64) ~[io.druid.tranquility-kafka-0.8.0.jar:0.8.0]
at com.metamx.tranquility.kafka.writer.WriterController.createWriter(WriterController.java:171) ~[io.druid.tranquility-kafka-0.8.0.jar:0.8.0]
at com.metamx.tranquility.kafka.writer.WriterController.getWriter(WriterController.java:98) ~[io.druid.tranquility-kafka-0.8.0.jar:0.8.0]
at com.metamx.tranquility.kafka.KafkaConsumer$2.run(KafkaConsumer.java:231) ~[io.druid.tranquility-kafka-0.8.0.jar:0.8.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_65]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_65]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_65]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_65]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]

Is there anything which we need to configure for tranquility as well.

Here is the historical cluster log which shows that extension is added in common config file.

2016-06-30T11:25:44,248 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, directory=‘extensions’, hadoopDependenciesDir=‘hadoop-dependencies’, hadoopContainerDruidClasspath=‘null’, loadList=[druid-datasketches, druid-kafka-indexing-service]}]

Waiting for your response.

Thanks,

mcapeloto · June 30, 2016, 2:46pm

Hi try adding -Ddruid.extensions.loadList=’[“druid-datasketches”]’ -Ddruid.extensions.directory=/druid/path/extensions to your tranquility JVM properties. https://github.com/druid-io/tranquility/blob/master/docs/configuration.md

Lovenish_Goyal · June 30, 2016, 3:00pm

Thanks Mcap, Its woking

Shantanu_Deshmukh · November 9, 2016, 10:22am

I tried the same. It’s not working for me. Where exactly did you add these options? I added them in bin/tranquility shell file. Added those arguments like
addJava “-Dlogback.configurationFile=${app_home}/…/conf/logback.xml -Ddruid.extensions.loadList=’[“druid-datasketches”]’ -Ddruid.extensions.directory=dist/druid/extensions”

``

I am still getting the same error

Lovenish_Goyal · November 9, 2016, 10:33am

Shantanu,

Add druid.extensions.loadList=[“druid-datasketches”,“druid-kafka-indexing-service”] in your common property file and pass same argument while running tranquility.

Eg:

bin/tranquility kafka -configFile conf/kafka.json -Ddruid.extensions.loadList=’[“druid-datasketches”]’ -Ddruid.extensions.directory=/opt/druid/extensions

druid-datasketches jar should be in your extension directory path.

Shantanu_Deshmukh · November 9, 2016, 1:56pm

Hey,
Thanks a lot. That helped!

ravi_bhatta · September 8, 2017, 3:31pm

Hi , I am trying to include a community extension “druid-distinctcount”. I have downoaded the extension by using pull-deps and included it in common.runtime.properties.
But i am getting error ;

{“error”:“Could not resolve type id ‘distinctCount’ into a subtype of [simple type, class io.druid.query.aggregation.AggregatorFactory]\n at [Source: HttpInputOverHTTP@30582c42[c=5289,q=1,[0]=EOF,s=STREAM]; line: 1, column: 4882] (through reference chain: Object[3])”}

I have included druid.extensions.loadList=[“druid-distinctcount”] in common.runtime.properties. I have also restarted all druid nodes after this.

Plz help me solve this issue.

Thank You

Ravi Bhatta