issue loading data in local

Hi ,
I am new user to druid.How to create data source and
how to load data to druid on local deep storage.I also want to know how to fetch stored data using java api.

Hi Neeraj, have you had a chance to look over our tutorials? They provide a basic introduction to working with Druid. Druid requests are sent via REST calls and results are returned as JSON objects. Our java api is limited right now, but I’d be curious to understand more about your use case.

Hi Yang,

I have deploy the druid and started all node on a local machine and it working fine for wikipedia.
Now i have created a new topic as druidTest1 and dataSchema as realtime.spec.
After that started kafka and kafka console producer and putting data on console and in consumer it showing the data.
after that i am starting my realtime node and having this error

java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.7.0_55]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[?:1.7.0_55]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.7.0_55]
at java.lang.reflect.Method.invoke(Method.java:606) ~[?:1.7.0_55]
at com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler.start(Lifecycle.java:331) ~[java-util-0.26.15.jar:?]
at com.metamx.common.lifecycle.Lifecycle.start(Lifecycle.java:250) ~[java-util-0.26.15.jar:?]
at io.druid.guice.LifecycleModule$2.start(LifecycleModule.java:136) ~[druid-api-0.3.5.jar:0.7.1.1]
at io.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:71) [druid-services-0.7.1.1.jar:0.7.1.1]
at io.druid.cli.ServerRunnable.run(ServerRunnable.java:38) [druid-services-0.7.1.1.jar:0.7.1.1]
at io.druid.cli.Main.main(Main.java:88) [druid-services-0.7.1.1.jar:0.7.1.1]
Caused by: java.lang.VerifyError: (class: scala/collection/immutable/Nil$, method: head signature: ()Ljava/lang/Object;) Can only throw Throwable objects
at kafka.utils.VerifiableProperties.(VerifiableProperties.scala:27) ~[kafka_2.10-0.8.2.1.jar:?]
at kafka.consumer.ConsumerConfig.(ConsumerConfig.scala:94) ~[kafka_2.10-0.8.2.1.jar:?]
at io.druid.firehose.kafka.KafkaEightFirehoseFactory.connect(KafkaEightFirehoseFactory.java:84) ~[?:?]
at io.druid.firehose.kafka.KafkaEightFirehoseFactory.connect(KafkaEightFirehoseFactory.java:45) ~[?:?]
at io.druid.segment.realtime.FireDepartment.connect(FireDepartment.java:97) ~[druid-server-0.7.1.1.jar:0.7.1.1]
at io.druid.segment.realtime.RealtimeManager$FireChief.init(RealtimeManager.java:207) ~[druid-server-0.7.1.1.jar:0.7.1.1]
at io.druid.segment.realtime.RealtimeManager.start(RealtimeManager.java:109) ~[druid-server-0.7.1.1.jar:0.7.1.1]
… 10 more

Hi Neeraj, can you share your spec? There appear to be some misconfiguration.

Hi Yang,

Actually i have modified the Wikipedia spec .That is realtime.spec as

[{“dataSchema” : {
“dataSource” : “druidTest1”,
“parser” : {
“type” : “string”,
“parseSpec” : {
“format” : “json”,
“timestampSpec” : {
“column” : “timestamp”,
“format” : “auto”
},
“dimensionsSpec” : {
“dimensions”: [“page”,“language”,“user”,“unpatrolled”,“newPage”,“robot”,“anonymous”,“namespace”,“continent”,“country”,“region”,“city”],
“dimensionExclusions” : ,
“spatialDimensions” :
}
}
},
“metricsSpec” : [{
“type” : “count”,
“name” : “count”
}, {
“type” : “doubleSum”,
“name” : “added”,
“fieldName” : “added”
}, {
“type” : “doubleSum”,
“name” : “deleted”,
“fieldName” : “deleted”
}, {
“type” : “doubleSum”,
“name” : “delta”,
“fieldName” : “delta”
}],
“granularitySpec” : {
“type” : “uniform”,
“segmentGranularity” : “DAY”,
“queryGranularity” : “NONE”
}
},
“ioConfig” : {
“type” : “realtime”,
“firehose”: {
“type”: “kafka-0.8”,
“consumerProps”: {
“zookeeper.connect”: “10.64.0.62:2181”,
“zookeeper.connection.timeout.ms” : “15000”,
“zookeeper.session.timeout.ms” : “15000”,
“zookeeper.sync.time.ms” : “5000”,
“group.id”: “test-consumer-group”,
“fetch.message.max.bytes” : “1048586”,
“auto.offset.reset”: “largest”,
“auto.commit.enable”: “false”
},
“feed”: “druidTest1”
},
“plumber”: {
“type”: “realtime”
}
},
“tuningConfig”: {
“type” : “realtime”,
“maxRowsInMemory”: 500000,
“intermediatePersistPeriod”: “PT3m”,
“windowPeriod”: “PT3m”,
“basePersistDirectory”: “/tmp/realtime/basePersist”,
“rejectionPolicy”: {
“type”: “serverTime”
}
}
}
]

How exactly are you invoking Druid when you get that VerifyError?

Recently posted to

Privacy - Terms of Service

Druid User

issue loading data in local

6 posts by 3 authors

15 Apr

me Hi
, I am new user to druid.How to create data source and how to load data
to druid on local deep storage.I also want to know how to fetch stored data using java api.

15 Apr

Fangjin Yang Hi
Neeraj, have you had a chance to look over our tutorials? They provide a
basic introduction to working with Druid. Druid requests are sent via REST calls and results are returned as JSON objects. Our java api is limited right now, but I’d be cu

me (neeraj chauhan change)

17 Apr

Hi Gian Merlino,

I have deploy the druid and started all node on a local machine and it working fine for wikipedia.
Now i have created a new topic as druidTest1 and dataSchema as realtime.spec.
After that started kafka and kafka console producer and putting data on console and in consumer it showing the data.
after that i am starting my realtime node and having this error

java.lang.reflect.
InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.7.0_55]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[?:1.7.0_55]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.7.0_55]
at java.lang.reflect.Method.invoke(Method.java:606) ~[?:1.7.0_55]
at com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler.start(Lifecycle.java:331) ~[java-util-0.26.15.jar:?]
at com.metamx.common.lifecycle.Lifecycle.start(Lifecycle.java:250) ~[java-util-0.26.15.jar:?]
at io.druid.guice.LifecycleModule$2.start(LifecycleModule.java:136) ~[druid-api-0.3.5.jar:0.7.1.1]
at io.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:71) [druid-services-0.7.1.1.jar:0.7.1.1]
at io.druid.cli.ServerRunnable.run(ServerRunnable.java:38) [druid-services-0.7.1.1.jar:0.7.1.1]
at io.druid.cli.Main.main(Main.java:88) [druid-services-0.7.1.1.jar:0.7.1.1]
Caused by: java.lang.VerifyError: (class: scala/collection/immutable/Nil$, method: head signature: ()Ljava/lang/Object;) Can only throw Throwable objects
at kafka.utils.VerifiableProperties.(VerifiableProperties.scala:27) ~[kafka_2.10-0.8.2.1.jar:?]
at kafka.consumer.ConsumerConfig.(ConsumerConfig.scala:94) ~[kafka_2.10-0.8.2.1.jar:?]
at io.druid.firehose.kafka.KafkaEightFirehoseFactory.connect(KafkaEightFirehoseFactory.java:84) ~[?:?]
at io.druid.firehose.kafka.KafkaEightFirehoseFactory.connect(KafkaEightFirehoseFactory.java:45) ~[?:?]
at io.druid.segment.realtime.FireDepartment.connect(FireDepartment.java:97) ~[druid-server-0.7.1.1.jar:0.7.1.1]
at io.druid.segment.realtime.RealtimeManager$FireChief.init(RealtimeManager.java:207) ~[druid-server-0.7.1.1.jar:0.7.1.1]
at io.druid.segment.realtime.RealtimeManager.start(RealtimeManager.java:109) ~[druid-server-0.7.1.1.jar:0.7.1.1]
… 10 more

Hi ,

I want to ingest data into druid and fetch data from druid.Is there any java api for that.
So that i can integrate that into my code and communicate with druid.

Hi Neeraj,
We don’t have a dedicated API for java clients however druid-processing jar contains all the query classes. You can include that and instantiate the query classes e.g TopNQuery, TimeseriesQuery directly or you can use the builder classes available in Druids.java. once you have the created the query query object, it can be serialized to JSON String using DefaultObjectMapper and DirectDruidClient.java (contained in druid-server jar) can be used to submit queries to druid server by providing the host details.

Thanks a lot Nishant,

Actually have gone through the all java class that you have mention in trail mail.
it will be very helpful for me if you can provide the some sample code how to connect with druid and fetch data.

For example you can have a look at druid integration-tests and try to understand how they send queries to brokers,
these files should be good enough to understand the flow -

https://github.com/druid-io/druid/blob/master/integration-tests/src/main/java/io/druid/testing/clients/QueryResourceTestClient.java
https://github.com/druid-io/druid/blob/master/integration-tests/src/main/java/io/druid/testing/utils/FromFileTestQueryHelper.java

Hi Nishant,
I am try to insert data in druid by Sql4D

  . Query is

List inserts = ImmutableList.of(
“INSERT INTO mydruid (timestamp , page , language, user,LONG_SUM(count) as count) "
+ " VALUES (‘2015-04-22 10:03:00’,‘test_page’,‘hi’,‘testuser’,‘1’)”
+ " WHERE interval BETWEEN ‘2015-04-22’ AND ‘2015-04-22’ BREAK BY ‘day’;"

    );

DDataSource driver = new DDataSource(“10.64.0.196”, 8082, “10.64.0.196”, 8081,“10.64.0.196”,8090,
“10.64.0.196”,3306,“diurd”,“druid”,“druid”);

for (String insert : inserts) {
Either<String, Either<Joiner4All,Mapper4All>> insres=driver.query(insert, null, null, true, “sql”);
System.out.println(“insert res–”+insres);
}

and am getting error

insert res–Left(Task failed with server error,

Error 500

HTTP ERROR: 500

Problem accessing /druid/indexer/v1/task. Reason:

    javax.servlet.ServletException: com.fasterxml.jackson.databind.JsonMappingException: Instantiation of [simple type, class io.druid.indexing.common.task.IndexTask] value failed: null

Powered by Jetty:// )

Hi Neeraj, for what it is worth, insertion via SQL4D will generate an index task and is an expensive process. If you want to insert some data into Druid to play with, you can just use a normal index task as outlined in the tutorial:
http://druid.io/docs/latest/Tutorial:-Loading-Batch-Data.html

Hi neeraj, please read the section “compatibility with druid” here. Make sure your are using the right druid and right Sql4D versions. Questions specific to Sql4D you could post it directly to me.

Starters for Sql4D
http://druidwithsql.tumblr.com/post/98578718282/a-first-look-at-druid-with-sql
http://druidwithsql.tumblr.com/post/108054375927/simple-insert-deleting-data-drop-table-in-druid
http://druidwithsql.tumblr.com/post/111014674922/accessing-druid-through-java-driver-interface

Thanks
kalyan

Srikaylan, we should respond to community questions in the community channels. This way others with the same question can search for answers.

Thats right Fangjin, i chime with your thoughts. I created a channel druid-sql4d for this purpose and was just mentioning about this to Eric. It would be great if you could have this note somewhere in the druid.io docs.

Thanks
kalyan