Real time data ingestion with rabbitmq

Hey guys,

I was trying out ingesting real time data into Druid using Rabbit-MQ as the firehose. I have added the druid-rabbitmq-0.7.1.jar file to the lib folder . Starting the realtime node gives the following error .

2015-08-01T19:58:23,291 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.client.ServerInventoryView.start() throws java.lang.Exception] on object[io.druid.client.BatchServerInventoryView@48527904].

2015-08-01T19:58:23,293 INFO [main-SendThread(localhost:2181)] org.apache.zookeeper.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)

2015-08-01T19:58:23,301 INFO [main-SendThread(localhost:2181)] org.apache.zookeeper.ClientCnxn - Socket connection established to localhost/127.0.0.1:2181, initiating session

2015-08-01T19:58:23,309 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.segment.realtime.RealtimeManager.start() throws java.io.IOException] on object[io.druid.segment.realtime.RealtimeManager@273f9f08].

2015-08-01T19:58:23,312 INFO [main] io.druid.segment.realtime.RealtimeManager - Calling the FireDepartment and getting a Firehose.

2015-08-01T19:58:23,312 ERROR [main] io.druid.cli.CliRealtime - Error when starting up. Failing.

java.lang.reflect.InvocationTargetException

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.7.0_75]

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[?:1.7.0_75]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.7.0_75]

at java.lang.reflect.Method.invoke(Method.java:606) ~[?:1.7.0_75]

at com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler.start(Lifecycle.java:331) ~[java-util-0.27.0.jar:?]

at com.metamx.common.lifecycle.Lifecycle.start(Lifecycle.java:250) ~[java-util-0.27.0.jar:?]

at io.druid.guice.LifecycleModule$2.start(LifecycleModule.java:136) ~[druid-api-0.3.8.jar:0.7.3]

at io.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:71) [druid-services-0.7.3.jar:0.7.3]

at io.druid.cli.ServerRunnable.run(ServerRunnable.java:38) [druid-services-0.7.3.jar:0.7.3]

at io.druid.cli.Main.main(Main.java:88) [druid-services-0.7.3.jar:0.7.3]

Caused by: java.lang.NoClassDefFoundError: net/jodah/lyra/ConnectionOptions

at io.druid.firehose.rabbitmq.RabbitMQFirehoseFactory.connect(RabbitMQFirehoseFactory.java:137) ~[druid-rabbitmq-0.7.1.jar:0.7.1]

at io.druid.firehose.rabbitmq.RabbitMQFirehoseFactory.connect(RabbitMQFirehoseFactory.java:100) ~[druid-rabbitmq-0.7.1.jar:0.7.1]

at io.druid.segment.realtime.FireDepartment.connect(FireDepartment.java:97) ~[druid-server-0.7.3.jar:0.7.3]

at io.druid.segment.realtime.RealtimeManager$FireChief.init(RealtimeManager.java:207) ~[druid-server-0.7.3.jar:0.7.3]

at io.druid.segment.realtime.RealtimeManager.start(RealtimeManager.java:109) ~[druid-server-0.7.3.jar:0.7.3]

… 10 more

Caused by: java.lang.ClassNotFoundException: net.jodah.lyra.ConnectionOptions

at java.net.URLClassLoader$1.run(URLClassLoader.java:366) ~[?:1.7.0_75]

at java.net.URLClassLoader$1.run(URLClassLoader.java:355) ~[?:1.7.0_75]

at java.security.AccessController.doPrivileged(Native Method) ~[?:1.7.0_75]

at java.net.URLClassLoader.findClass(URLClassLoader.java:354) ~[?:1.7.0_75]

at java.lang.ClassLoader.loadClass(ClassLoader.java:425) ~[?:1.7.0_75]

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) ~[?:1.7.0_75]

at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ~[?:1.7.0_75]

at io.druid.firehose.rabbitmq.RabbitMQFirehoseFactory.connect(RabbitMQFirehoseFactory.java:137) ~[druid-rabbitmq-0.7.1.jar:0.7.1]

at io.druid.firehose.rabbitmq.RabbitMQFirehoseFactory.connect(RabbitMQFirehoseFactory.java:100) ~[druid-rabbitmq-0.7.1.jar:0.7.1]

at io.druid.segment.realtime.FireDepartment.connect(FireDepartment.java:97) ~[druid-server-0.7.3.jar:0.7.3]

at io.druid.segment.realtime.RealtimeManager$FireChief.init(RealtimeManager.java:207) ~[druid-server-0.7.3.jar:0.7.3]

at io.druid.segment.realtime.RealtimeManager.start(RealtimeManager.java:109) ~[druid-server-0.7.3.jar:0.7.3]

… 10 more

RabbitMQ is up and running properly. Realtime spec file is also configured properly. I am stuck at the above. Please help me to resolve the above error.

Thanks ,

Shantanu

upgraded to druid-rabbitmq-0.7.3.jar but still the problem persists.

Regards,

Shantanu

Hi Shantanu, have you had a chance to check out: http://druid.io/docs/latest/operations/including-extensions.html

RabbitMQ is an extension to Druid, so you’ll likely need to run pull-deps to get all of your dependencies. It appears you are missing some jars in the CP right now.

Hey Fangjin,

I ran the pull-deps command too. Still the same error .

2015-08-02T17:05:50,682 INFO [main] io.druid.segment.realtime.RealtimeManager - Calling the FireDepartment and getting a Firehose.

2015-08-02T17:05:50,683 ERROR [main] io.druid.cli.CliRealtime - Error when starting up. Failing.

java.lang.reflect.InvocationTargetException

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.7.0_75]

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[?:1.7.0_75]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.7.0_75]

at java.lang.reflect.Method.invoke(Method.java:606) ~[?:1.7.0_75]

at com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler.start(Lifecycle.java:331) ~[java-util-0.27.0.jar:?]

at com.metamx.common.lifecycle.Lifecycle.start(Lifecycle.java:250) ~[java-util-0.27.0.jar:?]

at io.druid.guice.LifecycleModule$2.start(LifecycleModule.java:136) ~[druid-api-0.3.8.jar:0.7.3]

at io.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:71) [druid-services-0.7.3.jar:0.7.3]

at io.druid.cli.ServerRunnable.run(ServerRunnable.java:38) [druid-services-0.7.3.jar:0.7.3]

at io.druid.cli.Main.main(Main.java:88) [druid-services-0.7.3.jar:0.7.3]

Caused by: java.lang.NoClassDefFoundError: net/jodah/lyra/ConnectionOptions

at io.druid.firehose.rabbitmq.RabbitMQFirehoseFactory.connect(RabbitMQFirehoseFactory.java:137) ~[druid-rabbitmq-0.7.3.jar:0.7.3]

at io.druid.firehose.rabbitmq.RabbitMQFirehoseFactory.connect(RabbitMQFirehoseFactory.java:100) ~[druid-rabbitmq-0.7.3.jar:0.7.3]

at io.druid.segment.realtime.FireDepartment.connect(FireDepartment.java:97) ~[druid-server-0.7.3.jar:0.7.3]

at io.druid.segment.realtime.RealtimeManager$FireChief.init(RealtimeManager.java:207) ~[druid-server-0.7.3.jar:0.7.3]

at io.druid.segment.realtime.RealtimeManager.start(RealtimeManager.java:109) ~[druid-server-0.7.3.jar:0.7.3]

… 10 more

Caused by: java.lang.ClassNotFoundException: net.jodah.lyra.ConnectionOptions

at java.net.URLClassLoader$1.run(URLClassLoader.java:366) ~[?:1.7.0_75]

at java.net.URLClassLoader$1.run(URLClassLoader.java:355) ~[?:1.7.0_75]

at java.security.AccessController.doPrivileged(Native Method) ~[?:1.7.0_75]

at java.net.URLClassLoader.findClass(URLClassLoader.java:354) ~[?:1.7.0_75]

at java.lang.ClassLoader.loadClass(ClassLoader.java:425) ~[?:1.7.0_75]

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) ~[?:1.7.0_75]

at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ~[?:1.7.0_75]

at io.druid.firehose.rabbitmq.RabbitMQFirehoseFactory.connect(RabbitMQFirehoseFactory.java:137) ~[druid-rabbitmq-0.7.3.jar:0.7.3]

at io.druid.firehose.rabbitmq.RabbitMQFirehoseFactory.connect(RabbitMQFirehoseFactory.java:100) ~[druid-rabbitmq-0.7.3.jar:0.7.3]

at io.druid.segment.realtime.FireDepartment.connect(FireDepartment.java:97) ~[druid-server-0.7.3.jar:0.7.3]

at io.druid.segment.realtime.RealtimeManager$FireChief.init(RealtimeManager.java:207) ~[druid-server-0.7.3.jar:0.7.3]

at io.druid.segment.realtime.RealtimeManager.start(RealtimeManager.java:109) ~[druid-server-0.7.3.jar:0.7.3]

… 10 more

The jodah-time jar file is present in the repo directory(Downloaded after pull-deps command). The druid-rabbitmq-0.7.3. jar file is also present under lib directory.

Thanks ,

Shantanu

I’m having some trouble reproducing the problem with 0.7.3. Can you share how you are running the node and your common.runtime.properties?

I am running the real time node using the following command :

java -Xmx256m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Ddruid.realtime.specFile=config/realtime/realtime.spec -classpath lib/*:config/realtime io.druid.cli.Main server realtime

My common.runtime.properties file is as following :

Hi Shantanu, it doesn’t appear that you are including the common configs in the classpath. Can you try that first?

Hey,

Tried that too …

java -Xmx256m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Ddruid.realtime.specFile=config/realtime/realtime.spec -classpath config/_common:lib/*:config/realtime io.druid.cli.Main server realtime

Still the same error

io.druid.cli.CliRealtime - Error when starting up. Failing.

java.lang.reflect.InvocationTargetException

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.7.0_75]

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[?:1.7.0_75]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.7.0_75]

at java.lang.reflect.Method.invoke(Method.java:606) ~[?:1.7.0_75]

at com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler.start(Lifecycle.java:331) ~[java-util-0.27.0.jar:?]

at com.metamx.common.lifecycle.Lifecycle.start(Lifecycle.java:250) ~[java-util-0.27.0.jar:?]

at io.druid.guice.LifecycleModule$2.start(LifecycleModule.java:136) ~[druid-api-0.3.8.jar:0.7.3]

at io.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:71) [druid-services-0.7.3.jar:0.7.3]

at io.druid.cli.ServerRunnable.run(ServerRunnable.java:38) [druid-services-0.7.3.jar:0.7.3]

at io.druid.cli.Main.main(Main.java:88) [druid-services-0.7.3.jar:0.7.3]

Caused by: java.lang.NoClassDefFoundError: net/jodah/lyra/ConnectionOptions

at io.druid.firehose.rabbitmq.RabbitMQFirehoseFactory.connect(RabbitMQFirehoseFactory.java:137) ~[druid-rabbitmq-0.7.3.jar:0.7.3]

at io.druid.firehose.rabbitmq.RabbitMQFirehoseFactory.connect(RabbitMQFirehoseFactory.java:100) ~[druid-rabbitmq-0.7.3.jar:0.7.3]

at io.druid.segment.realtime.FireDepartment.connect(FireDepartment.java:97) ~[druid-server-0.7.3.jar:0.7.3]

at io.druid.segment.realtime.RealtimeManager$FireChief.init(RealtimeManager.java:207) ~[druid-server-0.7.3.jar:0.7.3]

at io.druid.segment.realtime.RealtimeManager.start(RealtimeManager.java:109) ~[druid-server-0.7.3.jar:0.7.3]

… 10 more

Caused by: java.lang.ClassNotFoundException: net.jodah.lyra.ConnectionOptions

at java.net.URLClassLoader$1.run(URLClassLoader.java:366) ~[?:1.7.0_75]

at java.net.URLClassLoader$1.run(URLClassLoader.java:355) ~[?:1.7.0_75]

at java.security.AccessController.doPrivileged(Native Method) ~[?:1.7.0_75]

at java.net.URLClassLoader.findClass(URLClassLoader.java:354) ~[?:1.7.0_75]

at java.lang.ClassLoader.loadClass(ClassLoader.java:425) ~[?:1.7.0_75]

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) ~[?:1.7.0_75]

at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ~[?:1.7.0_75]

at io.druid.firehose.rabbitmq.RabbitMQFirehoseFactory.connect(RabbitMQFirehoseFactory.java:137) ~[druid-rabbitmq-0.7.3.jar:0.7.3]

at io.druid.firehose.rabbitmq.RabbitMQFirehoseFactory.connect(RabbitMQFirehoseFactory.java:100) ~[druid-rabbitmq-0.7.3.jar:0.7.3]

at io.druid.segment.realtime.FireDepartment.connect(FireDepartment.java:97) ~[druid-server-0.7.3.jar:0.7.3]

at io.druid.segment.realtime.RealtimeManager$FireChief.init(RealtimeManager.java:207) ~[druid-server-0.7.3.jar:0.7.3]

at io.druid.segment.realtime.RealtimeManager.start(RealtimeManager.java:109) ~[druid-server-0.7.3.jar:0.7.3]

… 10 more

Hmmm, that is odd. What happens if you don’t use local extensions and try to use remote extensions? i.e. remove the local extensions config directory and try to have Druid download dependencies remotely

Hey,

Tried the remote extensions too but still the same error . Then I tried this , since the exception was being caused by net.jodah.lyra class , i manually added the jar file lyra-0.5.2.jar(from http://mvnrepository.com/artifact/net.jodah/lyra) to the lib folder (Since it is a compiletime dependency of rabbitmq). Then the exception changed to this :

Caused by: java.lang.NoClassDefFoundError: net/jodah/lyra/retry/RetryPolicy

at io.druid.firehose.rabbitmq.RabbitMQFirehoseFactory.connect(RabbitMQFirehoseFactory.java:138) ~[druid-rabbitmq-0.7.3.jar:0.7.3]

at io.druid.firehose.rabbitmq.RabbitMQFirehoseFactory.connect(RabbitMQFirehoseFactory.java:100) ~[druid-rabbitmq-0.7.3.jar:0.7.3]

at io.druid.segment.realtime.FireDepartment.connect(FireDepartment.java:97) ~[druid-server-0.7.3.jar:0.7.3]

at io.druid.segment.realtime.RealtimeManager$FireChief.init(RealtimeManager.java:207) ~[druid-server-0.7.3.jar:0.7.3]

at io.druid.segment.realtime.RealtimeManager.start(RealtimeManager.java:109) ~[druid-server-0.7.3.jar:0.7.3]

… 10 more

Caused by: java.lang.ClassNotFoundException: net.jodah.lyra.retry.RetryPolicy

at java.net.URLClassLoader$1.run(URLClassLoader.java:366) ~[?:1.7.0_75]

at java.net.URLClassLoader$1.run(URLClassLoader.java:355) ~[?:1.7.0_75]

at java.security.AccessController.doPrivileged(Native Method) ~[?:1.7.0_75]

at java.net.URLClassLoader.findClass(URLClassLoader.java:354) ~[?:1.7.0_75]

at java.lang.ClassLoader.loadClass(ClassLoader.java:425) ~[?:1.7.0_75]

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) ~[?:1.7.0_75]

at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ~[?:1.7.0_75]

at io.druid.firehose.rabbitmq.RabbitMQFirehoseFactory.connect(RabbitMQFirehoseFactory.java:138) ~[druid-rabbitmq-0.7.3.jar:0.7.3]

at io.druid.firehose.rabbitmq.RabbitMQFirehoseFactory.connect(RabbitMQFirehoseFactory.java:100) ~[druid-rabbitmq-0.7.3.jar:0.7.3]

at io.druid.segment.realtime.FireDepartment.connect(FireDepartment.java:97) ~[druid-server-0.7.3.jar:0.7.3]

at io.druid.segment.realtime.RealtimeManager$FireChief.init(RealtimeManager.java:207) ~[druid-server-0.7.3.jar:0.7.3]

at io.druid.segment.realtime.RealtimeManager.start(RealtimeManager.java:109) ~[druid-server-0.7.3.jar:0.7.3]

… 10 more

Does it make sense ??? Or am i doing something wrong ? I dont think compile time dependencies should be downloaded separately.

Thanks

HI Shantanu,
You need to add following jars to lib directory:

joda-time-2.6.jar

lyra-0.3.1.jar

Thanks,

Ravish

Hey Ravish,

Thanks for the update.

When I tried running the real time node after including lyra-0.5.0.jar , the exception raised was : Caused by: java.lang.ClassNotFoundException: net.jodah.lyra.retry.RetryPolicy

I changed it to 0.3.1.jar , the following exception was raised : java.lang.NoSuchMethodError: net.jodah.lyra.config.Config.withRecoveryPolicy(Lnet/jodah/lyra/retry/RetryPolicy;)Lnet/jodah/lyra/config/Config;

I think there is some problem with the version of lyra, which is compatible with druid-0.7.3. Please help !!!

Thanks,

Shantanu

I assume you are using druid-rabbitmq version 0.7.3. If that is the case, then based on its pom file you need to have following jars in lib:
Enter code here…

``

com.rabbitmq
amqp-client
3.2.1


net.jodah
lyra
0.3.1

Make sure you have the correct version of amqp-client.

Hey Ravish,

Yeah , I have included both files in my lib folder , but still stuck on the same error :

Caused by: java.lang.NoSuchMethodError: net.jodah.lyra.config.Config.withRecoveryPolicy(Lnet/jodah/lyra/retry/RetryPolicy;)Lnet/jodah/lyra/config/Config;

Thanks ,

Shantanu

Shantanu, I’m still a bit confused as to why pull-deps is not working for you (I couldn’t reproduce the problem locally) and perhaps more information would help.

Can you share what your .m2 directory looks like, and specifically that the dependencies required for rabbitmq exist in the directory?

There’s an older thread on this issue: https://groups.google.com/forum/#!msg/druid-development/q20FYeIyG7k/xEclJgJfen8J, not sure if any of that info is helpful.

Hey Fangjin,

I noticed this strange thing … even after running the pull-deps command the following directory is empty :

/.m2/repository/io/druid/extensions/druid-rabbitmq/0.7.3

The structure of my .m2 directory is as following :

aopalliance commons-beanutils commons-collections commons-httpclient commons-logging javax net

asm commons-cli commons-configuration commons-io commons-net jline org

com commons-codec commons-digester commons-lang io log4j xmlenc

The structure of my /druid-0.7.3/lib directory is as following

activation-1.1.1.jar extendedset-1.3.8.jar jetty-security-9.2.5.v20141112.jar

aether-api-0.9.0.M2.jar geoip2-0.4.0.jar jetty-server-9.2.5.v20141112.jar

aether-connector-file-0.9.0.M2.jar google-http-client-1.15.0-rc.jar jetty-servlet-9.2.5.v20141112.jar

aether-connector-okhttp-0.0.9.jar google-http-client-jackson2-1.15.0-rc.jar jetty-servlets-9.2.5.v20141112.jar

aether-impl-0.9.0.M2.jar guava-16.0.1.jar jetty-util-9.2.5.v20141112.jar

aether-spi-0.9.0.M2.jar guice-4.0-beta.jar jline-0.9.94.jar

aether-util-0.9.0.M2.jar guice-multibindings-4.0-beta.jar joda-time-2.6.jar

airline-0.6.jar guice-servlet-4.0-beta.jar jsr305-2.0.1.jar

amqp-client-3.2.1.jar hibernate-validator-5.1.3.Final.jar log4j-1.2-api-2.2.jar

antlr4-runtime-4.0.jar http-client-1.0.2.jar log4j-api-2.2.jar

aopalliance-1.0.jar httpclient-4.2.jar log4j-core-2.2.jar

asm-3.1.jar httpcore-4.2.jar log4j-jul-2.2.jar

aws-java-sdk-1.8.11.jar icu4j-4.8.1.jar log4j-slf4j-impl-2.2.jar

aws-java-sdk-core-1.8.11.jar irc-api-1.0-0011.jar lyra-0.3.1.jar

bcprov-jdk15on-1.51.jar jackson-annotations-2.4.4.jar lyra-0.4.2.jar

bytebuffer-collections-0.1.5.jar jackson-core-2.4.4.jar lyra-0.5.0.jar

classmate-1.0.0.jar jackson-core-asl-1.9.12.jar lyra-0.5.2.jar

commons-cli-1.2.jar jackson-databind-2.4.4.jar lz4-1.3.0.jar

commons-codec-1.7.jar jackson-dataformat-smile-2.4.4.jar mail-1.4.7.jar

commons-dbcp2-2.0.1.jar jackson-datatype-guava-2.4.4.jar mapdb-1.0.7.jar

commons-io-2.0.1.jar jackson-datatype-joda-2.4.4.jar maven-aether-provider-3.1.1.jar

commons-lang-2.6.jar jackson-jaxrs-base-2.4.4.jar maven-model-3.1.1.jar

commons-logging-1.1.1.jar jackson-jaxrs-json-provider-2.4.4.jar maven-model-builder-3.1.1.jar

commons-pool-1.6.jar jackson-jaxrs-smile-provider-2.4.4.jar maven-repository-metadata-3.1.1.jar

commons-pool2-2.2.jar jackson-mapper-asl-1.9.13.jar maven-settings-3.1.1.jar

compress-lzf-1.0.3.jar jackson-module-jaxb-annotations-2.4.4.jar maven-settings-builder-3.1.1.jar

config-magic-0.9.jar java-util-0.27.0.jar maxminddb-0.2.0.jar

curator-client-2.7.0.jar javax.el-3.0.0.jar mx4j-3.0.2.jar

curator-framework-2.7.0.jar javax.el-api-3.0.0.jar netty-3.9.5.Final.jar

curator-recipes-2.7.0.jar javax.inject-1.jar okhttp-1.0.2.jar

curator-x-discovery-2.7.0.jar java-xmlbuilder-0.4.jar opencsv-2.3.jar

derby-10.11.1.1.jar javax.servlet-api-3.1.0.jar org.abego.treelayout.core-1.0.1.jar

derbyclient-10.11.1.1.jar jboss-logging-3.1.3.GA.jar plexus-interpolation-1.19.jar

derbynet-10.11.1.1.jar jcl-over-slf4j-1.7.10.jar plexus-utils-3.0.15.jar

disruptor-3.3.0.jar jdbi-2.32.jar protobuf-java-2.5.0.jar

druid-api-0.3.8.jar jersey-core-1.17.1.jar rhino-1.7R5.jar

druid-aws-common-0.7.3.jar jersey-guice-1.17.1.jar RoaringBitmap-0.4.5.jar

druid-common-0.7.3.jar jersey-server-1.17.1.jar server-metrics-0.1.1.jar

druid-indexing-hadoop-0.7.3.jar jersey-servlet-1.17.1.jar slf4j-api-1.6.4.jar

druid-indexing-service-0.7.3.jar jets3t-0.9.3.jar spymemcached-2.11.4.jar

druid-processing-0.7.3.jar jetty-client-9.2.5.v20141112.jar tesla-aether-0.0.5.jar

druid-rabbitmq-0.7.3.jar jetty-continuation-9.2.5.v20141112.jar validation-api-1.1.0.Final.jar

druid-server-0.7.3.jar jetty-http-9.2.5.v20141112.jar wagon-provider-api-2.4.jar

druid-services-0.7.3.jar jetty-io-9.2.5.v20141112.jar xpp3-1.1.4c.jar

emitter-0.3.1.jar jetty-proxy-9.2.5.v20141112.jar zookeeper-3.4.6.jar

I m using the following command to run the realtime node :

java -Xmx256m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Ddruid.realtime.specFile=config/realtime/realtime.spec -classpath config/_common:lib/*:config/realtime io.druid.cli.Main server realtime

Thanks,

Shantanu

Hi, can you post how you are running pull-deps and the output of running the job?

Hey,

I cleared the m2 repository and replaced the lib folder with a fresh copy (downloaded a new copy of druid ) and then ran pull-deps … It worked . Thanks everyone for helping . Sorry for any inconveniences .

Thanks ,

Shantanu

Cool! Great to see the problem was finally resolved Shantanu.

hii SHANTANU SRIVASTAVA,
did you streaming success with rabbitmq to druid?

I have problem when ingestion data

curl -X ‘POST’ -H ‘Content-Type:application/json’ -d @Druid_Ingestion/rabbitmqToDruid-index.json http://localhost:8081/druid/indexer/v1/task

{“error”:“Unexpected token (START_OBJECT), expected VALUE_STRING: need JSON String that contains type id (for subtype of io.druid.indexing.common.task.Task)\n at [Source: HttpInputOverHTTP@7dfbfeea[c=1670,q=1,[0]=EOF,s=STREAM]; line: 1, column: 4]”}

ingestion file is:

[

{

“dataSchema” : {

“dataSource” : “rabbitmq_test”,

“parser” : {

“type” : “string”,

“parseSpec” : {

“format” : “json”,

“timestampSpec” : {

“column” : “submit_date”,

“format” : “auto”

},

“dimensionsSpec” : {

“dimensions”: [ “request_type”,“origin”,“description” ],

“dimensionExclusions” : ,

“spatialDimensions” :

}

}

},

“metricsSpec” : [{

“type” : “count”,

“name” : “count”

} ],

“granularitySpec” : {

“type” : “uniform”,

“segmentGranularity” : “HOUR”,

“queryGranularity” : “NONE”

}

},

“ioConfig” : {

“type” : “realtime”,

“firehose”: {

“type”: “rabbitmq”,

“connection” : {

“host”: “172.29.242.173”,

“port”: “5672”,

“username”: “guest”,

“password”: “guest”,

“virtualHost”: “”,

“uri”: “amqp://172.29.242.173:1234/vhost”

},

“config” : {

“exchange”: “NiFi_MySQL_to_Druid”,

“queue” : “NiFi_Druid_Test_1”,

“routingKey”: “NiFi_MySQL_to_Druid_Queue_1”,

“durable”: “true”,

“exclusive”: “false”,

“autoDelete”: “false”,

“maxRetries”: “10”,

“retryIntervalSeconds”: “1”,

“maxDurationSeconds”: “300”

}

}

},

“tuningConfig”: {

“type” : “realtime”,

“maxRowsInMemory”: 7500000,

“intermediatePersistPeriod”: “PT10m”,

“windowPeriod”: “PT10m”,

“basePersistDirectory”: “/tmp/realtime/basePersist”

}

}

]