How to run Druid when ZooKeeper and Kafka are running by other project on my machine

Hello:

I’m new to Druid. I’m currently have a click stream project (running by docker). It already has kafka and zookeeper running. When I trying to run Druid (which I followed the quickstart demo from druid document), it seems like to run Druid to get Kafka data, it needs to fire another instance of ZooKeeper and Kafka. It ended up with error (see attachement).

I wonder whether I’m on the right path. What’s the best practice to connect Druid to Kafka as consumer on my local machine? I definitely need to run the click stream project to generate the stream data, so I need ZooKeeper and Kafka running from there too.

Thanks!

Julie

What do the logs that the error is referring to say?

Hi, Rachel:

When I run

./bin/start-micro-quickstart

For ZK, the logs said

2020-02-04T14:41:56,972 ERROR [main] org.apache.zookeeper.server.ZooKeeperServerMain - Unexpected exception, exiting abnormally

java.net.BindException: Address already in use

and for broker, coordinator-overlord, historical router and middleManager, the same error said

java.lang.reflect.InaccessibleObjectException: Unable to make public long

I think it’s because my click stream project is running ZK, Kafka Broker…etc already. If I terminate the click stream project, Druid can be run successfully. So I don’t know what is the best practice to run them both when they all required ZK…

Thanks

You can configure druid to use your existing ZK and Kafka. You don’t have to use the one it bundles. For Kafka, just shut down the instance and connect to your existing instance instead.

https://druid.apache.org/docs/latest/tutorials/cluster.html#configure-zookeeper-connection