Unable to load messages to Druid from Kafka - error: "Broker may not be available"

Hello:

I’m new to Druid here. I have Druid installed on Window Hyper-V Machine(Linux) and have my kafka installed on my windows. I have some click stream data sending to Kafka on windows. I’m also able to launch Druid dashboard on windows by calling the Hyper-V IP address like below

When I Load Data->Fill out Bootstrap server and topic and click Apply, I keep getting error like this. (By the way, I installed Druid by docker and I included the druid-kafka-indexing-service extension in the environment file already. Not sure why Druid can’t load data from Kafka. Did I miss anything here?

Thank you very much!

Hi Julie
Looks like you’re trying to connect to Kafka broker using an internal port as it refers to the localhost address. This will work only if Kafka and Druid are within the same network i.e in your case the same docker network as Druid.
In case they are on different networks you need to configure and use a kafka listener which listens on all interfaces (0.0.0.0). Also if its outside the same network, you need to refer to the Kafka broker using IP address rather than using localhost or the Kafka docker service name.

See this article which explains on this in detail

https://rmoff.net/2018/08/02/kafka-listeners-explained/

Regards
Visakh

See this as an example

Thank you very much Visakh! I have modified the KAFKA_ADVERTISED_LISTENERS in docker-compose file to include my IP instead of localhost and it’s working! My druid on VM Linux can talk to my kafka on Windows.

Regards,

Julie

Awesome Julie
Glad that you got it sorted!