realtime ingestion same kafka feed for different datasources(schemas as well)


Can i send data via kafka with same topic(feed) to differen schemas ?

(Druid 0.9.0 kafka-eight)

Best regards


Yes, spin up a second realtime node with a different schema pointing to the same Kafka feed. The important thing is that you choose different values for [], otherwise your nodes will not each receive all the events.

Ozden - To make it much clear, are you saying your Kafka Topic get messages of different type and they should routed to different datasources or your Kafka Topic gets same messages and those messages should be converted into different type of indexes?

What Lim suggested above works for the later.

But if you are looking for a solution for the former one, I believe you would have to use Tranquility API and have a consumer decide to which sender to send the messages to.

David - is my assumption correct?


Yes that’s correct - if you need to route messages differently depending on their content, you’d be looking at a Tranquility-based solution, or for larger workloads a stream processing stage (think Storm, Samza, Spark Streaming).