could i set up multiple real time tasks by tranquility in different machines consume one kafka topic


I have a question about tranquility. My data is in one topic of kafka. And the topic has 10 partitions. And I want to use tranquility to consume data from the kafka topic and feed to druid.

the question is could I set up 3 nodes which use tranquility to kick off a real time task and feed data to the task?does it have a data overwritten issue?

my concern is 3 tranquility nodes will consume different partitions of the topic data of kafka and sent the data to their own task respectively. Then in the Druid side, there are two segments of the same time interval, but the segments does not have the same data, and each segment in one real time node, just have part of the data of the time interval.

that’s my question. hope for you response. thanks.

Tranquility can handle this situation. Different tranquility instances with the same configuration will coordinate with each other through ZooKeeper, and make sure they write to the same set of tasks. You can start up as many tranquility instances as you want, and you can scale up or down at any time. If you want to scale up or down the number of Druid tasks, change the “partitions” number you configure tranquility with when you’re using the DruidBeams builder. Otherwise, all of your tranquility instances will write to the same Druid task.

thanks, I’ll try.

在 2015年8月27日星期四 UTC+8下午11:32:29,Gian Merlino写道: