Hi - sorry if this is a duplicate post, I posted this last Friday but it didn’t seem to go through email…
We’re trying to get a better understanding of using Druid Realtime nodes with the Kafka firehose. There are several mailing list threads discussing potential issues with this setup and we just want to double-check some things.
So it sounds like if you have 1 Realtime node consuming 1 Kafka topic (i.e. no replication and no sharding of Realtime nodes) then everything is OK. Also, if you have 2 Realtime nodes in the same group.id but with different shard partitionNums then this is also OK.
Now let’s say there are 4 Realtime nodes with 2 per group.id (for replication) and using 2 shard partitions. Here’s a diagram: http://imgur.com/IesG0Wt.
 &  say that Kafka partition assignment may assign different topic partitions to the same Druid shard partition in the different consumer groups. It seems like a workaround to this problem would be to set the consumer.id in each Realtime to something containing the Druid shard partition number. The Kafka docs for consumer property partition.reassignment.strategy default “range”  seem to imply that as long as a) the group.ids have the same number of consumers and b) consumer.ids in the separate group.ids lexicographically sort the same way then the topic partition assignment should be the same across group.ids. The diagram  shows an example of this.
Does this seem accurate? Aside from the operational pain of maintaining all of these group.id & consumer.id settings, this seems like a way to do Realtime sharding & replication with the Kafka firehose such that shards with same partitionNum contain data from the same Kafka topic partitions.
Or are we missing some details?