Multiple supervisors ingesting to same data source using Kafka indexing service

Kafka indexing service’s supervisor can listen only to single topic. But we have two Kafka topics, but both should write to same data source. How to achieve this? We are unable to start two supervisors with same dataSource. What is the feasible work around for this case?

Is there any other optimal solution than merging multiple topics messages to a single topic and using that single topic in kafka indexing service

I’m correcting my previous response. Please don’t use Tranquility.
It’s not possible to write two different Kafka topics to the same data source.
The workaround would be as you suggested, using Kafka stream and KSQL to write the original topics into a third topic and ingest from there

Ooh another idea for you Amba… maybe you could use a UNION data source at query time - just check the special notes about it here: https://druid.apache.org/docs/latest/querying/datasource.html#union

There’s also a cool thing I saw Neilson publish about where they used the Hadoop Indexer to re-index Druid data - not sure if it’s an option to use two Druid datasources in one ingestion task…?? - see the bit on “Dimension Based TTL”
https://medium.com/nmc-techblog/data-retention-and-deletion-in-apache-druid-74ffd12398a8

https://druid.apache.org/docs/latest/ingestion/hadoop.html

Hi Amba - did you find a solution?