Ingesting data from multiple kafka topics into single datasource in Druid

TROUBLESHOOTING Google Groups
#1

Hi,

Is it possible to ingesting data from ‘multiple kafka topics’ into datasource in Druid using single supervisor task? If not, is there is any other way to achieve this i.e. have separate supervisor tasks for each kafka topic, and then pushing data to a single data source?

Regards,

Vinay Patil

#2

The Kafka Indexing service does not support topic patterns (like metrics*) in the KafkaSupervisorIOConfig. So a single supervisor task cannot be used for this.

#3

In Addition to Muthu’s comment, There are probably two alternative approaches:

  1. merge the streams into a single stream in kafka and then load the merged stream into druid

  2. load the different streams into different datasources and use the Druid ‘union datasource’ feature to query them together at query time

Note, it makes the druid setup a bit more complicated and also ‘union datasource’ is not supported everywhere: for example it isn’t supported through Druid SQL

#4

Thank you for the input Muthu and Ming.

Regards,

Vinay Patil