Combining firehose with kafka

Hi,

Is there a way to achieve combining firehose kind of functionality with kafka based ingestion?

thanks,

Roopini

Hi Roopini,

Could you please explain more on your use case to understand your issue better ?

Thanks and Regards,

Vaibhav

Hi Vaibhav,

Our usecase is to update the existing data in datasource through kafka ingestion. We tried giving appendToExisting flag in IOConfig of kafka input spec, but it is of no use, since it can be used only with ingestion task of type index. And we tried so many other features of druid to achieve this, combining firehose is one of those. So our main usecase is to update the existing data in datasource through kafka ingestion in realtime. Is there a way to achieve this.

Thanks,

Roopini.

Hi Roopini,

Thank you for clarifying.

Kafka indexing service reads the data from the Kafka topic, Once the data is ingested into the druid, you should use one of the below methods to update the existing data (Historical data)
https://druid.apache.org/docs/latest/ingestion/data-management.html#update

If you want to restructure your Kafka ingestion, you can submit the updated supervisor spec and realtime data will be indexed as per new spec. I don’t think you can update the exiting druid segments using Kafka indexing.

Thanks and Regards,

Vaibhav