Using tranquility with Kafka

Hi,

We are currently using tranquililty kafka in our application but we realized that our configuration is not static (i.e we add dimensions and metrics fairly frequently) and this causes a pain point due to having to restart tranquility each time with the new configuration file.

Druid Beams seem to also add too much complexity for a small issue. Is there any other ways to for example hot reload the config file or update it dynamically?

Thanks!

Hi Jamil,

current implementation doesn`t allow dynamic reload. I resolved this issue by custimizing Tranquility Kafka for our needs, thanks it is not very complex, just several classes in java.

Regards

@Sergii, It would be great If you could create a PR for your changes to tranquility-kafka, will really be helpful for the community.
@Jamil, have you also looked at the new kafka-indexing-service ?

FWIW, It allows you to update your config file via HTTP endpoints.

http://druid.io/docs/latest/development/extensions-core/kafka-ingestion.html

New kafka indexing service looks great but is anyone using them in production?More specifically is any one using kafka service for batch ingestion?

Regards,

Pushkar

It is still experimental feature, we are targeting druid 0.9.3 to declare it stable.
FWIW, the main missing feature automatic compaction of segments generated by kafka indexing service.