Hi, I’m getting started with Druid and so far it is ticking along nicely, but I’ve become confused when walking through the steps for kafka ingestion.
I’ve read both the kafka ingestion docs, and the kafka tutorial, and they seem so describe different methods of loading data into druid. Are these two approaches meant to be used together? Is one of them deprecated in favor of the other? Please help. I have followed all the steps in the ingestion docs, (using the indexing service, not tranquility) and data has not shown up.
Yes, there are a few ways to load from Kafka into Druid. Tranquility is a bit older and more battle tested, but for a new project I’d suggest the Kafka indexing service. It was meant to solve some of the issues with Tranquility, by being exactly once and by running as a supervised job within Druid. (Tranquility is best-effort and runs as a separate process, incurring operational and network overhead.) Both are supported at this time.
Maybe someone can help you if you go into some more detail about what you tried, and when it became clear that it wasn’t working.