kafka indexing service VS tranquility

Hi all~

Today, I tested kafka indexing service(http://druid.io/docs/latest/development/extensions-core/kafka-ingestion.html) for real time indexing.

I have a question.

What is difference between kafka indexing service and tranquility for kafka ingestion?

I’d like to reliable real time ingestion using kafka. Which one is best choice?

Thanks,

JerryJung

Hi Jerry,

I’m not an expert but the main difference is that Kafka Indexing Service assures an exactly-once ingestion, and that data from any date can be ingested (unlike tranquility, which only accepts data inside the windowPeriod).

Besides, I’ve had some problems with Tranquility from time to time, and I’m feeling more comfortable with Kafka Indexing service.

Thanks your comment. :slight_smile:
I’ll look at some more about kafka indexing service.

Long term we plan to focus on using Kafka indexing service for ingestion from Kafka->Druid. The biggest difference is that the kafka indexing service does not have a notion of a windowPeriod and will ingest even historical events with exactly once guarantees. The feature is new but several orgs have already deployed it into production. There’s a few more features we want to get in to make it easier to use and we also want people to run it in production for some time to catch any corner case problem we may have missed.

Tranquility is the old tried and true way of ingestion data. It has a concept of windowPeriod and will drop events outside the windowPeriod. At some point we also want to make Tranquility exactly once as well.

That’s good point!
Thank you :slight_smile: