Kafka Indexing Service - kafka client version needs upgrade to 0.10.0.1+

Kafka broker 0.8.2.2 to 0.10.0.0 have a critical memory-leak bug https://issues.apache.org/jira/browse/KAFKA-3933

The bug is fixed in 0.10.0.1, but current druid 0.9.2 release candidate is not compatible to Kafka 0.10.0.1. We should upgrade to Kafka 0.10.1.0 or 0.10.0.1

Thanks

Ashish

Hey Ashish,

Thanks for letting us know. There’s already an open issue for upgrading here: https://github.com/druid-io/druid/issues/3426

In the meantime, Kafka compatibility supports later versions of brokers working with earlier versions of consumers (but not the other way around). In other words, you should be able to use the Kafka indexing service (which uses an 0.9 producer) with 0.10.x brokers which will include the patch for the memory leak.

(‘The other way around’, i.e. newer consumers not working with older brokers, is the reason why we tend to be slow in upgrading the Kafka consumer, as it would force everyone to upgrade their brokers which may be non-trivial for some users/organizations).

I can confirm that this works with Druid 0.9.1.1. We upgraded our Kafka cluster to 0.10.1 last week and we had no issue with Druid’s Kafka Indexing Service talking to it. I haven’t tried Druid 0.9.2 but don’t see why it would be any different.

–Ben

Of course it “will work”, however: http://kafka.apache.org/documentation#upgrade_10_performance_impact - and that’s the primary reason I am interesting in a fast transition. We would love to use compression due to bandwidth issues but can’t because of cpu impact.