Group id using kafka indexing

TROUBLESHOOTING Google Groups
#1

Hi folks,
I’m doing some POC where I am using Kafka connector/indexer service to ingest the data into Druid.

We are going to have multiple topics and consumers will be Druid (using indexer service). Ingestion works

ok so far. But I am wondering what is the way to create SINGLE topic and let multiple consumers

can consume using group_id. Idea is to have different rollups (materialized views) using different consumer

on same kafka topic.

In the specfile I specify something like following, but it’s not really creating consumer group when

I verify the same on Kafka side. Please help.

Here is the excerpt for the specfile on the Druid side.

Basically this is part of ioConfig > consumerProperties.

“ioConfig:” {

…

“consumerProperties”: {

“group.id” : “group_id”

}

Thanks.

#2

The Kafka Ingestion Service depends on the # of partitions you have per Kafka topic. This is what enables the parallelism for indexing, You use the “group_id” as your partition key for that one topic and Druid automatically creates task based on the # of partitions you have.

Rommel Garcia