Moving the Kafka Indexing Supervisor from one Kafka cluster to another

Hi all,

We are in the situation where we are switching from one Kafka cluster to another, as opposed to upgrading in place. Druid has drained the topic on the old cluster and I want to repoint essentially the same supervisor task to the new cluster. I am concerned about how Druid will deal with the offsets. The new cluster’s topic has the same name as the previous one, and looking in the druid_dataSource table in MySQL I see:

{

“type”:“kafka”,

“partitions”:{

“topic”:“raw_shopkick_pylons_weblog_avro_v1”,

“partitionOffsetMap”:{

“0”:20223768,

“1”:20210577,

“2”:20032861,

“3”:20224767,

“4”:20091018,

“5”:20393470,

“6”:20145680,

“7”:20160493

}

}

}

I do not see any concept of cluster id in there, so I am guessing that it will try to use the same offsets when talking to the new cluster.

Is that correct? If so, can I just delete this row from the database?

Thanks,

This procedure should work:

  1. stop supervision (shutdown the supervisor)

  2. delete that row

  3. re-start supervision, paying attention to the “useEarliestOffset” setting and making sure it’s what you want

Thanks Gian. This was done and appears to have worked correctly.

Next question then, can I re-point Druid to a new zookeeper cluster without having to make any other changes?

Thanks,

–Ben

You should be able to bring down your cluster, update the common.runtime.properties to update the new ZK ips, and then start up the cluster again.

Thanks Fangjin, this was done and worked smoothly.