Druid-0.12.3: Detection of more than one Coordinator or Overlord leader

Hi,

We have:

  • 1 Zookeeper cluster with 5 hosts running in Docker.

  • 1 Druid cluster installed on 3 hosts running as Binary:

Host 1: Druid Broker, Historical, Overlord, Coodinator and MiddleManager

Host 2: Druid Broker, Historical, Overlord, Coodinator and MiddleManager

Host 3: Druid Broker, Historical, Overlord, Coodinator and MiddleManager

Zookeeper version: zookeeper:3.4.13 (Docker)

Druid version: druid-0.12.3 (Binary)

From time to time our Zookeeper have some kind of split-brain which causes new Druid election, but it happens to have more than 1 Druid leader. It can happen for Coordinator but also with Overlord.

The new Druid leaders do not detect that two leader exists. To fix the issue I need to manually restart Coordinator or Overlord to force new leader election.

I would like to understand if anyone had same issue, and if someone can explain How Druid leader election works.

Thanks.

BR.

Hey Catarina,

Maybe try double checking that the ZK cluster is totally set up right. Each server should have a ‘myid’ file with its own unique ID, and a zoo.cfg that lists out all 5 servers in the quorum.

Hi Catarina,

if someone can explain How Druid leader election works.

You can find some notes on leader election here http://druid.io/docs/latest/dependencies/zookeeper.html

Thanks,

Sashi