[druid-user] druid:master multi-node installation, is it done correctly

i installed 2 master nodes with this cli:

bin/run-druid coordinator-overlord conf/druid/cluster/master

in my first attempt, after master2 is up, master1 was stopped.

in my second attempt, both master1 and master2 were up and listening on port 8081.

in zk:

under /druid/discovery/druid:overlord, /druid/discovery/druid:coordinator I only found 1 node. I assume its expected and that node is leader node of each category such as coordinator and overlord?

under /druid/overlord/_OVERLORD /druid/coordinator/_COORDINATOR I found both master1 and master2

Are you running ZK on both master nodes?

The zk is an external independent installation

I installed 2 masters, 3 historical, 3 middle managers, 2 brokers, 2 routers.
after all the services are up for a while, some of the nodes automatically kicked off shutdown hook and the services went offline.
no exceptions, no errors found in log.
what happened?

It might be a ZK quorum issue. If you think that might be the case, you can enable ZK logging (the article was written by Imply, my employer).

You would want 3 masters if you want high availability. To be elected a leader, a quorum (more than 1/2) is needed. A quorum of 2 is 2.
So if either master goes down, you can’t get a quorum. With 3 masters, 1 can go down and the other 2 can make a quorum.