How nodes are setup in cluster

Just a basic deployment question
How and where do you config for the nodes to communicate with each other
for example how does coordinator , overlord knows where is historical, MM, query nodes are?

I have one Historical and one MM and I am planning to add more. I can spin up EC2 and set it up but how do I add these to Coordinator/overlord so that it is picked by master node

Hi @ranjan. There are example master server configurations located under conf/druid/cluster/master. The clustered deployment doc walks through a setup, and the master server doc has the configuration options.

Thanks @Mark_Herrera for your response. I am already running 1 master, 1 query, 1 MM and 1 Historical. I did by following the doc’s link you pasted in your response. However, I am just curious to know how and where the setup that tells master that here is the query node, here is the MM and here is the history?

For my own clarification, are you asking for the location of the runtime.properties files for different processes, including MM and Historical?

I know where the runtime property files are, but when I look inside I dont see any entry for query, mm, history nodes in master common property file or may be I am not looking at right property file

Thank you for the clarification. I’ve just had a long talk with a colleague, and I’m hopeful that this will give you what you’re asking for:

Screen Shot 2022-08-16 at 10.15.18 AM

The course is free, but, if you don’t want to register for it, here is the relevant language:

Want to know how the discovery process works?

On each server, in the common configuration file, we set druid.zk.service.host to tell the server how to contact ZooKeeper

In this same common configuration file we comment out druid.host, which forces the server to automatically determine its host name

Each server contacts ZooKeeper and registers who and where they are

Finally, ZooKeeper tells each of the servers how to contact other processes within the cluster

Please let me know if this helps.

2 Likes

Yep! @ranjan Druid is shared nothing via (by default) Zookeeper – you don’t explicitly tell the processes where each one is. That means you can add / remove them without telling all the others that you’ve added / removed them… amongst other things! The key dependencies you do need to explicitly tell the processes about are Deep Storage, the Metadata Database, and Zookeeper. I think that’s in the course that @Mark_Herrera has pointed you to :slight_smile:

1 Like

Thanks @Mark_Herrera and @petermarshallio , your inputs provided the info that I was looking for and I was able to figure out
Basically in each nodes /conf/cluster/_common/common.runtime.properties it’s defined

1 Like