druid cluster

Hi,

I am trying to set up druid cluster in three nodes.

In 1- node -> coordinator, overlord, zookeeper, kafka and hadoop is running.

In 2-node -> broker is running

In 3-node -> historical and middleManager is running.

I have done the configuration of deep storage(hadoop) from this site ‘http://druid.io/docs/latest/tutorials/cluster.html

I am trying kafka-indexing-service for sending data to druid but I am unable to see the records. I have added kafka-indexing-service plugin in common.runtime.properties also.

And also unable to see any data in ‘/druid/segments’.

druid.storage.type=hdfs

druid.storage.storageDirectory=/druid/segments

Kindly let me know the issue and solution for the same.

Thanks,

Monica

Hi Monica,

Have you ensured that you copied over all the configs across all the three nodes?

Have you gone through the log files to check if something went wrong with respect to the ingestion from Kafka?

Note: not related, but try to install zookeeper in all the three nodes. 3 is good number for fault-tolerance and leader election.

Thanks & Rgds

Venkat

Yes, I have copied all the configs (core-site.xml, hdfs-site.xml, yarn-site.xml, mapred-site.xml) in all the three nodes.
And there is no error logs with respect to the ingestion from Kafka.

When I am starting the supervisor I am see the logs in overlord console but after shutdown the supervisor I am not able to see the logs for that process and status is showing FAILED.

Thanks,

Monica