Druid broker and historical node NOT coming up

Hi All,
Trying to install the druid using helm chart. All the pods are coming up properly but broker and historical node NOT coming up i.e CrashLoopBackOff.

Any suggestion here which missing. The logs output below.

$ kubectl get pods

NAME READY STATUS RESTARTS AGE

druid-broker-74769d575f-wz4rw 0/1 CrashLoopBackOff 6 11m

druid-coordinator-5c49599bdb-4rbvv 1/1 Running 1 170m

druid-historical-0 0/1 CrashLoopBackOff 6 12m

druid-middle-manager-0 1/1 Running 2 3h

druid-mysql-b8f6777f9-jzqsr 1/1 Running 0 170m

druid-overlord-577d8d89fb-m4z74 1/1 Running 1 3h

druid-zookeeper-0 1/1 Running 0 3h

druid-zookeeper-1 1/1 Running 0 170m

druid-zookeeper-2 1/1 Running 0 178m

$ kubectl logs druid-broker-74769d575f-wz4rw

2019-06-16T10:30:02,427 INFO [main] io.druid.guice.PropertiesModule - Loading properties from common.runtime.properties

2019-06-16T10:30:02,431 INFO [main] io.druid.guice.PropertiesModule - Loading properties from runtime.properties

Jun 16, 2019 10:30:02 AM org.hibernate.validator.internal.util.Version

INFO: HV000001: Hibernate Validator 5.1.3.Final

2019-06-16T10:30:03,028 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, directory=’/opt/druid/extensions’, h

.

.

.

2019-06-16T10:30:05,379 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.metadata.SegmentMetadataQueryConfig] from props[druid.query.segmentMetadata.] as [io.druid.query.metadata.SegmentMetadataQueryConfig@476fe690]

2019-06-16T10:30:05,392 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.groupby.GroupByQueryConfig] from props[druid.query.groupBy.] as [GroupByQueryConfig{defaultStrategy=‘v2’, singleThreaded=false, maxIntermediateRows=50000, maxResults=500000, bufferGrouperMaxSize=2147483647, bufferGrouperMaxLoadFactor=0.0, bufferGrouperInitialBuckets=0, maxMergingDictionarySize=100000000, maxOnDiskStorage=0, forcePushDownLimit=false, forceHashAggregation=false, intermediateCombineDegree=8, numParallelCombineThreads=1}]

2019-06-16T10:30:05,421 INFO [main] io.druid.offheap.OffheapBufferGenerator - Allocating new intermediate processing buffer[0] of size[536,870,912]

2019-06-16T10:30:05,675 INFO [main] io.druid.offheap.OffheapBufferGenerator - Allocating new intermediate processing buffer[1] of size[536,870,912]

2019-06-16T10:30:05,941 INFO [main] io.druid.offheap.OffheapBufferGenerator - Allocating new intermediate processing buffer[2] of size[536,870,912]

2019-06-16T10:30:06,208 INFO [main] io.druid.offheap.OffheapBufferGenerator - Allocating new intermediate processing buffer[3] of size[536,870,912]

2019-06-16T10:30:06,468 INFO [main] io.druid.offheap.OffheapBufferGenerator - Allocating new intermediate processing buffer[4] of size[536,870,912]

2019-06-16T10:30:06,867 INFO [main] io.druid.offheap.OffheapBufferGenerator - Allocating new intermediate processing buffer[5] of size[536,870,912]

/start-druid.sh: line 44: 17 Killed java ${JAVA_OPTS} -cp /opt/druid/conf/druid/_common:/opt/druid/conf/druid/1:/opt/druid/lib/* io.druid.cli.Main server @

$ kubectl logs druid-historical-0

2019-06-16T10:34:42,812 INFO [main] io.druid.guice.PropertiesModule - Loading properties from common.runtime.properties

2019-06-16T10:34:42,817 INFO [main] io.druid.guice.PropertiesModule - Loading properties from runtime.properties

Jun 16, 2019 10:34:42 AM org.hibernate.validator.internal.util.Version

INFO: HV000001: Hibernate Validator 5.1.3.Final

2019-06-16T10:34:43,336 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, directory=’/opt/druid/extensions’, hadoopDependenciesDir=‘hadoop-dependencies’, hadoopContainerDruidClasspath=‘null’, addExtensionsToHadoopContainer=false, loadList=[druid-kafka-eight, druid-s3-extensions, druid-histogram, druid-datasketches, druid-lookups-cached-global, mysql-metadata-storage, druid-kafka-indexing-service, druid-avro-extensions]}]

2019-06-16T10:34:43,409 INFO [main] io.druid.initialization.Initialization - Loading extension [druid-kafka-eight] for class [interface io.druid.cli.CliCommandCreator]

.

.

.

2019-06-16T10:34:45,797 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.metadata.SegmentMetadataQueryConfig] from props[druid.query.segmentMetadata.] as [io.druid.query.metadata.SegmentMetadataQueryConfig@57562473]

2019-06-16T10:34:45,809 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.groupby.GroupByQueryConfig] from props[druid.query.groupBy.] as [GroupByQueryConfig{defaultStrategy=‘v2’, singleThreaded=false, maxIntermediateRows=50000, maxResults=500000, bufferGrouperMaxSize=2147483647, bufferGrouperMaxLoadFactor=0.0, bufferGrouperInitialBuckets=0, maxMergingDictionarySize=100000000, maxOnDiskStorage=0, forcePushDownLimit=false, forceHashAggregation=false, intermediateCombineDegree=8, numParallelCombineThreads=1}]

2019-06-16T10:34:45,843 INFO [main] io.druid.offheap.OffheapBufferGenerator - Allocating new intermediate processing buffer[0] of size[536,870,912]

2019-06-16T10:34:46,111 INFO [main] io.druid.offheap.OffheapBufferGenerator - Allocating new intermediate processing buffer[1] of size[536,870,912]

2019-06-16T10:34:46,374 INFO [main] io.druid.offheap.OffheapBufferGenerator - Allocating new intermediate processing buffer[2] of size[536,870,912]

2019-06-16T10:34:46,641 INFO [main] io.druid.offheap.OffheapBufferGenerator - Allocating new intermediate processing buffer[3] of size[536,870,912]

2019-06-16T10:34:46,937 INFO [main] io.druid.offheap.OffheapBufferGenerator - Allocating new intermediate processing buffer[4] of size[536,870,912]

2019-06-16T10:34:47,201 INFO [main] io.druid.offheap.OffheapBufferGenerator - Allocating new intermediate processing buffer[5] of size[536,870,912]

/start-druid.sh: line 44: 18 Killed java ${JAVA_OPTS} -cp /opt/druid/conf/druid/_common:/opt/druid/conf/druid/1:/opt/druid/lib/* io.druid.cli.Main server @

Looking at the logs I'm guessing it's not able to allocate enough memory to the merge buffers can you check

1) enough memory is allocated to XX:MaxDirectMemorySize in the jvm config of broker and historical

2) the num merge buffers is appropriately configured based on the resources available

Please refer to the historical process configuration in the below document

https://druid.apache.org/docs/latest/configuration/index.html#historical-process-config

Thanks . The issue was with the insufficient memory on the nodes.
(1) Could please let me know after successful installation of druid (all the pods are ready state) how I can make use of druid?

Please tell me the steps to use it.

(2) I want to make use of the kafka on my setup. How I can use the helm chart of kafka and install on the same cluster?

I don’t have to install the zookeeper on the cluster as the zookeeper is running part of the druid installation.