Middle manager not able to create new Peons

Hi,

For our druid cluster the middle manager is not able to handle more than 3 tasks. If any more task is supplied to the middle manager, it fails with following exception -

java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method) ~[?:1.7.0_79]
at java.lang.Thread.start(Thread.java:714) ~[?:1.7.0_79]
at org.eclipse.jetty.util.thread.QueuedThreadPool.startThreads(QueuedThreadPool.java:430) ~[jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.util.thread.QueuedThreadPool.doStart(QueuedThreadPool.java:104) ~[jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.server.Server.start(Server.java:387) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.server.Server.doStart(Server.java:354) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]
at io.druid.server.initialization.jetty.JettyServerModule$1.start(JettyServerModule.java:204) [druid-server-0.9.2.jar:0.9.2]
at com.metamx.common.lifecycle.Lifecycle.start(Lifecycle.java:259) [java-util-0.27.10.jar:?]
at io.druid.guice.LifecycleModule$2.start(LifecycleModule.java:155) [druid-api-0.9.2.jar:0.9.2]
at io.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:101) [druid-services-0.9.2.jar:0.9.2]
at io.druid.cli.CliPeon.run(CliPeon.java:274) [druid-services-0.9.2.jar:0.9.2]
at io.druid.cli.Main.main(Main.java:106) [druid-services-0.9.2.jar:0.9.2]

``

Our middle manager’s runtime.properties is -

druid.service=druid/middleManager
druid.host=
druid.port=

Number of tasks per middleManager

druid.worker.capacity=4

Task launch parameters

druid.indexer.runner.javaOpts=-server -Xms2g -Xmx9g -Duser.timezone=UTC -Dfile.encoding=UTF-8 -XX:MaxPermSize=2g

Task Logging

Processing threads and buffers

druid.processing.buffer.sizeBytes=536870912
druid.processing.numThreads=2

Hadoop indexing

druid.indexer.task.hadoopWorkingPath=hdfs://
druid.indexer.task.defaultHadoopCoordinates=[“org.apache.hadoop:hadoop-client:2.6.0”]
druid.indexer.fork.property.druid.indexer.task.hadoopWorkingPath=hdfs://

``

and the jvm.config looks like -

-server
-Xmx3g
-Xms1g
-XX:+PrintGCDetails
-Duser.timezone=UTC
-Dfile.encoding=UTF-8
-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
-Djava.io.tmpdir=/mnt/tmp
-Dlog4j.configurationFile=file:/opt/druid/druid/conf/druid/middleManager/log4j2.xml

``

Can someone please help us for this?

Thanks!

Try checking what your process limit is with “ulimit -u” and increasing it using "ulimit -u ".

Thanks Gian for getting back. So we have the following ulimit -

soft nofile 32768

hard nofile 32768

How much ulimit should we ideally have?

Those are open files limits; threads/processes are a different kind of limit.

Oh sorry. Yeah we have that set to 1024.

Thanks!