Unable to create new native thread when submitting hadoop ingestion task

Hey,

I am witnessing the following error when submitting hadoop ingestion tasks:

java.lang.RuntimeException: java.lang.reflect.InvocationTargetException

    at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.1.jar:?]

    at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:204) ~[druid-indexing-service-0.9.2.jar:0.9.2]

    at io.druid.indexing.common.task.HadoopIndexTask.run(HadoopIndexTask.java:175) ~[druid-indexing-service-0.9.2.jar:0.9.2]

    at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) [druid-indexing-service-0.9.2.jar:0.9.2]

    at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) [druid-indexing-service-0.9.2.jar:0.9.2]

    at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_79]

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_79]

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_79]

    at java.lang.Thread.run(Thread.java:745) [?:1.7.0_79]

Caused by: java.lang.reflect.InvocationTargetException

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.7.0_79]

    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[?:1.7.0_79]

    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.7.0_79]

    at java.lang.reflect.Method.invoke(Method.java:606) ~[?:1.7.0_79]

    at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:201) ~[druid-indexing-service-0.9.2.jar:0.9.2]

    ... 7 more

Caused by: java.lang.ExceptionInInitializerError

    at io.druid.indexing.common.task.HadoopIndexTask$HadoopDetermineConfigInnerProcessing.runTask(HadoopIndexTask.java:277) ~[druid-indexing-service-0.9.2.jar:0.9.2]

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.7.0_79]

    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[?:1.7.0_79]

    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.7.0_79]

    at java.lang.reflect.Method.invoke(Method.java:606) ~[?:1.7.0_79]

    at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:201) ~[druid-indexing-service-0.9.2.jar:0.9.2]

    ... 7 more

Caused by: com.google.inject.CreationException: Unable to create injector, see the following errors:

  1. Error in custom provider, java.lang.OutOfMemoryError: unable to create new native thread

at io.druid.server.initialization.HttpEmitterModule.getEmitter(HttpEmitterModule.java:73) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> io.druid.server.initialization.EmitterModule -> io.druid.server.initialization.HttpEmitterModule)

at io.druid.server.initialization.HttpEmitterModule.getEmitter(HttpEmitterModule.java:73) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> io.druid.server.initialization.EmitterModule -> io.druid.server.initialization.HttpEmitterModule)

while locating com.metamx.emitter.core.Emitter annotated with @com.google.inject.name.Named(value=http)

at io.druid.server.initialization.EmitterModule$EmitterProvider.inject(EmitterModule.java:104)

at io.druid.server.initialization.EmitterModule.configure(EmitterModule.java:75) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> io.druid.server.initialization.EmitterModule)

1 error

    at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:470) ~[guice-4.1.0.jar:?]

    at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:176) ~[guice-4.1.0.jar:?]

    at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110) ~[guice-4.1.0.jar:?]

    at com.google.inject.Guice.createInjector(Guice.java:99) ~[guice-4.1.0.jar:?]

    at com.google.inject.Guice.createInjector(Guice.java:73) ~[guice-4.1.0.jar:?]

    at com.google.inject.Guice.createInjector(Guice.java:62) ~[guice-4.1.0.jar:?]

    at io.druid.initialization.Initialization.makeInjectorWithModules(Initialization.java:366) ~[druid-server-0.9.2.jar:0.9.2]

    at io.druid.indexer.HadoopDruidIndexerConfig.<clinit>(HadoopDruidIndexerConfig.java:99) ~[druid-indexing-hadoop-0.9.2.jar:0.9.2]

    at io.druid.indexing.common.task.HadoopIndexTask$HadoopDetermineConfigInnerProcessing.runTask(HadoopIndexTask.java:277) ~[druid-indexing-service-0.9.2.jar:0.9.2]

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.7.0_79]

    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[?:1.7.0_79]

Caused by: java.lang.OutOfMemoryError: unable to create new native thread

    at java.lang.Thread.start0(Native Method) ~[?:1.7.0_79]

    at java.lang.Thread.start(Thread.java:714) ~[?:1.7.0_79]

    at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949) ~[?:1.7.0_79]

    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1371) ~[?:1.7.0_79]

    at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38) ~[netty-3.10.4.Final.jar:?]

    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:368) ~[netty-3.10.4.Final.jar:?]

    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:100) ~[netty-3.10.4.Final.jar:?]

    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:52) ~[netty-3.10.4.Final.jar:?]

    at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45) ~[netty-3.10.4.Final.jar:?]

    at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:44) ~[netty-3.10.4.Final.jar:?]

    at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:28) ~[netty-3.10.4.Final.jar:?]

    at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:80) ~[netty-3.10.4.Final.jar:?]

    at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39) ~[netty-3.10.4.Final.jar:?]

    at com.metamx.http.client.HttpClientInit.createBootstrap(HttpClientInit.java:133) ~[http-client-1.0.4.jar:?]

    at com.metamx.http.client.HttpClientInit.createClient(HttpClientInit.java:85) ~[http-client-1.0.4.jar:?]

    at io.druid.server.initialization.HttpEmitterModule.getEmitter(HttpEmitterModule.java:81) ~[druid-server-0.9.2.jar:0.9.2]

    at io.druid.server.initialization.HttpEmitterModule$$FastClassByGuice$$cdfcd77b.invoke(<generated>) ~[druid-server-0.9.2.jar:0.9.2]

    at com.google.inject.internal.ProviderMethod$FastClassProviderMethod.doProvision(ProviderMethod.java:264) ~[guice-4.1.0.jar:?]

    at com.google.inject.internal.ProviderMethod$Factory.provision(ProviderMethod.java:401) ~[guice-4.1.0.jar:?]

    at com.google.inject.internal.ProviderMethod$Factory.get(ProviderMethod.java:376) ~[guice-4.1.0.jar:?]

    at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) ~[guice-4.1.0.jar:?]

    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) ~[guice-4.1.0.jar:?]

    at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ~[guice-4.1.0.jar:?]

    at io.druid.guice.LifecycleScope$1.get(LifecycleScope.java:68) ~[druid-api-0.9.2.jar:0.9.2]

    at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) ~[guice-4.1.0.jar:?]

    at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019) ~[guice-4.1.0.jar:?]

    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) ~[guice-4.1.0.jar:?]

My middle manager runtime properties are:

druid.service=druid/middleManager

druid.port=8091

Number of tasks per middleManager

druid.worker.capacity=3

Task launch parameters

druid.indexer.runner.javaOpts=-server -Xmx3g -Duser.timezone=UTC -Dfile.encoding=UTF-8 -XX:MaxPermSize=512m -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -XX:+UseG1GC -XX:MaxGCPauseMillis=100 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps

Task Logging

druid.indexer.logs.directory=/mnt/log/tasks

HTTP server threads

druid.server.http.numThreads=25

Processing threads and buffers

druid.processing.buffer.sizeBytes=536870912

druid.processing.numThreads=4

Hadoop indexing

druid.indexer.task.hadoopWorkingPath=hdfs://stampy/druid/workingPath

druid.indexer.task.defaultHadoopCoordinates=[“org.apache.hadoop:hadoop-client:2.6.0”]

druid.indexer.fork.property.druid.indexer.task.hadoopWorkingPath=hdfs://stampy/druid/workingPath

druid.indexer.fork.property.druid.monitoring.monitors=[“com.metamx.metrics.JvmMonitor”]

and the jvm.config of the middle manager is:

-server

-Xmx3g

-Xms1g

-XX:+PrintGCDetails

-Duser.timezone=UTC

-Dfile.encoding=UTF-8

-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager

-Djava.io.tmpdir=/mnt/tmp

When I do a ulimit -u from the user that is running middlemanager I get 1024 as the response.

Any help in resolving this i issue will be appreciated

Thanks,

Asra

Any updates on this?