java.lang.OutOfMemoryError: unable to create new native thread while running druid ingestion

Hi ,

Even I am facing error while running druid ingestion job.

This is my error stack trace:

2018-02-12T16:40:06,363 ERROR [main] io.druid.cli.CliPeon - Error when starting up. Failing.
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.7.0_71]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[?:1.7.0_71]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.7.0_71]
at java.lang.reflect.Method.invoke(Method.java:606) ~[?:1.7.0_71]
at com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler.start(Lifecycle.java:350) ~[java-util-0.27.10.jar:?]
at com.metamx.common.lifecycle.Lifecycle.start(Lifecycle.java:259) ~[java-util-0.27.10.jar:?]
at io.druid.guice.LifecycleModule$2.start(LifecycleModule.java:155) ~[druid-api-0.9.2.1.jar:0.9.2.1]
at io.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:101) [druid-services-0.9.2.1.jar:0.9.2.1]
at io.druid.cli.CliPeon.run(CliPeon.java:274) [druid-services-0.9.2.1.jar:0.9.2.1]
at io.druid.cli.Main.main(Main.java:106) [druid-services-0.9.2.1.jar:0.9.2.1]

Caused by: java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method) ~[?:1.7.0_71]
at java.lang.Thread.start(Thread.java:714) ~[?:1.7.0_71]
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949) ~[?:1.7.0_71]
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1360) ~[?:1.7.0_71]
at java.util.concurrent.Executors$DelegatedExecutorService.execute(Executors.java:628) ~[?:1.7.0_71]
at io.druid.curator.ShutdownNowIgnoringExecutorService.execute(ShutdownNowIgnoringExecutorService.java:132) ~[druid-server-0.9.2.1.jar:0.9.2.1]
at org.apache.curator.utils.CloseableExecutorService.submit(CloseableExecutorService.java:191) ~[curator-client-2.11.0.jar:?]
at org.apache.curator.framework.recipes.cache.PathChildrenCache.submitToExecutor(PathChildrenCache.java:813) ~[curator-recipes-2.11.0.jar:?]
at org.apache.curator.framework.recipes.cache.PathChildrenCache.offerOperation(PathChildrenCache.java:763) ~[curator-recipes-2.11.0.jar:?]
at org.apache.curator.framework.recipes.cache.PathChildrenCache.start(PathChildrenCache.java:297) ~[curator-recipes-2.11.0.jar:?]
at org.apache.curator.framework.recipes.cache.PathChildrenCache.start(PathChildrenCache.java:236) ~[curator-recipes-2.11.0.jar:?]
at io.druid.curator.announcement.Announcer.startCache(Announcer.java:373) ~[druid-server-0.9.2.1.jar:0.9.2.1]
at io.druid.curator.announcement.Announcer.announce(Announcer.java:259) ~[druid-server-0.9.2.1.jar:0.9.2.1]
at io.druid.curator.announcement.Announcer.announce(Announcer.java:152) ~[druid-server-0.9.2.1.jar:0

2018-02-12T16:50:01,687 ERROR [task-runner-0-priority-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running task[HadoopIndexTask{id=index_hadoop_wikiticker_2018-02-12T16:49:49.956Z, type=index_hadoop, dataSource=wikiticker}]
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.1.jar:?]
at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:204) ~[druid-indexing-service-0.9.2.1.jar:0.9.2.1]
at io.druid.indexing.common.task.HadoopIndexTask.run(HadoopIndexTask.java:175) ~[druid-indexing-service-0.9.2.1.jar:0.9.2.1]
at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) [druid-indexing-service-0.9.2.1.jar:0.9.2.1]
at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) [druid-indexing-service-0.9.2.1.jar:0.9.2.1]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_71]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_71]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_71]
at java.lang.Thread.run(Thread.java:745) [?:1.7.0_71]
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.7.0_71]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[?:1.7.0_71]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.7.0_71]
at java.lang.reflect.Method.invoke(Method.java:606) ~[?:1.7.0_71]
at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:201) ~[druid-indexing-service-0.9.2.1.jar:0.9.2.1]
… 7 more
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method) ~[?:1.7.0_71]

at org.apache.hadoop.hdfs.DFSOutputStream.start(DFSOutputStream.java:2141) ~[?:?]
at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1656) ~[?:?]
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1703) ~[?:?]
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1638) ~[?:?]
at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448) ~[?:?]
at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444) ~[?:?]
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) ~[?:?]
at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459) ~[?:?]
at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387) ~[?:?]
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:909) ~[?:?]
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:890) ~[?:?]
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:787) ~[?:?]
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:776) ~[?:?]
at io.druid.indexer.JobHelper$3.getOutput(JobHelper.java:297) ~[druid-indexing-hadoop-0.9.2.1.jar:0.9.2.1]
at io.druid.indexer.JobHelper$3.getOutput(JobHelper.java:293) ~[druid-indexing-hadoop-0.9.2.1.jar:0.9.2.1]
at com.google.common.io.ByteStreams$4.openStream(ByteStreams.java:1031) ~[guava-16.0.1.jar:?]
at com.google.common.io.ByteSource.copyTo(ByteSource.java:254) ~[guava-16.0.1.jar:?]
at com.google.common.io.ByteStreams.copy(ByteStreams.java:119) ~[guava-16.0.1.jar:?]
at io.druid.indexer.JobHelper.uploadJar(JobHelper.java:290) ~[druid-indexing-hadoop-0.9.2.1.jar:0.9.2.1]
at io.druid.indexer.JobHelper.addJarToClassPath(JobHelper.java:222) ~[druid-indexing-hadoop-0.9.2.1.jar:0.9.2.1]
at io.druid.indexer.JobHelper$1.call(JobHelper.java:169) ~[druid-indexing-hadoop-0.9.2.1.jar:0.9.2.1]
at io.druid.indexer.JobHelper$1.call(JobHelper.java:162) ~[druid-indexing-

I have 6GB RAM

These are my middle manager runtime and jvm properties

runtime_properties

druid.service=druid/middleManager
druid.port=8091

Number of tasks per middleManager

druid.worker.capacity=3

Task launch parameters

druid.indexer.runner.javaOpts=-server -Xmx2g -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
druid.indexer.task.baseTaskDir=var/druid/task

HTTP server threads

druid.server.http.numThreads=8

Processing threads and buffers

druid.processing.buffer.sizeBytes=256000000
druid.processing.numThreads=2

Hadoop indexing

druid.indexer.task.hadoopWorkingPath=var/druid/hadoop-tmp
druid.indexer.task.defaultHadoopCoordinates=[“org.apache.hadoop:hadoop-client:2.3.0”]

jvm_properties

-server
-Xms64m
-Xmx64m
-Duser.timezone=UTC
-Dfile.encoding=UTF-8
-Djava.io.tmpdir=var/tmp
-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager

What could possibly be causing this issue?

Hi Saurabh,

You might need to bump up your processes ulimit. You can check what it is by running “ulimit -u” and change it by running, for example, “ulimit -u 15000”.

Hi Gina,

Thanks for reply, I have already considered this approach of increasing u limit bt I am running druid in environment on which I don’t have root access. Tried different arrangements of java conf for middle manager bt nothing works. Although it works fine in my local. U limit for user is 1024 and ram is still available .Is druid is suppose to open more then 1000 files ?. I jave 6 gb of ram and 1 core .is there is any configurations changes which could help me out.

Thanks ,

Saurabh

It’s 1000 processes (or threads), not files. You could try doing a “jstack -l [pid]” of the peon to see what they are. 1000 does seem a bit excessive for a peon, there might be some thread pool getting greedy.

You could also try upgrading Druid. Version 0.9.2 is pretty old.

As i earlier said i doesn’t have the root access therefore i am binded to use java 7 along with hadoop running on java 7 (to avoid major-minor version error).

Ulimit for my local mac system is 709 and druid works well on it.

Is there any conf changes which can make it work ?