ERROR [task-runner-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Uncaught Throwable while run

Hi. I am testing the druid cluster for our company’s realtime analytics platform.

I am testing at 5 Separate Node cluster. ( broker, historical , middlemanager , coordinator , overlord )

I am used zookeeper , mysql ,S3 storage and local file storage at this test.

In our production Hadoop eco system, we have hadoop and memcached layer. But, that system is production environment. So, At this testing, I don’t use hadoop and memcached.

But, At this test, the batch data loading process alaways fail.

Is there any problem in my setting file???

I use this command.

./load-druid.sh druid05_domain:8080 /data01/druid/druid-0.7.0/examples/indexing/wikipedia_index_task.json

ps.

i don’t know the baseKey’s meaning . so, at below property, i use the s3 folder name.

druid.indexer.fork.property.druid.storage.baseKey = prod

---------_common --------

Extensions

druid.extensions.coordinates=[“io.druid.extensions:druid-s3-extensions”,“io.druid.extensions:druid-examples”,“io.druid.extensions:druid-histogram”,“io.druid.extensions:mysql-metadata-storage”]

Zookeeper

druid.zk.service.host=vm-druid04

druid.zk.base=/druid/prod

druid.discovery.curator.path=/prod/discovery

Request logging, monitoring, and metrics

druid.request.logging.type=emitter

druid.request.logging.feed=druid_requests

Metadata Storage (mysql)

druid.metadata.storage.type=mysql

druid.metadata.storage.connector.connectURI=jdbc:mysql://vm-druid05:3306/druid

druid.metadata.storage.connector.user=druid

druid.metadata.storage.connector.password=***********

Deep storage (local filesystem for examples - don’t use this in production)

druid.storage.type=s3

druid.s3.accessKey=******

druid.s3.secretKey=**********

druid.storage.storage.storageDirectory=/mnt/resource/data02/druid/localStorage

Query Cache (we use a simple 1000mb heap-based local cache on the broker)

druid.cache.type=local

druid.cache.sizeInBytes=1000000000

Indexing service discovery

druid.selectors.indexing.serviceName=druid:prod:overlord

Monitoring (disabled for examples)

druid.monitoring.monitors=[“com.metamx.metrics.SysMonitor”,“com.metamx.metrics.JvmMonitor”]

Metrics logging (disabled for examples)

druid.emitter=noop

--------- broker ----------

druid.host=vm-broker01

druid.port=8080

druid.service=druid/prod/broker

We enable using the local query cache here

druid.broker.cache.useCache=true

druid.broker.cache.populateCache=true

druid.broker.http.numConnections=20

druid.broker.http.readTimeout=PT5M

Bump these up only for faster nested groupBy

druid.processing.buffer.sizeBytes=2147483647

druid.processing.numThreads=31

druid.server.http.numThreads=50

druid.announcer.type=batch

druid.curator.compress=true

---------middle manager --------

druid.host:8080=vm-druid02.cloudapp.net

druid.port=8080

druid.service=druid/prod/middlemanager

Store task logs in deep storage

#druid.indexer.logs.type=s3

#druid.indexer.logs.s3Bucket=log-hoondongkim

#druid.indexer.logs.s3Prefix=prod/logs/v1

Resources for peons

druid.indexer.runner.javaOpts=-server -Xmx3g -XX:+UseG1GC -XX:MaxGCPauseMillis=100 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps

druid.indexer.task.baseTaskDir=/mnt/resource/data02/persistent/task/

Peon properties

druid.indexer.fork.property.druid.monitoring.monitors=[“com.metamx.metrics.JvmMonitor”]

druid.indexer.fork.property.druid.processing.buffer.sizeBytes=536870912

druid.indexer.fork.property.druid.processing.numThreads=2

druid.indexer.fork.property.druid.segmentCache.locations=[{“path”: “/mnt/resource/data02/persistent/zk_druid”, “maxSize”: 0}]

druid.indexer.fork.property.druid.server.http.numThreads=50

druid.indexer.fork.property.druid.storage.archiveBaseKey=prod

druid.indexer.fork.property.druid.storage.archiveBucket=archive-hoondongkim

druid.indexer.fork.property.druid.storage.baseKey=prod/v1

druid.indexer.fork.property.druid.storage.bucket=hoondongkim

druid.indexer.fork.property.druid.storage.type=s3

druid.worker.capacity=9

druid.worker.ip=vm-druid02

druid.worker.version=0

---------- overload ----------

druid.host=vm-druid05:8080

druid.port=8080

druid.service=druid/prod/overlord

Run the overlord in local mode with a single peon to execute tasks

#druid.indexer.queue.startDelay=PT5M

#druid.indexer.runner.javaOpts="-server -Xmx16g"

#druid.indexer.fork.property.druid.processing.numThreads=1

#druid.indexer.fork.property.druid.computation.buffer.size=100000000

Only required if you are autoscaling middle managers

druid.indexer.autoscale.doAutoscale=true

druid.indexer.autoscale.strategy=ec2

druid.indexer.autoscale.workerIdleTimeout=PT90m

druid.indexer.autoscale.terminatePeriod=PT5M

druid.indexer.autoscale.workerVersion=0

Upload all task logs to deep storage

druid.indexer.logs.type=s3

druid.indexer.logs.s3Bucket=log-hoondongkim

druid.indexer.logs.s3Prefix=prod/logs/v1

Run in remote mode

druid.indexer.runner.type=remote

druid.indexer.runner.minWorkerVersion=0

Store all task state in the metadata storage

druid.indexer.storage.type=metadata

------------------------------ Error Message ---------------------------------

WARNING: Multiple Servlet injectors detected. This is a warning indicating that you have more than one GuiceFilter running in your web application. If this is deliberate, you may safely ignore this message. If this is NOT deliberate however, your application may not work as expected.

2015-04-22T03:47:51,815 INFO [task-runner-0] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.metrics.DruidMonitorSchedulerConfig] from props[druid.monitoring.] as [io.druid.server.metrics.DruidMonitorSchedulerConfig@8c7bf95]

2015-04-22T03:47:51,824 INFO [task-runner-0] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.metrics.MonitorsConfig] from props[druid.monitoring.] as [MonitorsConfig{monitors=[class com.metamx.metrics.JvmMonitor]}]

2015-04-22T03:47:51,825 INFO [task-runner-0] io.druid.server.metrics.MetricsModule - Adding monitor[com.metamx.metrics.JvmMonitor@5828da3d]

2015-04-22T03:47:51,848 INFO [task-runner-0] io.druid.indexing.common.task.IndexTask - Estimated approximately [5.006113] rows of data.

2015-04-22T03:47:51,849 INFO [task-runner-0] io.druid.indexing.common.task.IndexTask - Will require [1] shard(s).

2015-04-22T03:47:51,850 INFO [task-runner-0] io.druid.segment.realtime.firehose.LocalFirehoseFactory - Searching for all [wikipedia_data.json] in [/data01/druid/druid-0.7.0/examples/indexing]

2015-04-22T03:47:51,935 ERROR [task-runner-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Uncaught Throwable while running task[IndexTask{id=index_wikipedia_2015-04-22T03:47:42.851Z, type=index, dataSource=wikipedia}]

java.lang.ExceptionInInitializerError

at io.druid.indexing.common.task.IndexTask.generateSegment(IndexTask.java:323) ~[druid-indexing-service-0.7.0.jar:0.7.0]

at io.druid.indexing.common.task.IndexTask.run(IndexTask.java:185) ~[druid-indexing-service-0.7.0.jar:0.7.0]

at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:235) [druid-indexing-service-0.7.0.jar:0.7.0]

at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:214) [druid-indexing-service-0.7.0.jar:0.7.0]

at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_75]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_75]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_75]

at java.lang.Thread.run(Thread.java:745) [?:1.7.0_75]

Caused by: java.lang.IllegalStateException: Failed to create directory within 10000 attempts (tried 1429674471861-0 to 1429674471861-9999)

at com.google.common.io.Files.createTempDir(Files.java:600) ~[guava-16.0.1.jar:?]

at io.druid.segment.indexing.RealtimeTuningConfig.(RealtimeTuningConfig.java:40) ~[druid-server-0.7.0.jar:0.7.0]

… 8 more

2015-04-22T03:47:51,947 ERROR [main] io.druid.cli.CliPeon - Error when starting up. Failing.

java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.ExceptionInInitializerError

at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.1.jar:?]

at io.druid.indexing.worker.executor.ExecutorLifecycle.join(ExecutorLifecycle.java:162) ~[druid-indexing-service-0.7.0.jar:0.7.0]

at io.druid.cli.CliPeon.run(CliPeon.java:209) [druid-services-0.7.0.jar:0.7.0]

at io.druid.cli.Main.main(Main.java:88) [druid-services-0.7.0.jar:0.7.0]

Caused by: java.util.concurrent.ExecutionException: java.lang.ExceptionInInitializerError

at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) ~[guava-16.0.1.jar:?]

at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) ~[guava-16.0.1.jar:?]

at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-16.0.1.jar:?]

at io.druid.indexing.worker.executor.ExecutorLifecycle.join(ExecutorLifecycle.java:159) ~[druid-indexing-service-0.7.0.jar:0.7.0]

… 2 more

Caused by: java.lang.ExceptionInInitializerError

at io.druid.indexing.common.task.IndexTask.generateSegment(IndexTask.java:323) ~[druid-indexing-service-0.7.0.jar:0.7.0]

at io.druid.indexing.common.task.IndexTask.run(IndexTask.java:185) ~[druid-indexing-service-0.7.0.jar:0.7.0]

at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:235) ~[druid-indexing-service-0.7.0.jar:0.7.0]

at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:214) ~[druid-indexing-service-0.7.0.jar:0.7.0]

at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[?:1.7.0_75]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[?:1.7.0_75]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[?:1.7.0_75]

at java.lang.Thread.run(Thread.java:745) ~[?:1.7.0_75]

Caused by: java.lang.IllegalStateException: Failed to create directory within 10000 attempts (tried 1429674471861-0 to 1429674471861-9999)

at com.google.common.io.Files.createTempDir(Files.java:600) ~[guava-16.0.1.jar:?]

at io.druid.segment.indexing.RealtimeTuningConfig.(RealtimeTuningConfig.java:40) ~[druid-server-0.7.0.jar:0.7.0]

at io.druid.indexing.common.task.IndexTask.generateSegment(IndexTask.java:323) ~[druid-indexing-service-0.7.0.jar:0.7.0]

at io.druid.indexing.common.task.IndexTask.run(IndexTask.java:185) ~[druid-indexing-service-0.7.0.jar:0.7.0]

at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:235) ~[druid-indexing-service-0.7.0.jar:0.7.0]

at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:214) ~[druid-indexing-service-0.7.0.jar:0.7.0]

at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[?:1.7.0_75]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[?:1.7.0_75]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[?:1.7.0_75]

at java.lang.Thread.run(Thread.java:745) ~[?:1.7.0_75]

Heap

garbage-first heap total 902144K, used 181248K [0x0000000735a00000, 0x000000076cb00000, 0x00000007f5a00000)

region size 1024K, 174 young (178176K), 14 survivors (14336K)

compacting perm gen total 43008K, used 42130K [0x00000007f5a00000, 0x00000007f8400000, 0x0000000800000000)

the space 43008K, 97% used [0x00000007f5a00000, 0x00000007f8324ba0, 0x00000007f8324c00, 0x00000007f8400000)

No shared spaces configured.

Hi,
the above stack trace is failing to do Files.createTempDir(),

have you set your “java.io.tmpdir” to a directory where you do not have permissions to create a file ?
can you try changing your java.io.tmpdir to some other directory ?

Hi Nishant Bangarwa~~

Thank you very much!

I Chanaged the java.io.tmpdir path. And, All works fine!

Great!

Thank you!!!

2015년 4월 22일 수요일 오후 3시 59분 32초 UTC+9, Nishant Bangarwa 님의 말: