Druid ingestion data show java.util.concurrent.TimeoutException

Apache Druid Q&A Google Groups
#1

Hello! Everyone.

I have a question when I ingest data from google storage to druid.

My file size is 7GB.

But…It shows java.util.concurrent.TimeoutException.

Error is following:

2019-04-11T02:04:44,394 ERROR [task-runner-0-priority-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running task[IndexTask{id=index_dmp_referer_2019-04-11T02:02:31.915Z, type=index, dataSource=dmp_referer}]
java.lang.RuntimeException: java.util.concurrent.TimeoutException
	at io.druid.data.input.impl.PrefetchableTextFilesFirehoseFactory$1.openObjectFromLocal(PrefetchableTextFilesFirehoseFactory.java:369) ~[druid-api-0.10.1.2.6.5.0-292.jar:0.10.1.2.6.5.0-292]
	at io.druid.data.input.impl.PrefetchableTextFilesFirehoseFactory$1.next(PrefetchableTextFilesFirehoseFactory.java:317) ~[druid-api-0.10.1.2.6.5.0-292.jar:0.10.1.2.6.5.0-292]
	at io.druid.data.input.impl.PrefetchableTextFilesFirehoseFactory$1.next(PrefetchableTextFilesFirehoseFactory.java:190) ~[druid-api-0.10.1.2.6.5.0-292.jar:0.10.1.2.6.5.0-292]
	at io.druid.data.input.impl.FileIteratingFirehose.getNextLineIterator(FileIteratingFirehose.java:88) ~[druid-api-0.10.1.2.6.5.0-292.jar:0.10.1.2.6.5.0-292]
	at io.druid.data.input.impl.FileIteratingFirehose.hasMore(FileIteratingFirehose.java:66) ~[druid-api-0.10.1.2.6.5.0-292.jar:0.10.1.2.6.5.0-292]
	at io.druid.indexing.common.task.IndexTask.determineShardSpecs(IndexTask.java:272) ~[druid-indexing-service-0.10.1.2.6.5.0-292.jar:0.10.1.2.6.5.0-292]
	at io.druid.indexing.common.task.IndexTask.run(IndexTask.java:188) ~[druid-indexing-service-0.10.1.2.6.5.0-292.jar:0.10.1.2.6.5.0-292]
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) [druid-indexing-service-0.10.1.2.6.5.0-292.jar:0.10.1.2.6.5.0-292]
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) [druid-indexing-service-0.10.1.2.6.5.0-292.jar:0.10.1.2.6.5.0-292]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_201]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_201]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_201]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
	Suppressed: java.lang.IllegalStateException
		at com.google.common.base.Preconditions.checkState(Preconditions.java:161) ~[guava-16.0.1.jar:?]
		at io.druid.data.input.impl.PrefetchableTextFilesFirehoseFactory.lambda$connect$0(PrefetchableTextFilesFirehoseFactory.java:415) ~[druid-api-0.10.1.2.6.5.0-292.jar:0.10.1.2.6.5.0-292]
		at io.druid.data.input.impl.FileIteratingFirehose.close(FileIteratingFirehose.java:119) ~[druid-api-0.10.1.2.6.5.0-292.jar:0.10.1.2.6.5.0-292]
		at io.druid.indexing.common.task.IndexTask.determineShardSpecs(IndexTask.java:320) ~[druid-indexing-service-0.10.1.2.6.5.0-292.jar:0.10.1.2.6.5.0-292]
		at io.druid.indexing.common.task.IndexTask.run(IndexTask.java:188) ~[druid-indexing-service-0.10.1.2.6.5.0-292.jar:0.10.1.2.6.5.0-292]
		at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) [druid-indexing-service-0.10.1.2.6.5.0-292.jar:0.10.1.2.6.5.0-292]
		at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) [druid-indexing-service-0.10.1.2.6.5.0-292.jar:0.10.1.2.6.5.0-292]
		at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_201]
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_201]
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_201]
		at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
Caused by: java.util.concurrent.TimeoutException
	... 13 more
2019-04-11T02:04:44,400 INFO [task-runner-0-priority-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_dmp_referer_2019-04-11T02:02:31.915Z] status changed to [FAILED].
2019-04-11T02:04:44,405 INFO [task-runner-0-priority-0] io.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {
  "id" : "index_dmp_referer_2019-04-11T02:02:31.915Z",
  "status" : "FAILED",
  "duration" : 120102
}



I hope someone can help me.
#2

Hey Huei-Yuan,

You might want to try raising the “fetchTimeout” config. Also, your Druid version is pretty old, and if you upgrade you should get better timeout / error handling behavior.