Hi,
My static-s3 ingestion is timing out. I tried specifying the fetchTimeout but it is being neglected/removed when I submit the task.
Here is my configuration when I submit the task
“ioConfig” : {
“type” : “index”,
“firehose” : {
“type” : “static-s3”,
“uris”: [“S3_FILE_PATH”],
“fetchTimeout” : “111111”
},
“appendToExisting” : true,
“fetchTimeout” : “222222”
}
But when I see the payload after the task is submitted, druid removed the fetchTimeout.
“ioConfig”: {
“type”: “index”,
“firehose”: {
“type”: “static-s3”,
“uris”: [
“s3://bucket/part-00000.csv”
],
“prefixes”: [
]
},
“appendToExisting”: true
},
“tuningConfig”: {
“type”: “index”,
“targetPartitionSize”: 5000000,
“maxRowsInMemory”: 75000,
“numShards”: null,
“indexSpec”: {
“bitmap”: {
“type”: “concise”
},
“dimensionCompression”: “lz4”,
“metricCompression”: “lz4”,
“longEncoding”: “longs”
},
“maxPendingPersists”: 0,
“buildV9Directly”: true,
“forceExtendableShardSpecs”: false,
“reportParseExceptions”: false,
“publishTimeout”: 0
}
}
I am getting timeout during the ingestion of s3 files…
java.lang.RuntimeException: java.util.concurrent.TimeoutException
at io.druid.data.input.impl.PrefetchableTextFilesFirehoseFactory$1.openObjectFromLocal(PrefetchableTextFilesFirehoseFactory.java:369) ~[druid-api-0.10.1.jar:0.10.1]
at io.druid.data.input.impl.PrefetchableTextFilesFirehoseFactory$1.next(PrefetchableTextFilesFirehoseFactory.java:317) ~[druid-api-0.10.1.jar:0.10.1]
at io.druid.data.input.impl.PrefetchableTextFilesFirehoseFactory$1.next(PrefetchableTextFilesFirehoseFactory.java:190) ~[druid-api-0.10.1.jar:0.10.1]
at io.druid.data.input.impl.FileIteratingFirehose.getNextLineIterator(FileIteratingFirehose.java:88) ~[druid-api-0.10.1.jar:0.10.1]
at io.druid.data.input.impl.FileIteratingFirehose.hasMore(FileIteratingFirehose.java:66) ~[druid-api-0.10.1.jar:0.10.1]
at io.druid.indexing.common.task.IndexTask.determineShardSpecs(IndexTask.java:272) ~[druid-indexing-service-0.10.1.jar:0.10.1]
at io.druid.indexing.common.task.IndexTask.run(IndexTask.java:187) ~[druid-indexing-service-0.10.1.jar:0.10.1]
at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) [druid-indexing-service-0.10.1.jar:0.10.1]
at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) [druid-indexing-service-0.10.1.jar:0.10.1]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
Caused by: java.util.concurrent.TimeoutException
... 13 more
How do I specify fetchTimeout for static-s3 ingestion?
Regards,
Mohan