How to specify fetchTimeout for static-s3 ingestion?

Hi,

My static-s3 ingestion is timing out. I tried specifying the fetchTimeout but it is being neglected/removed when I submit the task.

Here is my configuration when I submit the task

“ioConfig” : {

“type” : “index”,

“firehose” : {

“type” : “static-s3”,

“uris”: [“S3_FILE_PATH”],

“fetchTimeout” : “111111”

},

“appendToExisting” : true,

“fetchTimeout” : “222222”

}

But when I see the payload after the task is submitted, druid removed the fetchTimeout.

“ioConfig”: {

“type”: “index”,

“firehose”: {

“type”: “static-s3”,

“uris”: [

“s3://bucket/part-00000.csv”

],

“prefixes”: [

]

},

“appendToExisting”: true

},

“tuningConfig”: {

“type”: “index”,

“targetPartitionSize”: 5000000,

“maxRowsInMemory”: 75000,

“numShards”: null,

“indexSpec”: {

“bitmap”: {

“type”: “concise”

},

“dimensionCompression”: “lz4”,

“metricCompression”: “lz4”,

“longEncoding”: “longs”

},

“maxPendingPersists”: 0,

“buildV9Directly”: true,

“forceExtendableShardSpecs”: false,

“reportParseExceptions”: false,

“publishTimeout”: 0

}

}

I am getting timeout during the ingestion of s3 files…

java.lang.RuntimeException: java.util.concurrent.TimeoutException
	at io.druid.data.input.impl.PrefetchableTextFilesFirehoseFactory$1.openObjectFromLocal(PrefetchableTextFilesFirehoseFactory.java:369) ~[druid-api-0.10.1.jar:0.10.1]
	at io.druid.data.input.impl.PrefetchableTextFilesFirehoseFactory$1.next(PrefetchableTextFilesFirehoseFactory.java:317) ~[druid-api-0.10.1.jar:0.10.1]
	at io.druid.data.input.impl.PrefetchableTextFilesFirehoseFactory$1.next(PrefetchableTextFilesFirehoseFactory.java:190) ~[druid-api-0.10.1.jar:0.10.1]
	at io.druid.data.input.impl.FileIteratingFirehose.getNextLineIterator(FileIteratingFirehose.java:88) ~[druid-api-0.10.1.jar:0.10.1]
	at io.druid.data.input.impl.FileIteratingFirehose.hasMore(FileIteratingFirehose.java:66) ~[druid-api-0.10.1.jar:0.10.1]
	at io.druid.indexing.common.task.IndexTask.determineShardSpecs(IndexTask.java:272) ~[druid-indexing-service-0.10.1.jar:0.10.1]
	at io.druid.indexing.common.task.IndexTask.run(IndexTask.java:187) ~[druid-indexing-service-0.10.1.jar:0.10.1]
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) [druid-indexing-service-0.10.1.jar:0.10.1]
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) [druid-indexing-service-0.10.1.jar:0.10.1]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_151]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
Caused by: java.util.concurrent.TimeoutException
	... 13 more


How do I specify fetchTimeout for static-s3 ingestion?

Regards,

Mohan

Any suggestion on this?

Can someone share a sample to specify fetchTimeout…It would be really helpful. None of my attempts have been successful so far with the timeouts…

Thanks in advance…