indexer Auto-Scaling with static-s3 firehose connection Reset

Hi Guys,

I am doing indexer Auto-Scaling with static-s3 firehose.

Its throwing exception like below.

FireHose reading events from S3 thats why connection are in very important mode.

Any Other option we can do on this.

2015-03-23T10:24:51,288 INFO [task-runner-0] io.druid.segment.LoggingProgressIndicator - [/tmp/persistent/task/index_ec2_test_cluster_2015-03-23T10:16:19.241Z/work/ec2_test_cluster_2015-03-19T00:09:00.000Z_2015-03-19T00:10:00.000Z_2015-03-23T10:16:19.242Z_0/ec2_test_cluster_2015-03-19T00:09:00.000Z_2015-03-19T00:10:00.000Z_2015-03-23T10:16:19.242Z/spill0]: [make zip] has completed. Elapsed time: [8,641] millis

2015-03-23T10:24:51,288 INFO [task-runner-0] io.druid.segment.LoggingProgressIndicator - [/tmp/persistent/task/index_ec2_test_cluster_2015-03-23T10:16:19.241Z/work/ec2_test_cluster_2015-03-19T00:09:00.000Z_2015-03-19T00:10:00.000Z_2015-03-23T10:16:19.242Z_0/ec2_test_cluster_2015-03-19T00:09:00.000Z_2015-03-19T00:10:00.000Z_2015-03-23T10:16:19.242Z/spill0]: [make dimension columns] has completed. Elapsed time: [285,785] millis

2015-03-23T10:24:51,288 INFO [task-runner-0] io.druid.segment.LoggingProgressIndicator - [/tmp/persistent/task/index_ec2_test_cluster_2015-03-23T10:16:19.241Z/work/ec2_test_cluster_2015-03-19T00:09:00.000Z_2015-03-19T00:10:00.000Z_2015-03-23T10:16:19.242Z_0/ec2_test_cluster_2015-03-19T00:09:00.000Z_2015-03-19T00:10:00.000Z_2015-03-23T10:16:19.242Z/spill0]: Starting [make metric columns]

2015-03-23T10:24:51,288 INFO [task-runner-0] io.druid.segment.LoggingProgressIndicator - [/tmp/persistent/task/index_ec2_test_cluster_2015-03-23T10:16:19.241Z/work/ec2_test_cluster_2015-03-19T00:09:00.000Z_2015-03-19T00:10:00.000Z_2015-03-23T10:16:19.242Z_0/ec2_test_cluster_2015-03-19T00:09:00.000Z_2015-03-19T00:10:00.000Z_2015-03-23T10:16:19.242Z/spill0]: Starting [make column[numz]]

2015-03-23T10:24:59,518 INFO [task-runner-0] io.druid.segment.LoggingProgressIndicator - [/tmp/persistent/task/index_ec2_test_cluster_2015-03-23T10:16:19.241Z/work/ec2_test_cluster_2015-03-19T00:09:00.000Z_2015-03-19T00:10:00.000Z_2015-03-23T10:16:19.242Z_0/ec2_test_cluster_2015-03-19T00:09:00.000Z_2015-03-19T00:10:00.000Z_2015-03-23T10:16:19.242Z/spill0]: [make column[numz]] has completed. Elapsed time: [8,229] millis

2015-03-23T10:24:59,518 INFO [task-runner-0] io.druid.segment.LoggingProgressIndicator - [/tmp/persistent/task/index_ec2_test_cluster_2015-03-23T10:16:19.241Z/work/ec2_test_cluster_2015-03-19T00:09:00.000Z_2015-03-19T00:10:00.000Z_2015-03-23T10:16:19.242Z_0/ec2_test_cluster_2015-03-19T00:09:00.000Z_2015-03-19T00:10:00.000Z_2015-03-23T10:16:19.242Z/spill0]: [make metric columns] has completed. Elapsed time: [8,229] millis

2015-03-23T10:24:59,518 INFO [task-runner-0] io.druid.segment.LoggingProgressIndicator - [/tmp/persistent/task/index_ec2_test_cluster_2015-03-23T10:16:19.241Z/work/ec2_test_cluster_2015-03-19T00:09:00.000Z_2015-03-19T00:10:00.000Z_2015-03-23T10:16:19.242Z_0/ec2_test_cluster_2015-03-19T00:09:00.000Z_2015-03-19T00:10:00.000Z_2015-03-23T10:16:19.242Z/spill0]: Starting [building index.drd]

2015-03-23T10:24:59,520 INFO [task-runner-0] io.druid.segment.LoggingProgressIndicator - [/tmp/persistent/task/index_ec2_test_cluster_2015-03-23T10:16:19.241Z/work/ec2_test_cluster_2015-03-19T00:09:00.000Z_2015-03-19T00:10:00.000Z_2015-03-23T10:16:19.242Z_0/ec2_test_cluster_2015-03-19T00:09:00.000Z_2015-03-19T00:10:00.000Z_2015-03-23T10:16:19.242Z/spill0]: [building index.drd] has completed. Elapsed time: [1] millis

2015-03-23T10:24:59,521 INFO [task-runner-0] io.druid.segment.LoggingProgressIndicator - [/tmp/persistent/task/index_ec2_test_cluster_2015-03-23T10:16:19.241Z/work/ec2_test_cluster_2015-03-19T00:09:00.000Z_2015-03-19T00:10:00.000Z_2015-03-23T10:16:19.242Z_0/ec2_test_cluster_2015-03-19T00:09:00.000Z_2015-03-19T00:10:00.000Z_2015-03-23T10:16:19.242Z/spill0] complete. Elapsed time: [313,023] millis

2015-03-23T10:24:59,521 INFO [task-runner-0] io.druid.segment.ReferenceCountingSegment - Closing ec2_test_cluster_2015-03-19T00:09:00.000Z_2015-03-19T00:10:00.000Z_2015-03-23T10:16:19.242Z

2015-03-23T10:24:59,522 INFO [task-runner-0] io.druid.segment.ReferenceCountingSegment - Closing ec2_test_cluster_2015-03-19T00:09:00.000Z_2015-03-19T00:10:00.000Z_2015-03-23T10:16:19.242Z, numReferences: 0

2015-03-23T10:24:59,981 ERROR [task-runner-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running task[IndexTask{id=index_ec2_test_cluster_2015-03-23T10:16:19.241Z, type=index, dataSource=ec2_test_cluster}]

java.lang.IllegalStateException: java.net.SocketException: Connection reset

at org.apache.commons.io.LineIterator.hasNext(LineIterator.java:107) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]

at io.druid.data.input.impl.FileIteratingFirehose.hasMore(FileIteratingFirehose.java:34) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]

at io.druid.indexing.common.task.IndexTask.generateSegment(IndexTask.java:343) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]

at io.druid.indexing.common.task.IndexTask.run(IndexTask.java:185) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]

at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:235) [druid-services-0.7.0-selfcontained.jar:0.7.0]

at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:214) [druid-services-0.7.0-selfcontained.jar:0.7.0]

at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_75]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_75]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_75]

at java.lang.Thread.run(Thread.java:745) [?:1.7.0_75]

Caused by: java.net.SocketException: Connection reset

at java.net.SocketInputStream.read(SocketInputStream.java:196) ~[?:1.7.0_75]

at java.net.SocketInputStream.read(SocketInputStream.java:122) ~[?:1.7.0_75]

at sun.security.ssl.InputRecord.readFully(InputRecord.java:442) ~[?:1.7.0_75]

at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:554) ~[?:1.7.0_75]

at sun.security.ssl.InputRecord.read(InputRecord.java:509) ~[?:1.7.0_75]

at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:934) ~[?:1.7.0_75]

at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:891) ~[?:1.7.0_75]

at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) ~[?:1.7.0_75]

at org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:204) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]

at org.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:182) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]

at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:138) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]

at org.jets3t.service.io.InterruptableInputStream.read(InterruptableInputStream.java:78) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]

at org.jets3t.service.impl.rest.httpclient.HttpMethodReleaseInputStream.read(HttpMethodReleaseInputStream.java:146) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]

at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283) ~[?:1.7.0_75]

at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325) ~[?:1.7.0_75]

at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177) ~[?:1.7.0_75]

at java.io.InputStreamReader.read(InputStreamReader.java:184) ~[?:1.7.0_75]

at java.io.BufferedReader.fill(BufferedReader.java:154) ~[?:1.7.0_75]

at java.io.BufferedReader.readLine(BufferedReader.java:317) ~[?:1.7.0_75]

at java.io.BufferedReader.readLine(BufferedReader.java:382) ~[?:1.7.0_75]

at org.apache.commons.io.LineIterator.hasNext(LineIterator.java:96) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]

… 9 more

Thanks.

Jitesh

The Hadoop indexer is generally a faster and more reliable method of ingesting data from S3.

Thanks Gian for reply.

Any Other option if we want to run indexer for S3 data?

Thanks

Jitesh

S3’s currently only supported by the hadoop indexer and by the regular index task through the static s3 firehose. If you feel like getting your hands dirty, you could modify the static S3 firehose to handle errors better (probably retry?). If you end up doing that, a PR would be awesome.

Hi Gian and Fangjin ,

I Don’t want to use hadoop indexer. If we want to run indexer on S3 cloud storage file we have static-s3 firehose.

But this is not reliable and I am facing issue from this. What I did . I copy file from S3 to local by task submitting with running indexer by local firehose.

I think it is good idea to run indexer in Autoscaling by copy s3 file to local instead of running Statis-S3 firehose .

I written a few line code under IndexTesk.java to copy S3 file to local and use local firehose on it.

It is running Successfully for me.

Thanks

Jitesh

java-util just got some patches related to retries:

I would love feedback to see if they are useful if you happen to go down this route.