Some exception when historical node load segment from s3 deep storage

Hi:

I have no idea the exception described as below when historical node load segment from s3 deep storage

io.druid.segment.loading.SegmentLoadingException: Exception loading segment[nrt_dashboard_2015-06-24T06:35:00.000Z_2015-06-24T06:40:00.000Z_2015-06-24T06:35:00.000Z_1]

at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:138) ~[druid-server-0.7.1.1.jar:0.7.1.1]

at io.druid.server.coordination.ZkCoordinator.addSegment(ZkCoordinator.java:163) [druid-server-0.7.1.1.jar:0.7.1.1]

at io.druid.server.coordination.SegmentChangeRequestLoad.go(SegmentChangeRequestLoad.java:42) [druid-server-0.7.1.1.jar:0.7.1.1]

at io.druid.server.coordination.BaseZkCoordinator$1.childEvent(BaseZkCoordinator.java:125) [druid-server-0.7.1.1.jar:0.7.1.1]

at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:516) [curator-recipes-2.7.0.jar:?]

at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:510) [curator-recipes-2.7.0.jar:?]

at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92) [curator-framework-2.7.0.jar:?]

at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) [guava-16.0.1.jar:?]

at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:83) [curator-framework-2.7.0.jar:?]

at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:507) [curator-recipes-2.7.0.jar:?]

at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35) [curator-recipes-2.7.0.jar:?]

at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:759) [curator-recipes-2.7.0.jar:?]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [?:1.7.0_65]

at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_65]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [?:1.7.0_65]

at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_65]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_65]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_65]

at java.lang.Thread.run(Thread.java:745) [?:1.7.0_65]

Caused by: io.druid.segment.loading.SegmentLoadingException: S3 fail! Key[s3://stat.bus.ijinshan/datahero/druid_prod/index/nrt_dashboard/2015-06-24T06:35:00.000Z_2015-06-24T06:40:00.000Z/2015-06-24T06:35:00.000Z/1/index.zip]

at io.druid.storage.s3.S3DataSegmentPuller.isObjectInBucket(S3DataSegmentPuller.java:161) ~[?:?]

at io.druid.storage.s3.S3DataSegmentPuller.getSegmentFiles(S3DataSegmentPuller.java:69) ~[?:?]

at io.druid.segment.loading.OmniSegmentLoader.getSegmentFiles(OmniSegmentLoader.java:137) ~[druid-server-0.7.1.1.jar:0.7.1.1]

at io.druid.segment.loading.OmniSegmentLoader.getSegment(OmniSegmentLoader.java:93) ~[druid-server-0.7.1.1.jar:0.7.1.1]

at io.druid.server.coordination.ServerManager.loadSegment(ServerManager.java:150) ~[druid-server-0.7.1.1.jar:0.7.1.1]

at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:134) ~[druid-server-0.7.1.1.jar:0.7.1.1]

… 18 more

Caused by: org.jets3t.service.S3ServiceException: Request Error.

at org.jets3t.service.S3Service.getObjectDetails(S3Service.java:1775) ~[jets3t-0.9.3.jar:0.9.3]

at io.druid.storage.s3.S3Utils.isObjectInBucket(S3Utils.java:84) ~[?:?]

at io.druid.storage.s3.S3DataSegmentPuller$2.call(S3DataSegmentPuller.java:155) ~[?:?]

at io.druid.storage.s3.S3DataSegmentPuller$2.call(S3DataSegmentPuller.java:151) ~[?:?]

at com.metamx.common.RetryUtils.retry(RetryUtils.java:38) ~[java-util-0.26.15.jar:?]

at io.druid.storage.s3.S3Utils.retryS3Operation(S3Utils.java:77) ~[?:?]

at io.druid.storage.s3.S3DataSegmentPuller.isObjectInBucket(S3DataSegmentPuller.java:149) ~[?:?]

at io.druid.storage.s3.S3DataSegmentPuller.getSegmentFiles(S3DataSegmentPuller.java:69) ~[?:?]

at io.druid.segment.loading.OmniSegmentLoader.getSegmentFiles(OmniSegmentLoader.java:137) ~[druid-server-0.7.1.1.jar:0.7.1.1]

at io.druid.segment.loading.OmniSegmentLoader.getSegment(OmniSegmentLoader.java:93) ~[druid-server-0.7.1.1.jar:0.7.1.1]

at io.druid.server.coordination.ServerManager.loadSegment(ServerManager.java:150) ~[druid-server-0.7.1.1.jar:0.7.1.1]

at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:134) ~[druid-server-0.7.1.1.jar:0.7.1.1]

… 18 more

Caused by: org.jets3t.service.impl.rest.HttpException

at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:423) ~[jets3t-0.9.3.jar:0.9.3]

at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:277) ~[jets3t-0.9.3.jar:0.9.3]

at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRestHead(RestStorageService.java:1038) ~[jets3t-0.9.3.jar:0.9.3]

at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectImpl(RestStorageService.java:2250) ~[jets3t-0.9.3.jar:0.9.3]

at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectDetailsImpl(RestStorageService.java:2179) ~[jets3t-0.9.3.jar:0.9.3]

at org.jets3t.service.S3Service.getObjectDetails(S3Service.java:2574) ~[jets3t-0.9.3.jar:0.9.3]

at org.jets3t.service.S3Service.getObjectDetails(S3Service.java:1773) ~[jets3t-0.9.3.jar:0.9.3]

at io.druid.storage.s3.S3Utils.isObjectInBucket(S3Utils.java:84) ~[?:?]

at io.druid.storage.s3.S3DataSegmentPuller$2.call(S3DataSegmentPuller.java:155) ~[?:?]

at io.druid.storage.s3.S3DataSegmentPuller$2.call(S3DataSegmentPuller.java:151) ~[?:?]

at com.metamx.common.RetryUtils.retry(RetryUtils.java:38) ~[java-util-0.26.15.jar:?]

at io.druid.storage.s3.S3Utils.retryS3Operation(S3Utils.java:77) ~[?:?]

at io.druid.storage.s3.S3DataSegmentPuller.isObjectInBucket(S3DataSegmentPuller.java:149) ~[?:?]

at io.druid.storage.s3.S3DataSegmentPuller.getSegmentFiles(S3DataSegmentPuller.java:69) ~[?:?]

at io.druid.segment.loading.OmniSegmentLoader.getSegmentFiles(OmniSegmentLoader.java:137) ~[druid-server-0.7.1.1.jar:0.7.1.1]

at io.druid.segment.loading.OmniSegmentLoader.getSegment(OmniSegmentLoader.java:93) ~[druid-server-0.7.1.1.jar:0.7.1.1]

at io.druid.server.coordination.ServerManager.loadSegment(ServerManager.java:150) ~[druid-server-0.7.1.1.jar:0.7.1.1]

at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:134) ~[druid-server-0.7.1.1.jar:0.7.1.1]

… 18 more

The version of druid I used is “0.7.1.1”, and the “jets3t” package is the latest version.

I also check whether the access key and secret key are right. I can access the segments using “aws s3 ls” command from historical node.

Are you accessing one of the AWS s3 endpoints that requires v4 auth?

I have found the reason, the Access Key ID only have write permission, add the read permission will fix it.
Thanks charles very much.

在 2015年6月25日星期四 UTC+8上午12:40:44,charles.allen写道: