Historical node can't pull the segments

Hi:
There are some trouble when historical node pull the segments from s3.

The exception is described as below:

2017-09-21 16:29:26,465 ERROR i.d.s.c.ZkCoordinator [ZkCoordinator-0] Failed to load segment for dataSource: {class=io.druid.server.coordination.ZkCoordinator, exceptionType=class io.druid.segment.loading.SegmentLoadingException, exceptionMessage=Exception loading segment[wikipedia_2015-09-01T00:00:00.000Z_2015-09-02T00:00:00.000Z_2017-09-21T16:27:24.875Z], segment=DataSegment{size=2898, shardSpec=NoneShardSpec, metrics=[count, latencyMs], dimensions=[url, user], version=‘2017-09-21T16:27:24.875Z’, loadSpec={type=s3_zip, bucket=sas, key=druid_test/segments/wikipedia/2015-09-01T00:00:00.000Z_2015-09-02T00:00:00.000Z/2017-09-21T16:27:24.875Z/0/index.zip, S3Schema=s3n}, interval=2015-09-01T00:00:00.000Z/2015-09-02T00:00:00.000Z, dataSource=‘wikipedia’, binaryVersion=‘9’}}

io.druid.segment.loading.SegmentLoadingException: Exception loading segment[wikipedia_2015-09-01T00:00:00.000Z_2015-09-02T00:00:00.000Z_2017-09-21T16:27:24.875Z]

at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:327) ~[druid-server-0.10.1.jar:0.10.1]

at io.druid.server.coordination.ZkCoordinator.addSegment(ZkCoordinator.java:368) [druid-server-0.10.1.jar:0.10.1]

at io.druid.server.coordination.SegmentChangeRequestLoad.go(SegmentChangeRequestLoad.java:44) [druid-server-0.10.1.jar:0.10.1]

at io.druid.server.coordination.ZkCoordinator$1.childEvent(ZkCoordinator.java:158) [druid-server-0.10.1.jar:0.10.1]

at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:522) [curator-recipes-2.11.0.jar:?]

at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:516) [curator-recipes-2.11.0.jar:?]

at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:93) [curator-framework-2.11.0.jar:?]

at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) [guava-16.0.1.jar:?]

at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:84) [curator-framework-2.11.0.jar:?]

at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:513) [curator-recipes-2.11.0.jar:?]

at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35) [curator-recipes-2.11.0.jar:?]

at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:773) [curator-recipes-2.11.0.jar:?]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_141]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_141]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_141]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_141]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_141]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]

Caused by: java.lang.RuntimeException: org.jets3t.service.ServiceException: Request Error. – ResponseCode: 403, ResponseStatus: Forbidden, RequestId: 717B417B7E3A0995, HostId: 9+elMd+iSJ+4AUq5Ks/bP6G9Npw5oBJt4/B0a8Xvee5rswu38mDyHLs0u+exGHBKUjwihgxfxZw=

at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.1.jar:?]

at io.druid.storage.s3.S3DataSegmentPuller.isObjectInBucket(S3DataSegmentPuller.java:328) ~[?:?]

at io.druid.storage.s3.S3DataSegmentPuller.getSegmentFiles(S3DataSegmentPuller.java:174) ~[?:?]

at io.druid.storage.s3.S3LoadSpec.loadSegment(S3LoadSpec.java:60) ~[?:?]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocation(SegmentLoaderLocalCacheManager.java:206) ~[druid-server-0.10.1.jar:0.10.1]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocationWithStartMarker(SegmentLoaderLocalCacheManager.java:195) ~[druid-server-0.10.1.jar:0.10.1]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadSegmentWithRetry(SegmentLoaderLocalCacheManager.java:154) ~[druid-server-0.10.1.jar:0.10.1]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegmentFiles(SegmentLoaderLocalCacheManager.java:130) ~[druid-server-0.10.1.jar:0.10.1]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegment(SegmentLoaderLocalCacheManager.java:105) ~[druid-server-0.10.1.jar:0.10.1]

at io.druid.server.SegmentManager.getAdapter(SegmentManager.java:197) ~[druid-server-0.10.1.jar:0.10.1]

at io.druid.server.SegmentManager.loadSegment(SegmentManager.java:158) ~[druid-server-0.10.1.jar:0.10.1]

at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:323) ~[druid-server-0.10.1.jar:0.10.1]

… 18 more

Caused by: org.jets3t.service.ServiceException: Request Error.

at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:426) ~[jets3t-0.9.4.jar:0.9.4]

at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:279) ~[jets3t-0.9.4.jar:0.9.4]

at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRestHead(RestStorageService.java:1052) ~[jets3t-0.9.4.jar:0.9.4]

at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectImpl(RestStorageService.java:2264) ~[jets3t-0.9.4.jar:0.9.4]

at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectDetailsImpl(RestStorageService.java:2193) ~[jets3t-0.9.4.jar:0.9.4]

at org.jets3t.service.StorageService.getObjectDetails(StorageService.java:1120) ~[jets3t-0.9.4.jar:0.9.4]

at org.jets3t.service.StorageService.getObjectDetails(StorageService.java:575) ~[jets3t-0.9.4.jar:0.9.4]

at io.druid.storage.s3.S3Utils.isObjectInBucket(S3Utils.java:96) ~[?:?]

at io.druid.storage.s3.S3DataSegmentPuller$4.call(S3DataSegmentPuller.java:319) ~[?:?]

at io.druid.storage.s3.S3DataSegmentPuller$4.call(S3DataSegmentPuller.java:315) ~[?:?]

at io.druid.java.util.common.RetryUtils.retry(RetryUtils.java:63) ~[java-util-0.10.1.jar:0.10.1]

at io.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81) ~[java-util-0.10.1.jar:0.10.1]

at io.druid.storage.s3.S3Utils.retryS3Operation(S3Utils.java:89) ~[?:?]

at io.druid.storage.s3.S3DataSegmentPuller.isObjectInBucket(S3DataSegmentPuller.java:313) ~[?:?]

at io.druid.storage.s3.S3DataSegmentPuller.getSegmentFiles(S3DataSegmentPuller.java:174) ~[?:?]

at io.druid.storage.s3.S3LoadSpec.loadSegment(S3LoadSpec.java:60) ~[?:?]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocation(SegmentLoaderLocalCacheManager.java:206) ~[druid-server-0.10.1.jar:0.10.1]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocationWithStartMarker(SegmentLoaderLocalCacheManager.java:195) ~[druid-server-0.10.1.jar:0.10.1]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadSegmentWithRetry(SegmentLoaderLocalCacheManager.java:154) ~[druid-server-0.10.1.jar:0.10.1]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegmentFiles(SegmentLoaderLocalCacheManager.java:130) ~[druid-server-0.10.1.jar:0.10.1]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegment(SegmentLoaderLocalCacheManager.java:105) ~[druid-server-0.10.1.jar:0.10.1]

at io.druid.server.SegmentManager.getAdapter(SegmentManager.java:197) ~[druid-server-0.10.1.jar:0.10.1]

at io.druid.server.SegmentManager.loadSegment(SegmentManager.java:158) ~[druid-server-0.10.1.jar:0.10.1]

at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:323) ~[druid-server-0.10.1.jar:0.10.1]

… 18 more

The druid version is 0.10.1.

I have double checked the s3 access key and secret key historical node configured, it’s all right.

And I can access the segments from s3 using the aws command on historical node.

Fix it, By this thread:https://groups.google.com/forum/#!searchin/druid-user/s3a|sort:relevance/druid-user/VYAySNm7PUw/uCE_vaaCBgAJ

在 2017年9月22日星期五 UTC+8上午12:40:01,tao写道: