S3 Get request issue

Dear Team,

Since last week I was trying to setting up S3 as a Deep data storage in druid but I was getting error while access in the data.

Would be great if you can help us to resolve this issue.

Getting below error in historical.log

019-03-25T09:03:26,235 ERROR [ZkCoordinator] io.druid.server.coordination.SegmentLoadDropHandler - Failed to load segment for dataSource: {class=io.druid.server.coordination.SegmentLoadDropHandler, exceptionType=class io.druid.segment.loading.SegmentLoadingException, exceptionMessage=Exception loading segment[sample-data_2015-09-12T00:00:00.000Z_2015-09-13T00:00:00.000Z_2019-03-22T07:18:37.993Z], segment=DataSegment{size=4826887, shardSpec=NumberedShardSpec{partitionNum=0, partitions=0}, metrics=, dimensions=[channel, cityName, comment, countryIsoCode, countryName, isAnonymous, isMinor, isNew, isRobot, isUnpatrolled, metroCode, namespace, page, regionIsoCode, regionName, user, added, deleted, delta], version=‘2019-03-22T07:18:37.993Z’, loadSpec={type=>s3_zip, bucket=>gi-druid, key=>druid/segments/sample-data/2015-09-12T00:00:00.000Z_2015-09-13T00:00:00.000Z/2019-03-22T07:18:37.993Z/0/index.zip, S3Schema=>s3n}, interval=2015-09-12T00:00:00.000Z/2015-09-13T00:00:00.000Z, dataSource=‘sample-data’, binaryVersion=‘9’}}

io.druid.segment.loading.SegmentLoadingException: Exception loading segment[sample-data_2015-09-12T00:00:00.000Z_2015-09-13T00:00:00.000Z_2019-03-22T07:18:37.993Z]

at io.druid.server.coordination.SegmentLoadDropHandler.loadSegment(SegmentLoadDropHandler.java:265) ~[druid-server-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]

at io.druid.server.coordination.SegmentLoadDropHandler.addSegment(SegmentLoadDropHandler.java:307) [druid-server-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]

at io.druid.server.coordination.SegmentChangeRequestLoad.go(SegmentChangeRequestLoad.java:47) [druid-server-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]

at io.druid.server.coordination.ZkCoordinator$1.childEvent(ZkCoordinator.java:118) [druid-server-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]

at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:520) [curator-recipes-4.0.0.jar:4.0.0]

at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:514) [curator-recipes-4.0.0.jar:4.0.0]

at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:93) [curator-framework-4.0.0.jar:4.0.0]

at org.apache.curator.shaded.com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:296) [curator-client-4.0.0.jar:?]

at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:85) [curator-framework-4.0.0.jar:4.0.0]

at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:512) [curator-recipes-4.0.0.jar:4.0.0]

at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35) [curator-recipes-4.0.0.jar:4.0.0]

at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:771) [curator-recipes-4.0.0.jar:4.0.0]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_112]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_112]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_112]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_112]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_112]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_112]

at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]

Caused by: java.lang.RuntimeException: org.jets3t.service.ServiceException: Request Error. – ResponseCode: 400, ResponseStatus: Bad Request, RequestId: 77A29FC0BC77C548, HostId: 3lzWI3pz67xokOKj1zYLvNCdzsK3BfVk8aAViMyayvd9DDmKo+VL1plzAWbvzsDFoD0+GU2ieRw=

at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.1.jar:?]

at io.druid.storage.s3.S3DataSegmentPuller.isObjectInBucket(S3DataSegmentPuller.java:327) ~[?:?]

at io.druid.storage.s3.S3DataSegmentPuller.getSegmentFiles(S3DataSegmentPuller.java:176) ~[?:?]

at io.druid.storage.s3.S3LoadSpec.loadSegment(S3LoadSpec.java:60) ~[?:?]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocation(SegmentLoaderLocalCacheManager.java:205) ~[druid-server-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocationWithStartMarker(SegmentLoaderLocalCacheManager.java:193) ~[druid-server-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadSegmentWithRetry(SegmentLoaderLocalCacheManager.java:151) ~[druid-server-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegmentFiles(SegmentLoaderLocalCacheManager.java:133) ~[druid-server-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegment(SegmentLoaderLocalCacheManager.java:108) ~[druid-server-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]

at io.druid.server.SegmentManager.getAdapter(SegmentManager.java:196) ~[druid-server-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]

at io.druid.server.SegmentManager.loadSegment(SegmentManager.java:157) ~[druid-server-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]

at io.druid.server.coordination.SegmentLoadDropHandler.loadSegment(SegmentLoadDropHandler.java:261) ~[druid-server-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]

… 18 more

Caused by: org.jets3t.service.ServiceException: Request Error.

at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:426) ~[jets3t-0.9.4.jar:0.9.4]

at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:279) ~[jets3t-0.9.4.jar:0.9.4]

at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRestHead(RestStorageService.java:1052) ~[jets3t-0.9.4.jar:0.9.4]

at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectImpl(RestStorageService.java:2264) ~[jets3t-0.9.4.jar:0.9.4]

at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectDetailsImpl(RestStorageService.java:2193) ~[jets3t-0.9.4.jar:0.9.4]

at org.jets3t.service.StorageService.getObjectDetails(StorageService.java:1120) ~[jets3t-0.9.4.jar:0.9.4]

at org.jets3t.service.StorageService.getObjectDetails(StorageService.java:575) ~[jets3t-0.9.4.jar:0.9.4]

at io.druid.storage.s3.S3Utils.isObjectInBucket(S3Utils.java:96) ~[?:?]

at io.druid.storage.s3.S3DataSegmentPuller$4.call(S3DataSegmentPuller.java:318) ~[?:?]

at io.druid.storage.s3.S3DataSegmentPuller$4.call(S3DataSegmentPuller.java:314) ~[?:?]

at io.druid.java.util.common.RetryUtils.retry(RetryUtils.java:63) ~[java-util-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]

at io.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81) ~[java-util-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]

at io.druid.storage.s3.S3Utils.retryS3Operation(S3Utils.java:89) ~[?:?]

at io.druid.storage.s3.S3DataSegmentPuller.isObjectInBucket(S3DataSegmentPuller.java:312) ~[?:?]

at io.druid.storage.s3.S3DataSegmentPuller.getSegmentFiles(S3DataSegmentPuller.java:176) ~[?:?]

at io.druid.storage.s3.S3LoadSpec.loadSegment(S3LoadSpec.java:60) ~[?:?]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocation(SegmentLoaderLocalCacheManager.java:205) ~[druid-server-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocationWithStartMarker(SegmentLoaderLocalCacheManager.java:193) ~[druid-server-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadSegmentWithRetry(SegmentLoaderLocalCacheManager.java:151) ~[druid-server-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegmentFiles(SegmentLoaderLocalCacheManager.java:133) ~[druid-server-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegment(SegmentLoaderLocalCacheManager.java:108) ~[druid-server-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]

at io.druid.server.SegmentManager.getAdapter(SegmentManager.java:196) ~[druid-server-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]

at io.druid.server.SegmentManager.loadSegment(SegmentManager.java:157) ~[druid-server-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]

at io.druid.server.coordination.SegmentLoadDropHandler.loadSegment(SegmentLoadDropHandler.java:261) ~[druid-server-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]

… 18 more

Caused by: org.jets3t.service.impl.rest.HttpException: 400 Bad Request

at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:425) ~[jets3t-0.9.4.jar:0.9.4]

at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:279) ~[jets3t-0.9.4.jar:0.9.4]

at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRestHead(RestStorageService.java:1052) ~[jets3t-0.9.4.jar:0.9.4]

at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectImpl(RestStorageService.java:2264) ~[jets3t-0.9.4.jar:0.9.4]

at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectDetailsImpl(RestStorageService.java:2193) ~[jets3t-0.9.4.jar:0.9.4]

at org.jets3t.service.StorageService.getObjectDetails(StorageService.java:1120) ~[jets3t-0.9.4.jar:0.9.4]

at org.jets3t.service.StorageService.getObjectDetails(StorageService.java:575) ~[jets3t-0.9.4.jar:0.9.4]

at io.druid.storage.s3.S3Utils.isObjectInBucket(S3Utils.java:96) ~[?:?]

at io.druid.storage.s3.S3DataSegmentPuller$4.call(S3DataSegmentPuller.java:318) ~[?:?]

at io.druid.storage.s3.S3DataSegmentPuller$4.call(S3DataSegmentPuller.java:314) ~[?:?]

at io.druid.java.util.common.RetryUtils.retry(RetryUtils.java:63) ~[java-util-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]

at io.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81) ~[java-util-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]

at io.druid.storage.s3.S3Utils.retryS3Operation(S3Utils.java:89) ~[?:?]

at io.druid.storage.s3.S3DataSegmentPuller.isObjectInBucket(S3DataSegmentPuller.java:312) ~[?:?]

at io.druid.storage.s3.S3DataSegmentPuller.getSegmentFiles(S3DataSegmentPuller.java:176) ~[?:?]

at io.druid.storage.s3.S3LoadSpec.loadSegment(S3LoadSpec.java:60) ~[?:?]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocation(SegmentLoaderLocalCacheManager.java:205) ~[druid-server-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocationWithStartMarker(SegmentLoaderLocalCacheManager.java:193) ~[druid-server-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadSegmentWithRetry(SegmentLoaderLocalCacheManager.java:151) ~[druid-server-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegmentFiles(SegmentLoaderLocalCacheManager.java:133) ~[druid-server-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegment(SegmentLoaderLocalCacheManager.java:108) ~[druid-server-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]

at io.druid.server.SegmentManager.getAdapter(SegmentManager.java:196) ~[druid-server-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]

at io.druid.server.SegmentManager.loadSegment(SegmentManager.java:157) ~[druid-server-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]

at io.druid.server.coordination.SegmentLoadDropHandler.loadSegment(SegmentLoadDropHandler.java:261) ~[druid-server-0.12.1.3.1.0.0-78.jar:0.12.1.3.1.0.0-78]

… 18 more

2019-03-25T09:03:26,242 INFO [ZkCoordinator] io.druid.server.coordination.ZkCoordinator - Completed request [LOAD: sample-data_2015-09-12T00:00:00.000Z_2015-09-13T00:00:00.000Z_2019-03-22T07:18:37.993Z]

broker.log (17.3 KB)

coordinator.log (19.1 KB)

historical.log (13.2 KB)

common.runtime.properties (1.88 KB)

Manish,

400 error typically happens on the client side. Check to see if the common configuration has the proper configuration to read/write from/to S3.

Paste here as well the Overlord log.

Rommel Garcia

Hi Rommel,

I am able to push the data in S3 segments but while fetching the records I was getting 400 bad request .

I have attached S3 configurations and overload.log .

below is jets3t.properties

/etc/druid/3.1.0.0-78/0/druid/_common/jets3t.properties

s3service.s3-endpoint=s3.ap-south-1.amazonaws.com

storage-service.request-signature-version=AWS4-HMAC-SHA256

uploads.stream-retry-buffer-size=2147483646

druid.s3.endpoint.signingRegion=ap-south-1

s3service.disable-dns-buckets=true

s3service.https-only=false

s3service.s3-endpoint-http-port=443

Thanks !!!

overload.log (17.3 KB)

common.runtime.properties (1.88 KB)