Historical can't pull segments from Digital Ocean Spaces (S3 compatible)

Hello,

I have been trying to configure druid to use Digital Ocean Spaces as deep storage provider (S3 compatible). I was able to index segments and push to the bucket, but historical cannot load these segments for some reason. I’m using 0.13.0-incubating version that uses AWS SDK to connect with S3-compatible services.

These are the logs from historical:

2019-03-11T10:08:25,106 INFO [ZkCoordinator] org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager - Deleting directory[/opt/druid/data/5556aa55-f493-4369-af3a-95778bff45db_digitalocean/2018-10-05T13:00:00.000Z_2018-10-05T14:00:00.000Z/2019-03-11T09:06:30.271Z/0]
2019-03-11T10:08:25,107 INFO [ZkCoordinator] org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager - Deleting directory[/opt/druid/data/5556aa55-f493-4369-af3a-95778bff45db_digitalocean/2018-10-05T13:00:00.000Z_2018-10-05T14:00:00.000Z/2019-03-11T09:06:30.271Z]
2019-03-11T10:08:25,108 INFO [ZkCoordinator] org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager - Deleting directory[/opt/druid/data/5556aa55-f493-4369-af3a-95778bff45db_digitalocean/2018-10-05T13:00:00.000Z_2018-10-05T14:00:00.000Z]
2019-03-11T10:08:25,109 INFO [ZkCoordinator] org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager - Deleting directory[/opt/druid/data/5556aa55-f493-4369-af3a-95778bff45db_digitalocean]
2019-03-11T10:08:25,110 INFO [ZkCoordinator] org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager - Asked to cleanup something[DataSegment{size=4923, shardSpec=NumberedShardSpec{partitionNum=0, partitions=0}, metrics=[events, tempInside, tempOutside, tempDiff, binPump, temperature, humidity, windSpeed, pressure], dimensions=[sensorID, zone, coordinates], version=‘2019-03-11T09:06:30.271Z’, loadSpec={type=>s3_zip, bucket=>jotalabs3, key=>druid/segments/5556aa55-f493-4369-af3a-95778bff45db_digitalocean/2018-10-05T13:00:00.000Z_2018-10-05T14:00:00.000Z/2019-03-11T09:06:30.271Z/0/index.zip, S3Schema=>s3n}, interval=2018-10-05T13:00:00.000Z/2018-10-05T14:00:00.000Z, dataSource=‘5556aa55-f493-4369-af3a-95778bff45db_digitalocean’, binaryVersion=‘9’}] that didn’t exist. Skipping.
2019-03-11T10:08:25,111 WARN [ZkCoordinator] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - No path to unannounce segment[5556aa55-f493-4369-af3a-95778bff45db_digitalocean_2018-10-05T13:00:00.000Z_2018-10-05T14:00:00.000Z_2019-03-11T09:06:30.271Z]
2019-03-11T10:08:25,111 INFO [ZkCoordinator] org.apache.druid.server.SegmentManager - Told to delete a queryable for a dataSource[5556aa55-f493-4369-af3a-95778bff45db_digitalocean] that doesn’t exist.
2019-03-11T10:08:25,111 INFO [ZkCoordinator] org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager - Asked to cleanup something[DataSegment{size=4923, shardSpec=NumberedShardSpec{partitionNum=0, partitions=0}, metrics=[events, tempInside, tempOutside, tempDiff, binPump, temperature, humidity, windSpeed, pressure], dimensions=[sensorID, zone, coordinates], version=‘2019-03-11T09:06:30.271Z’, loadSpec={type=>s3_zip, bucket=>jotalabs3, key=>druid/segments/5556aa55-f493-4369-af3a-95778bff45db_digitalocean/2018-10-05T13:00:00.000Z_2018-10-05T14:00:00.000Z/2019-03-11T09:06:30.271Z/0/index.zip, S3Schema=>s3n}, interval=2018-10-05T13:00:00.000Z/2018-10-05T14:00:00.000Z, dataSource=‘5556aa55-f493-4369-af3a-95778bff45db_digitalocean’, binaryVersion=‘9’}] that didn’t exist. Skipping.
2019-03-11T10:08:25,111 WARN [ZkCoordinator] org.apache.druid.server.coordination.SegmentLoadDropHandler - Unable to delete segmentInfoCacheFile[/opt/druid/data/info_dir/5556aa55-f493-4369-af3a-95778bff45db_digitalocean_2018-10-05T13:00:00.000Z_2018-10-05T14:00:00.000Z_2019-03-11T09:06:30.271Z]
2019-03-11T10:08:25,111 ERROR [ZkCoordinator] org.apache.druid.server.coordination.SegmentLoadDropHandler - Failed to load segment for dataSource: {class=org.apache.druid.server.coordination.SegmentLoadDropHandler, exceptionType=class org.apache.druid.segment.loading.SegmentLoadingException, exceptionMessage=Exception loading segment[5556aa55-f493-4369-af3a-95778bff45db_digitalocean_2018-10-05T13:00:00.000Z_2018-10-05T14:00:00.000Z_2019-03-11T09:06:30.271Z], segment=DataSegment{size=4923, shardSpec=NumberedShardSpec{partitionNum=0, partitions=0}, metrics=[events, tempInside, tempOutside, tempDiff, binPump, temperature, humidity, windSpeed, pressure], dimensions=[sensorID, zone, coordinates], version=‘2019-03-11T09:06:30.271Z’, loadSpec={type=>s3_zip, bucket=>jotalabs3, key=>druid/segments/5556aa55-f493-4369-af3a-95778bff45db_digitalocean/2018-10-05T13:00:00.000Z_2018-10-05T14:00:00.000Z/2019-03-11T09:06:30.271Z/0/index.zip, S3Schema=>s3n}, interval=2018-10-05T13:00:00.000Z/2018-10-05T14:00:00.000Z, dataSource=‘5556aa55-f493-4369-af3a-95778bff45db_digitalocean’, binaryVersion=‘9’}}
org.apache.druid.segment.loading.SegmentLoadingException: Exception loading segment[5556aa55-f493-4369-af3a-95778bff45db_digitalocean_2018-10-05T13:00:00.000Z_2018-10-05T14:00:00.000Z_2019-03-11T09:06:30.271Z]
at org.apache.druid.server.coordination.SegmentLoadDropHandler.loadSegment(SegmentLoadDropHandler.java:265) ~[druid-server-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.server.coordination.SegmentLoadDropHandler.addSegment(SegmentLoadDropHandler.java:307) [druid-server-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.server.coordination.SegmentChangeRequestLoad.go(SegmentChangeRequestLoad.java:47) [druid-server-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.server.coordination.ZkCoordinator$1.childEvent(ZkCoordinator.java:118) [druid-server-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:520) [curator-recipes-4.0.0.jar:4.0.0]
at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:514) [curator-recipes-4.0.0.jar:4.0.0]
at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:93) [curator-framework-4.0.0.jar:4.0.0]
at org.apache.curator.shaded.com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:296) [curator-client-4.0.0.jar:?]
at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:85) [curator-framework-4.0.0.jar:4.0.0]
at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:512) [curator-recipes-4.0.0.jar:4.0.0]
at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35) [curator-recipes-4.0.0.jar:4.0.0]
at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:771) [curator-recipes-4.0.0.jar:4.0.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_181]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_181]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_181]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
Caused by: org.apache.druid.segment.loading.SegmentLoadingException: Failed to load segment 5556aa55-f493-4369-af3a-95778bff45db_digitalocean_2018-10-05T13:00:00.000Z_2018-10-05T14:00:00.000Z_2019-03-11T09:06:30.271Z in all locations.
at org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager.loadSegmentWithRetry(SegmentLoaderLocalCacheManager.java:167) ~[druid-server-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegmentFiles(SegmentLoaderLocalCacheManager.java:133) ~[druid-server-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegment(SegmentLoaderLocalCacheManager.java:108) ~[druid-server-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.server.SegmentManager.getAdapter(SegmentManager.java:196) ~[druid-server-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.server.SegmentManager.loadSegment(SegmentManager.java:157) ~[druid-server-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.server.coordination.SegmentLoadDropHandler.loadSegment(SegmentLoadDropHandler.java:261) ~[druid-server-0.13.0-incubating.jar:0.13.0-incubating]
… 18 more
2019-03-11T10:08:25,114 INFO [ZkCoordinator] org.apache.druid.server.coordination.ZkCoordinator - Completed request [LOAD: 5556aa55-f493-4369-af3a-95778bff45db_digitalocean_2018-10-05T13:00:00.000Z_2018-10-05T14:00:00.000Z_2019-03-11T09:06:30.271Z]
2019-03-11T10:08:25,114 INFO [ZkCoordinator] org.apache.druid.server.coordination.ZkCoordinator - zNode[/druid/loadQueue/10.40.0.3:8083/5556aa55-f493-4369-af3a-95778bff45db_digitalocean_2018-10-05T13:00:00.000Z_2018-10-05T14:00:00.000Z_2019-03-11T09:06:30.271Z] was removed
2019-03-11T10:08:25,161 INFO [ZkCoordinator] org.apache.druid.server.coordination.ZkCoordinator - New request[LOAD: 5556aa55-f493-4369-af3a-95778bff45db_digitalocean_2018-10-05T12:00:00.000Z_2018-10-05T13:00:00.000Z_2019-03-11T09:06:30.271Z] with zNode[/druid/loadQueue/10.40.0.3:8083/5556aa55-f493-4369-af3a-95778bff45db_digitalocean_2018-10-05T12:00:00.000Z_2018-10-05T13:00:00.000Z_2019-03-11T09:06:30.271Z].
2019-03-11T10:08:25,161 INFO [ZkCoordinator] org.apache.druid.server.coordination.SegmentLoadDropHandler - Loading segment 5556aa55-f493-4369-af3a-95778bff45db_digitalocean_2018-10-05T12:00:00.000Z_2018-10-05T13:00:00.000Z_2019-03-11T09:06:30.271Z
2019-03-11T10:08:25,165 INFO [ZkCoordinator] org.apache.druid.storage.s3.S3DataSegmentPuller - Pulling index at path[s3://jotalabs3/druid/segments/5556aa55-f493-4369-af3a-95778bff45db_digitalocean/2018-10-05T12:00:00.000Z_2018-10-05T13:00:00.000Z/2019-03-11T09:06:30.271Z/0/index.zip] to outDir[/opt/druid/data/5556aa55-f493-4369-af3a-95778bff45db_digitalocean/2018-10-05T12:00:00.000Z_2018-10-05T13:00:00.000Z/2019-03-11T09:06:30.271Z/0]
2019-03-11T10:08:25,315 ERROR [ZkCoordinator] org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager - Failed to load segment in current location /opt/druid/data, try next location if any: {class=org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager, exceptionType=class org.apache.druid.segment.loading.SegmentLoadingException, exceptionMessage=Cannot find object for bucket[jotalabs3] and key[druid/segments/5556aa55-f493-4369-af3a-95778bff45db_digitalocean/2018-10-05T12:00:00.000Z_2018-10-05T13:00:00.000Z/2019-03-11T09:06:30.271Z/0/index.zip], location=/opt/druid/data}
org.apache.druid.segment.loading.SegmentLoadingException: Cannot find object for bucket[jotalabs3] and key[druid/segments/5556aa55-f493-4369-af3a-95778bff45db_digitalocean/2018-10-05T12:00:00.000Z_2018-10-05T13:00:00.000Z/2019-03-11T09:06:30.271Z/0/index.zip]
at org.apache.druid.storage.s3.S3DataSegmentPuller.getSegmentFiles(S3DataSegmentPuller.java:139) ~[?:?]
at org.apache.druid.storage.s3.S3LoadSpec.loadSegment(S3LoadSpec.java:60) ~[?:?]
at org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocation(SegmentLoaderLocalCacheManager.java:200) ~[druid-server-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocationWithStartMarker(SegmentLoaderLocalCacheManager.java:188) ~[druid-server-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager.loadSegmentWithRetry(SegmentLoaderLocalCacheManager.java:151) [druid-server-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegmentFiles(SegmentLoaderLocalCacheManager.java:133) [druid-server-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegment(SegmentLoaderLocalCacheManager.java:108) [druid-server-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.server.SegmentManager.getAdapter(SegmentManager.java:196) [druid-server-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.server.SegmentManager.loadSegment(SegmentManager.java:157) [druid-server-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.server.coordination.SegmentLoadDropHandler.loadSegment(SegmentLoadDropHandler.java:261) [druid-server-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.server.coordination.SegmentLoadDropHandler.addSegment(SegmentLoadDropHandler.java:307) [druid-server-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.server.coordination.SegmentChangeRequestLoad.go(SegmentChangeRequestLoad.java:47) [druid-server-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.server.coordination.ZkCoordinator$1.childEvent(ZkCoordinator.java:118) [druid-server-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:520) [curator-recipes-4.0.0.jar:4.0.0]
at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:514) [curator-recipes-4.0.0.jar:4.0.0]
at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:93) [curator-framework-4.0.0.jar:4.0.0]
at org.apache.curator.shaded.com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:296) [curator-client-4.0.0.jar:?]
at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:85) [curator-framework-4.0.0.jar:4.0.0]
at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:512) [curator-recipes-4.0.0.jar:4.0.0]
at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35) [curator-recipes-4.0.0.jar:4.0.0]
at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:771) [curator-recipes-4.0.0.jar:4.0.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_181]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_181]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_181]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
Caused by: org.apache.druid.java.util.common.ISE: Cannot find object for bucket[jotalabs3] and key[druid/segments/5556aa55-f493-4369-af3a-95778bff45db_digitalocean/2018-10-05T12:00:00.000Z_2018-10-05T13:00:00.000Z/2019-03-11T09:06:30.271Z/0/index.zip]
at org.apache.druid.storage.s3.S3Utils.getSingleObjectSummary(S3Utils.java:255) ~[?:?]
at org.apache.druid.storage.s3.S3DataSegmentPuller.buildFileObject(S3DataSegmentPuller.java:167) ~[?:?]
at org.apache.druid.storage.s3.S3DataSegmentPuller.access$000(S3DataSegmentPuller.java:57) ~[?:?]
at org.apache.druid.storage.s3.S3DataSegmentPuller$1.openStream(S3DataSegmentPuller.java:95) ~[?:?]
at org.apache.druid.java.util.common.CompressionUtils.lambda$unzip$0(CompressionUtils.java:169) ~[java-util-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:86) ~[java-util-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:114) ~[java-util-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:104) ~[java-util-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.java.util.common.CompressionUtils.unzip(CompressionUtils.java:168) ~[java-util-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.storage.s3.S3DataSegmentPuller.getSegmentFiles(S3DataSegmentPuller.java:108) ~[?:?]
… 27 more

``

And this is the common configuration I used for deep storage:

Deep Storage

druid.storage.type=s3
druid.storage.bucket=jotalabs3
druid.storage.baseKey=druid/segments
druid.s3.accessKey=<ACCESS_KEY>
druid.s3.secretKey=<SECRET_KEY>
druid.s3.endpoint.url=ams3.digitaloceanspaces.com
druid.s3.forceGlobalBucketAccessEnabled=true
druid.s3.enablePathStyleAccess=false

``

Any idea about what is going on? Using similar configuration with Amazon S3 works fine.

Thank you so much!

I’m not sure what’s going on, but Druid is using aws-java-sdk, so if you can find a way to get that to work by itself, you should be able to apply the same config to Druid. If not (maybe because Druid is missing a config parameter) then that could lead to a new Druid feature (to set that config parameter).

It’s weird, because using Digital Ocean Spaces as static data firehose works well, so Druid is able to push and pull files from S3. Only historical is problematic due to the exception I posted. I also have checked that aws-sdk works fine with Digital Ocean.

I am trying more configuration options, so I’ll keep you informed. Please tell me if you have found similar issues with historical of if somebody know how to fix that.

Thank you so much,

Alberto Rodríguez.