Cannot connect to S3 Service with a null path

Hi, I’m using the Cluster tutorial http://druid.io/docs/latest/tutorials/cluster.html to run Druid in the AWS environment. However, I’m keep getting this error message on a historical node.

016-12-27T20:16:28,435 INFO [ZkCoordinator-0] io.druid.server.coordination.ZkCoordinator - zNode[/druid/loadQueue/ip-172-36-0-203.ec2.internal:8083/session_2016-12-27T15:00:00.000Z_2016-12-27T16:00:00.000Z_2016-12-27T15:58:34.041Z] was removed

2016-12-27T20:16:28,437 INFO [ZkCoordinator-0] io.druid.server.coordination.ZkCoordinator - New request[LOAD: node_2016-12-27T15:00:00.000Z_2016-12-27T16:00:00.000Z_2016-12-27T15:58:24.526Z] with zNode[/druid/loadQueue/ip-172-36-0-203.ec2.internal:8083/node_2016-12-27T15:00:00.000Z_2016-12-27T16:00:00.000Z_2016-12-27T15:58:24.526Z].

2016-12-27T20:16:28,437 INFO [ZkCoordinator-0] io.druid.server.coordination.ZkCoordinator - Loading segment node_2016-12-27T15:00:00.000Z_2016-12-27T16:00:00.000Z_2016-12-27T15:58:24.526Z

2016-12-27T20:16:28,438 INFO [ZkCoordinator-0] io.druid.storage.s3.S3DataSegmentPuller - Pulling index at path[s3://druid_segments_bucket/druid/segments/node/2016-12-27T15:00:00.000Z_2016-12-27T16:00:00.000Z/2016-12-27T15:58:24.526Z/0/index.zip] to outDir[var/druid/segment-cache/node/2016-12-27T15:00:00.000Z_2016-12-27T16:00:00.000Z/2016-12-27T15:58:24.526Z/0]

2016-12-27T20:16:28,457 INFO [ZkCoordinator-0] io.druid.segment.loading.SegmentLoaderLocalCacheManager - Asked to cleanup something[DataSegment{size=49845, shardSpec=LinearShardSpec{partitionNum=0}, metrics=[cpu_load, memory_load], dimensions=[browser, node_id, os, user_id, long, cpu, storage, lat, memory], version=‘2016-12-27T15:58:24.526Z’, loadSpec={type=s3_zip, bucket=druid_segments_bucket, key=druid/segments/node/2016-12-27T15:00:00.000Z_2016-12-27T16:00:00.000Z/2016-12-27T15:58:24.526Z/0/index.zip}, interval=2016-12-27T15:00:00.000Z/2016-12-27T16:00:00.000Z, dataSource=‘node’, binaryVersion=‘9’}] that didn’t exist. Skipping.

2016-12-27T20:16:28,457 WARN [ZkCoordinator-0] io.druid.server.coordination.BatchDataSegmentAnnouncer - No path to unannounce segment[node_2016-12-27T15:00:00.000Z_2016-12-27T16:00:00.000Z_2016-12-27T15:58:24.526Z]

2016-12-27T20:16:28,457 INFO [ZkCoordinator-0] io.druid.server.coordination.ZkCoordinator - Completely removing [node_2016-12-27T15:00:00.000Z_2016-12-27T16:00:00.000Z_2016-12-27T15:58:24.526Z] in [30,000] millis

2016-12-27T20:16:28,459 INFO [ZkCoordinator-0] io.druid.server.coordination.ZkCoordinator - Completed request [LOAD: node_2016-12-27T15:00:00.000Z_2016-12-27T16:00:00.000Z_2016-12-27T15:58:24.526Z]

2016-12-27T20:16:28,460 ERROR [ZkCoordinator-0] io.druid.server.coordination.ZkCoordinator - Failed to load segment for dataSource: {class=io.druid.server.coordination.ZkCoordinator, exceptionType=class io.druid.segment.loading.SegmentLoadingException, exceptionMessage=Exception loading segment[node_2016-12-27T15:00:00.000Z_2016-12-27T16:00:00.000Z_2016-12-27T15:58:24.526Z], segment=DataSegment{size=49845, shardSpec=LinearShardSpec{partitionNum=0}, metrics=[cpu_load, memory_load], dimensions=[browser, node_id, os, user_id, long, cpu, storage, lat, memory], version=‘2016-12-27T15:58:24.526Z’, loadSpec={type=s3_zip, bucket=druid_segments_bucket, key=druid/segments/node/2016-12-27T15:00:00.000Z_2016-12-27T16:00:00.000Z/2016-12-27T15:58:24.526Z/0/index.zip}, interval=2016-12-27T15:00:00.000Z/2016-12-27T16:00:00.000Z, dataSource=‘node’, binaryVersion=‘9’}}

io.druid.segment.loading.SegmentLoadingException: Exception loading segment[node_2016-12-27T15:00:00.000Z_2016-12-27T16:00:00.000Z_2016-12-27T15:58:24.526Z]

at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:310) ~[druid-server-0.9.2.jar:0.9.2]

at io.druid.server.coordination.ZkCoordinator.addSegment(ZkCoordinator.java:351) [druid-server-0.9.2.jar:0.9.2]

at io.druid.server.coordination.SegmentChangeRequestLoad.go(SegmentChangeRequestLoad.java:44) [druid-server-0.9.2.jar:0.9.2]

at io.druid.server.coordination.ZkCoordinator$1.childEvent(ZkCoordinator.java:153) [druid-server-0.9.2.jar:0.9.2]

at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:522) [curator-recipes-2.11.0.jar:?]

at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:516) [curator-recipes-2.11.0.jar:?]

at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:93) [curator-framework-2.11.0.jar:?]

at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) [guava-16.0.1.jar:?]

at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:84) [curator-framework-2.11.0.jar:?]

at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:513) [curator-recipes-2.11.0.jar:?]

at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35) [curator-recipes-2.11.0.jar:?]

at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:773) [curator-recipes-2.11.0.jar:?]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [?:1.7.0_101]

at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_101]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [?:1.7.0_101]

at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_101]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_101]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_101]

at java.lang.Thread.run(Thread.java:745) [?:1.7.0_101]

Caused by: io.druid.segment.loading.SegmentLoadingException: org.jets3t.service.S3ServiceException: Cannot connect to S3 Service with a null path

at io.druid.storage.s3.S3DataSegmentPuller.getSegmentFiles(S3DataSegmentPuller.java:238) ~[?:?]

at io.druid.storage.s3.S3LoadSpec.loadSegment(S3LoadSpec.java:62) ~[?:?]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegmentFiles(SegmentLoaderLocalCacheManager.java:143) ~[druid-server-0.9.2.jar:0.9.2]

at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegment(SegmentLoaderLocalCacheManager.java:95) ~[druid-server-0.9.2.jar:0.9.2]

at io.druid.server.coordination.ServerManager.loadSegment(ServerManager.java:152) ~[druid-server-0.9.2.jar:0.9.2]

at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:306) ~[druid-server-0.9.2.jar:0.9.2]

I have the following config file for all nodes:

druid.extensions.loadList=[“druid-s3-extensions”]

druid.startup.logging.logProperties=true

druid.zk.service.host=172.36.0.161

druid.zk.paths.base=/druid

For Derby server on your Druid Coordinator (only viable in a cluster with a single Coordinator, no fail-over):

druid.metadata.storage.type=derby

druid.metadata.storage.connector.connectURI=jdbc:derby://172.36.0.161:1527/var/druid/metadata.db;create=true

druid.metadata.storage.connector.host=172.36.0.161

druid.metadata.storage.connector.port=1527

For S3:

druid.storage.type=s3

druid.storage.bucket=druid_segments_bucket

druid.storage.baseKey=druid/segments

druid.s3.accessKey=XXXXX

druid.s3.secretKey=XXXXXXX

For S3:

druid.indexer.logs.type=s3

druid.indexer.logs.s3Bucket=druid_index_bucket

druid.indexer.logs.s3Prefix=druid/indexing-logs

druid.selectors.indexing.serviceName=druid/overlord

druid.selectors.coordinator.serviceName=druid/coordinator

druid.monitoring.monitors=[“com.metamx.metrics.JvmMonitor”]

druid.emitter=logging

druid.emitter.logging.logLevel=info

I can confirm that on the S3 admin page all buckets have been created and populated with the segment files.

What could be wrong?

I have the same problem , do you slove the problem?

在 2016年12月28日星期三 UTC+8上午4:31:43,m…@nikolaypavlov.com写道:

Problem Solved !

because of my s3 bucket name contains underscores(’_’)

in java.net.URI hostname can’t contain underscores, if contains , uri.getHost will return null ,so exceptionMessage “Cannot connect to S3 Service with a null path”

in druid when download segments from s3,the uri=s3://bucketName/key…(which use http rest api)

so when get bucketName(uri.getHost) ,return null

best wishes

在 2016年12月28日星期三 UTC+8上午4:31:43,m…@nikolaypavlov.com写道: