Druid indexingService (TaskRunner) not working with alternative S3 endpoints

Hi,

I’m running druid-services-0.6.154 and it has been working well for quite some time.

There we use S3 storage from an alternative provider and with jet3t and some configuration that has been working well.

The problem is that the indexing service spun up by the Overlord try to access amazon no matter what.

The Config as it is defined for all other roles (runtime.properties):

druid.s3.accessKey=

druid.s3.secretKey=

druid.storage.type=s3

druid.storage.bucket=

druid.storage.baseKey=

druid.storage.disableAcl=true

druid.pusher.s3.disableAcl=true

s3service.s3-endpoint=.com

The config as it is defined for all index service:

druid.indexer.fork.property.druid.indexer.hadoopWorkingPath=/tmp/druid-indexing

druid.indexer.fork.property.druid.computation.buffer.size=536870912

druid.indexer.fork.property.druid.processing.numThreads=3

druid.indexer.fork.property.druid.request.logging.type=file

druid.indexer.fork.property.druid.request.logging.dir=request_logs/

druid.indexer.fork.property.druid.segmentCache.locations=[{“path”: “/tmp/druid-indexing/zk_druid”, “maxSize”: 0}]

druid.indexer.fork.property.druid.server.http.numThreads=50

druid.indexer.fork.property.druid.storage.type=s3

druid.indexer.fork.property.druid.storage.baseKey=

druid.indexer.fork.property.druid.storage.bucket=

druid.indexer.fork.property.s3service.s3-endpoint=.com

I have tried to add this to the java ops (among other things).

Does anyone know what I can do to make the index service use the “s3service.s3-endpoint” setting ?

Regards,

-Stefan

Hi stefan,

I believe you need to add the s3 endpoint configuration to jets3t.properties file and add that to the classpath.

more info on jets3t.properties supported configs is here - http://www.jets3t.org/toolkit/configuration.html#jets3t

Hi,

I have that as well and keep in mind that this is already working fo all the other roles.

Regards,

-Stefán

HI again,

The problem seems to be that the indexing service is run with explicit classpath where all the jar files are listed.

I find no way for it to pick up the jets3t config file or to add it to that rtuntime generated classpath.

Any ideas?

Regards,

-Stefan

Hi Stefan,

I had the same problem and I solved it. Putting the jets3.properties on classpath when I run the middleManager but I need to put at the start of the classpath, like this:

ls /tmp/druid/middleManager/

jets3t.properties log4j.properties runtime.properties

``

java -XX:+UseConcMarkSweepGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -verbose:gc -server -Xmx64m -Xms64m -XX:+UseConcMarkSweepGC -XX:OnOutOfMemoryError=/opt/rb/var/sv/druid_middleManager/kill -Dlog4j.configuration=file:/tmp/druid/middleManager/log4j.properties -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djute.maxbuffer=8388608 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Djava.rmi.server.hostname=192.168.110.124 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=9085 -cp /tmp/druid/middleManager:/opt/rb/var/druid/app/druid-services.jar:/opt/rb/var/druid/app/postgresql.jar io.druid.cli.Main server middleManager

other options that you can try is added the jets3t.properties at peon’s classpath to do this, you can use this property on middleManager runtime.properties

druid.indexer.runner.javaOpts=

Regards,

Andres

Thank you. I had tried this already but I must have been doing something wrong.