Problem pushing s3object from realtime node

Hi all,

I am currently using druid 0.7.0, but when my realtime node try push object to my local S3 (riak-cs), it can’t push the object and report this:

2015-03-16_17:44:30.49674 2015-03-16T17:44:30,490 INFO [rb_monitor-2015-03-16T15:00:00.000Z-persist-n-merge] io.druid.storage.s3.S3DataSegmentPusher - Pushing S3Object [key=rbdata/rb_monitor/2015-03-16T15:00:00.000Z_2015-03-16T16:00:00.000Z/2015-03-16T15:00:00.000Z/0/index.zip, bucket=redborder, lastModified=null, dataInputStream=null, Metadata={x-amz-acl=bucket-owner-full-control, Content-Length=3146, Content-MD5=d/IMvCNLnHyzdzTNk+Txdw==, md5-hash=77f20cbc234b9c7cb37734cd93e4f177, Content-Type=application/zip}].

2015-03-16_17:44:31.85804 2015-03-16T17:44:31,856 ERROR [rb_monitor-2015-03-16T15:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Failed to persist merged index[rb_monitor]: {class=io.druid.segment.realtime.plumber.RealtimePlumber, exceptionType=class java.io.IOException, exceptionMessage=org.jets3t.service.S3ServiceException: Service Error Message. – ResponseCode: 403, ResponseStatus: Forbidden, XML Error Message: <?xml version="1.0" encoding="UTF-8"?>InvalidAccessKeyIdThe AWS Access Key Id you provided does not exist in our records.BEMDOVMREUPFF7OOWN9H3D9E9D8664CD67B1EA8TGdR71EoEJg/rQEQIzPr8kbv8HzUTjHYop4RNXnP6mnYBd5kf6r8tqDnVbwGrlIKXl1nJ00Y=, interval=2015-03-16T15:00:00.000Z/2015-03-16T16:00:00.000Z}

2015-03-16_17:44:31.85809 java.io.IOException: org.jets3t.service.S3ServiceException: Service Error Message. – ResponseCode: 403, ResponseStatus: Forbidden, XML Error Message: <?xml version="1.0" encoding="UTF-8"?>InvalidAccessKeyIdThe AWS Access Key Id you provided does not exist in our records.BEMDOVMREUPFF7OOWN9H3D9E9D8664CD67B1EA8TGdR71EoEJg/rQEQIzPr8kbv8HzUTjHYop4RNXnP6mnYBd5kf6r8tqDnVbwGrlIKXl1nJ00Y=

2015-03-16_17:44:31.85810 at io.druid.storage.s3.S3DataSegmentPusher.push(S3DataSegmentPusher.java:132) ~[?:?]

2015-03-16_17:44:31.85811 at io.druid.segment.realtime.plumber.RealtimePlumber$4.doRun(RealtimePlumber.java:447) [druid-services-0.7.0-selfcontained.jar:0.7.0]

2015-03-16_17:44:31.85811 at io.druid.common.guava.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:40) [druid-services-0.7.0-selfcontained.jar:0.7.0]

2015-03-16_17:44:31.85813 at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.7.0_03]

2015-03-16_17:44:31.85813 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.7.0_03]

2015-03-16_17:44:31.85813 at java.lang.Thread.run(Unknown Source) [?:1.7.0_03]

2015-03-16_17:44:31.85814 Caused by: org.jets3t.service.S3ServiceException: Service Error Message.

2015-03-16_17:44:31.85814 at org.jets3t.service.S3Service.putObject(S3Service.java:2069) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]

2015-03-16_17:44:31.85814 at io.druid.storage.s3.S3DataSegmentPusher$1.call(S3DataSegmentPusher.java:93) ~[?:?]

2015-03-16_17:44:31.85815 at io.druid.storage.s3.S3DataSegmentPusher$1.call(S3DataSegmentPusher.java:77) ~[?:?]

2015-03-16_17:44:31.85815 at com.metamx.common.RetryUtils.retry(RetryUtils.java:38) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]

2015-03-16_17:44:31.85816 at io.druid.storage.s3.S3Utils.retryS3Operation(S3Utils.java:77) ~[?:?]

2015-03-16_17:44:31.85816 at io.druid.storage.s3.S3DataSegmentPusher.push(S3DataSegmentPusher.java:75) ~[?:?]

2015-03-16_17:44:31.85816 … 5 more

``

I have a jets3t.properties on the realtime classpath. This is my jets3t.properties:

s3service.s3-endpoint=s3.redborder.cluster

s3service.https-only=false

s3service.s3-endpoint-http-port=8088

s3service.disable-dns-buckets=true

``

But my realtime try to push de s3Object to AWS S3, instead of upload to my local s3 …

Any idea about this?

Regards,

Andres Gomez

Hi Andres, is this something new that is broken because of the update to 0.7.0? If you turn on debug logging for jets3t, do you see the endpoint being overwritten?

Hi FY,

The problem isn’t new at 0.7.0 release. I had the problem on the old releases but I fixed it doing something like this:

Jets3tProperties jets3=new Jets3tProperties();
jets3.loadAndReplaceProperties(new FileInputStream("/root/Desktop/jets3t.properties"), "jets3t.properties”);


but … I think this isn’t a definitely solution.

I have run the realtime again using the debug mode on jet3st and I have could see this:

2015-03-17 08:24:59,077 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.Jets3tProperties - s3service.s3-endpoint=s3.amazonaws.com

2015-03-17 08:24:59,077 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.impl.rest.httpclient.RestStorageService - Creating object bucketName=redborder, objectKey=rbdata/rb_flow/2015-03-16T21:00:00.000Z_2015-03-16T22:00:00.000Z/2015-03-16T21:00:00.000Z/0/index.zip, storageClass=null. Content-Type=application/zip Including data? true Metadata: {x-amz-acl=bucket-owner-full-control, x-amz-storage-class=STANDARD, Content-Length=128121024, Content-MD5=ujqr11BThp/O7iEZOahI4g==, md5-hash=ba3aabd75053869fceee211939a848e2, Content-Type=application/zip} ACL: GSAccessControlList [owner=null, grants=[]]

2015-03-17 08:24:59,077 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.Jets3tProperties - s3service.disable-dns-buckets=false

2015-03-17 08:24:59,077 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.Jets3tProperties - **s3service.s3-endpoint=s3.amazonaws.com**

2015-03-17 08:24:59,077 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.Jets3tProperties - s3service.s3-endpoint-virtual-path=

2015-03-17 08:24:59,077 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.Jets3tProperties - **s3service.s3-endpoint-https-port=443**

2015-03-17 08:24:59,077 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.impl.rest.httpclient.RestStorageService - S3 URL: https://redborder.s3.amazonaws.com:443/rbdata%2Frb_flow%2F2015-03-16T21%3A00%3A00.000Z_2015-03-16T22%3A00%3A00.000Z%2F2015-03-16T21%3A00%3A00.000Z%2F0%2Findex.zip

2015-03-17 08:24:59,078 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.impl.rest.httpclient.RestStorageService - Performing PUT request for 'https://redborder.s3.amazonaws.com:443/rbdata%2Frb_flow%2F2015-03-16T21%3A00%3A00.000Z_2015-03-16T22%3A00%3A00.000Z%2F2015-03-16T21%3A00%3A00.000Z%2F0%2Findex.zip', expecting response codes: [200,204]

2015-03-17 08:24:59,079 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.impl.rest.httpclient.RestStorageService - Headers: [Date: Tue, 17 Mar 2015 08:24:59 GMT, x-amz-acl: bucket-owner-full-control, x-amz-storage-class: STANDARD, Content-MD5: ujqr11BThp/O7iEZOahI4g==, Content-Type: application/zip, x-amz-meta-md5-hash: ba3aabd75053869fceee211939a848e2]

2015-03-17 08:24:59,079 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.Jets3tProperties - httpclient.retry-max=5

2015-03-17 08:24:59,079 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.impl.rest.httpclient.RestStorageService - Adding authorization for Access Key 'BEMDOVMREUPFF7OOWN9H'.

2015-03-17 08:24:59,079 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.Jets3tProperties - s3service.s3-endpoint=s3.amazonaws.com

2015-03-17 08:24:59,079 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.impl.rest.httpclient.RestStorageService - For creating canonical string, using uri: /redborder/rbdata%2Frb_flow%2F2015-03-16T21%3A00%3A00.000Z_2015-03-16T22%3A00%3A00.000Z%2F2015-03-16T21%3A00%3A00.000Z%2F0%2Findex.zip

2015-03-17 08:24:59,080 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.impl.rest.httpclient.RestStorageService - Canonical string ('|' is a newline): PUT|ujqr11BThp/O7iEZOahI4g==|application/zip|Tue, 17 Mar 2015 08:24:59 GMT|x-amz-acl:bucket-owner-full-control|x-amz-meta-md5-hash:ba3aabd75053869fceee211939a848e2|x-amz-storage-class:STANDARD|/redborder/rbdata%2Frb_flow%2F2015-03-16T21%3A00%3A00.000Z_2015-03-16T22%3A00%3A00.000Z%2F2015-03-16T21%3A00%3A00.000Z%2F0%2Findex.zip

2015-03-17 08:24:59,080 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.utils.RestUtils$ThreadSafeConnManager - Get connection: {s}->https://redborder.s3.amazonaws.com:443, timeout = 0

2015-03-17 08:25:00,473 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.impl.rest.httpclient.RestStorageService - Response for 'PUT'. Content-Type: application/xml, Headers: [x-amz-request-id: DCF046C10174E725, x-amz-id-2: jARjSK979X7sJigbog2LkiwfVszNP5Scq/K65FzXtsCxnkt7qdmnEKQFc1cpLUIX, Content-Type: application/xml, Transfer-Encoding: chunked, Date: Tue, 17 Mar 2015 08:25:00 GMT, Connection: close, Server: AmazonS3]

2015-03-17 08:25:00,473 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.impl.rest.httpclient.RestStorageService - Response entity: org.apache.http.conn.BasicManagedEntity@587b8159

2015-03-17 08:25:00,473 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.impl.rest.httpclient.RestStorageService - Entity length: -1

2015-03-17 08:25:00,473 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.impl.rest.httpclient.RestStorageService - Received response code 403; matches one of expected set ([200, 204])? false

2015-03-17 08:25:00,473 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.impl.rest.httpclient.RestStorageService - Error response xml: true

2015-03-17 08:25:00,473 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.impl.rest.httpclient.RestStorageService - Error response entity: org.apache.http.conn.BasicManagedEntity@587b8159

2015-03-17 08:25:00,473 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.impl.rest.httpclient.RestStorageService - Error response entity length: -1

2015-03-17 08:25:00,473 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.impl.rest.httpclient.RestStorageService - Response '/rbdata%2Frb_flow%2F2015-03-16T21%3A00%3A00.000Z_2015-03-16T22%3A00%3A00.000Z%2F2015-03-16T21%3A00%3A00.000Z%2F0%2Findex.zip' - Received error response with XML message

2015-03-17 08:25:00,474 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.utils.RestUtils$ThreadSafeConnManager - Released connection is not reusable.

2015-03-17 08:25:00,474 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.impl.rest.httpclient.HttpMethodReleaseInputStream - Released HttpMethod as its response data stream is fully consumed

2015-03-17 08:25:00,475 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.impl.rest.httpclient.RestStorageService - Rethrowing as a ServiceException error in performRequest: org.jets3t.service.ServiceException: Service Error Message. -- ResponseCode: 403, ResponseStatus: null, XML Error Message: <?xml version="1.0" encoding="UTF-8"?><Error><Code>InvalidAccessKeyId</Code><Message>The AWS Access Key Id you provided does not exist in our records.</Message><AWSAccessKeyId>BEMDOVMREUPFF7OOWN9H</AWSAccessKeyId><RequestId>DCF046C10174E725</RequestId><HostId>jARjSK979X7sJigbog2LkiwfVszNP5Scq/K65FzXtsCxnkt7qdmnEKQFc1cpLUIX</HostId></Error>

2015-03-17 08:25:00,475 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.impl.rest.httpclient.RestStorageService - Releasing HttpClient connection after error: Service Error Message.

I have seen that the endpoint and port are:

2015-03-17 08:24:59,077 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.Jets3tProperties - **s3service.s3-endpoint=s3.amazonaws.com**

2015-03-17 08:24:59,077 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.Jets3tProperties - **s3service.s3-endpoint-https-port=443**

but in my jets3t.properties are:

> > `**s3service.s3-endpoint=s3. redborder.cluster**`

> > `**s3service.https-only=false**`

> > `**s3service.s3-endpoint-http- port=8088**`

> > `s3service.disable-dns-buckets= true`

any idea??

Regards and thanks FY.

Andrés Gómez

**Developer

I’ve looked into this a bit, ant it seems that you can specify a classpath in the overlord and include a jets3t.properties in the classpath to set how jets3t handles endpoints.

The major downside is that the classpath is not simply append able currently. Meaning that if you simply want to specify a jets3t.properties or log4j2.xml for the peons, you cannot do so without specifying the entire classpath.

Here’s an example of the parameter I set in the overlord. Only the first entry is my custom entry. The rest was grabbed from the log of a prior task run where the classpath option was NOT overridden:

-Ddruid.indexer.runner.classpath="/Users/charlesallen/bin/wrk/classpath:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/lib/ant-javafx.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/lib/dt.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/lib/javafx-doclet.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/lib/javafx-mx.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/lib/jconsole.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/lib/sa-jdi.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/lib/tools.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/jre/lib/charsets.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/jre/lib/deploy.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/jre/lib/htmlconverter.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/jre/lib/javaws.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/jre/lib/jce.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/jre/lib/jfr.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/jre/lib/jfxrt.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/jre/lib/jsse.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/jre/lib/management-agent.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/jre/lib/plugin.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/jre/lib/resources.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/jre/lib/rt.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/jre/lib/ext/dnsns.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/jre/lib/ext/localedata.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/jre/lib/ext/sunec.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/jre/lib/ext/sunjce_provider.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/jre/lib/ext/sunpkcs11.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/jre/lib/ext/zipfs.jar:/Users/charlesallen/src/druid/services/target/classes:/Users/charlesallen/src/druid/common/target/classes:/Users/charlesallen/.m2/repository/com/metamx/java-util/0.26.14/java-util-0.26.14.jar:/Users/charlesallen/.m2/repository/org/slf4j/slf4j-api/1.6.4/slf4j-api-1.6.4.jar:/Users/charlesallen/.m2/repository/net/sf/opencsv/opencsv/2.3/opencsv-2.3.jar:/Users/charlesallen/.m2/repository/io/druid/druid-api/0.3.4/druid-api-0.3.4.jar:/Users/charlesallen/.m2/repository/commons-codec/commons-codec/1.7/commons-codec-1.7.jar:/Users/charlesallen/.m2/repository/org/apache/commons/commons-dbcp2/2.0.1/commons-dbcp2-2.0.1.jar:/Users/charlesallen/.m2/repository/org/apache/commons/commons-pool2/2.2/commons-pool2-2.2.jar:/Users/charlesallen/.m2/repository/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar:/Users/charlesallen/.m2/repository/commons-pool/commons-pool/1.6/commons-pool-1.6.jar:/Users/charlesallen/.m2/repository/org/skife/config/config-magic/0.9/config-magic-0.9.jar:/Users/charlesallen/.m2/repository/org/hibernate/hibernate-validator/5.1.3.Final/hibernate-validator-5.1.3.Final.jar:/Users/charlesallen/.m2/repository/org/jboss/logging/jboss-logging/3.1.3.GA/jboss-logging-3.1.3.GA.jar:/Users/charlesallen/.m2/repository/com/fasterxml/classmate/1.0.0/classmate-1.0.0.jar:/Users/charlesallen/.m2/repository/javax/el/javax.el-api/3.0.0/javax.el-api-3.0.0.jar:/Users/charlesallen/.m2/repository/javax/validation/validation-api/1.1.0.Final/validation-api-1.1.0.Final.jar:/Users/charlesallen/.m2/repository/com/google/guava/guava/16.0.1/guava-16.0.1.jar:/Users/charlesallen/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.4.4/jackson-core-2.4.4.jar:/Users/charlesallen/.m2/repository/com/fasterxml/jackson/core/jackson-annotations/2.4.4/jackson-annotations-2.4.4.jar:/Users/charlesallen/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.4.4/jackson-databind-2.4.4.jar:/Users/charlesallen/.m2/repository/com/fasterxml/jackson/datatype/jackson-datatype-guava/2.4.4/jackson-datatype-guava-2.4.4.jar:/Users/charlesallen/.m2/repository/com/fasterxml/jackson/datatype/jackson-datatype-joda/2.4.4/jackson-datatype-joda-2.4.4.jar:/Users/charlesallen/.m2/repository/com/fasterxml/jackson/dataformat/jackson-dataformat-smile/2.4.4/jackson-dataformat-smile-2.4.4.jar:/Users/charlesallen/.m2/repository/com/google/inject/guice/4.0-beta/guice-4.0-beta.jar:/Users/charlesallen/.m2/repository/aopalliance/aopalliance/1.0/aopalliance-1.0.jar:/Users/charlesallen/.m2/repository/com/google/inject/extensions/guice-multibindings/4.0-beta/guice-multibindings-4.0-beta.jar:/Users/charlesallen/.m2/repository/org/jdbi/jdbi/2.32/jdbi-2.32.jar:/Users/charlesallen/.m2/repository/joda-time/joda-time/2.6/joda-time-2.6.jar:/Users/charlesallen/.m2/repository/com/google/code/findbugs/jsr305/2.0.1/jsr305-2.0.1.jar:/Users/charlesallen/.m2/repository/org/apache/logging/log4j/log4j-api/2.1/log4j-api-2.1.jar:/Users/charlesallen/.m2/repository/org/apache/logging/log4j/log4j-core/2.1/log4j-core-2.1.jar:/Users/charlesallen/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.1/log4j-slf4j-impl-2.1.jar:/Users/charlesallen/.m2/repository/com/lmax/disruptor/3.3.0/disruptor-3.3.0.jar:/Users/charlesallen/src/druid/server/target/classes:/Users/charlesallen/src/druid/processing/target/classes:/Users/charlesallen/.m2/repository/com/metamx/bytebuffer-collections/0.1.5/bytebuffer-collections-0.1.5.jar:/Users/charlesallen/.m2/repository/com/metamx/extendedset/1.3.8/extendedset-1.3.8.jar:/Users/charlesallen/.m2/repository/org/roaringbitmap/RoaringBitmap/0.4.5/RoaringBitmap-0.4.5.jar:/Users/charlesallen/.m2/repository/com/metamx/emitter/0.3.0/emitter-0.3.0.jar:/Users/charlesallen/.m2/repository/com/ning/compress-lzf/1.0.3/compress-lzf-1.0.3.jar:/Users/charlesallen/.m2/repository/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.jar:/Users/charlesallen/.m2/repository/com/ibm/icu/icu4j/4.8.1/icu4j-4.8.1.jar:/Users/charlesallen/.m2/repository/org/mozilla/rhino/1.7R4/rhino-1.7R4.jar:/Users/charlesallen/.m2/repository/org/mapdb/mapdb/1.0.6/mapdb-1.0.6.jar:/Users/charlesallen/src/druid/aws-common/target/classes:/Users/charlesallen/.m2/repository/com/amazonaws/aws-java-sdk/1.8.11/aws-java-sdk-1.8.11.jar:/Users/charlesallen/.m2/repository/com/amazonaws/aws-java-sdk-core/1.8.11/aws-java-sdk-core-1.8.11.jar:/Users/charlesallen/.m2/repository/com/metamx/http-client/1.0.0/http-client-1.0.0.jar:/Users/charlesallen/.m2/repository/io/netty/netty/3.9.5.Final/netty-3.9.5.Final.jar:/Users/charlesallen/.m2/repository/com/metamx/server-metrics/0.1.0/server-metrics-0.1.0.jar:/Users/charlesallen/.m2/repository/commons-cli/commons-cli/1.2/commons-cli-1.2.jar:/Users/charlesallen/.m2/repository/commons-lang/commons-lang/2.6/commons-lang-2.6.jar:/Users/charlesallen/.m2/repository/javax/inject/javax.inject/1/javax.inject-1.jar:/Users/charlesallen/.m2/repository/org/glassfish/javax.el/3.0.0/javax.el-3.0.0.jar:/Users/charlesallen/.m2/repository/org/apache/curator/curator-framework/2.7.0/curator-framework-2.7.0.jar:/Users/charlesallen/.m2/repository/org/apache/curator/curator-client/2.7.0/curator-client-2.7.0.jar:/Users/charlesallen/.m2/repository/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6.jar:/Users/charlesallen/.m2/repository/log4j/log4j/1.2.16/log4j-1.2.16.jar:/Users/charlesallen/.m2/repository/jline/jline/0.9.94/jline-0.9.94.jar:/Users/charlesallen/.m2/repository/org/apache/curator/curator-x-discovery/2.7.0/curator-x-discovery-2.7.0.jar:/Users/charlesallen/.m2/repository/org/apache/curator/curator-recipes/2.7.0/curator-recipes-2.7.0.jar:/Users/charlesallen/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.9.13/jackson-mapper-asl-1.9.13.jar:/Users/charlesallen/.m2/repository/com/fasterxml/jackson/jaxrs/jackson-jaxrs-json-provider/2.4.4/jackson-jaxrs-json-provider-2.4.4.jar:/Users/charlesallen/.m2/repository/com/fasterxml/jackson/jaxrs/jackson-jaxrs-base/2.4.4/jackson-jaxrs-base-2.4.4.jar:/Users/charlesallen/.m2/repository/com/fasterxml/jackson/module/jackson-module-jaxb-annotations/2.4.4/jackson-module-jaxb-annotations-2.4.4.jar:/Users/charlesallen/.m2/repository/com/fasterxml/jackson/jaxrs/jackson-jaxrs-smile-provider/2.4.4/jackson-jaxrs-smile-provider-2.4.4.jar:/Users/charlesallen/.m2/repository/com/sun/jersey/jersey-server/1.17.1/jersey-server-1.17.1.jar:/Users/charlesallen/.m2/repository/asm/asm/3.1/asm-3.1.jar:/Users/charlesallen/.m2/repository/com/sun/jersey/jersey-core/1.17.1/jersey-core-1.17.1.jar:/Users/charlesallen/.m2/repository/com/google/inject/extensions/guice-servlet/4.0-beta/guice-servlet-4.0-beta.jar:/Users/charlesallen/.m2/repository/com/sun/jersey/contribs/jersey-guice/1.17.1/jersey-guice-1.17.1.jar:/Users/charlesallen/.m2/repository/com/sun/jersey/jersey-servlet/1.17.1/jersey-servlet-1.17.1.jar:/Users/charlesallen/.m2/repository/org/eclipse/jetty/jetty-server/9.2.5.v20141112/jetty-server-9.2.5.v20141112.jar:/Users/charlesallen/.m2/repository/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar:/Users/charlesallen/.m2/repository/org/eclipse/jetty/jetty-http/9.2.5.v20141112/jetty-http-9.2.5.v20141112.jar:/Users/charlesallen/.m2/repository/org/eclipse/jetty/jetty-io/9.2.5.v20141112/jetty-io-9.2.5.v20141112.jar:/Users/charlesallen/.m2/repository/org/eclipse/jetty/jetty-proxy/9.2.5.v20141112/jetty-proxy-9.2.5.v20141112.jar:/Users/charlesallen/.m2/repository/org/eclipse/jetty/jetty-client/9.2.5.v20141112/jetty-client-9.2.5.v20141112.jar:/Users/charlesallen/.m2/repository/org/eclipse/jetty/jetty-util/9.2.5.v20141112/jetty-util-9.2.5.v20141112.jar:/Users/charlesallen/.m2/repository/io/tesla/aether/tesla-aether/0.0.5/tesla-aether-0.0.5.jar:/Users/charlesallen/.m2/repository/org/eclipse/aether/aether-spi/0.9.0.M2/aether-spi-0.9.0.M2.jar:/Users/charlesallen/.m2/repository/org/eclipse/aether/aether-util/0.9.0.M2/aether-util-0.9.0.M2.jar:/Users/charlesallen/.m2/repository/org/eclipse/aether/aether-impl/0.9.0.M2/aether-impl-0.9.0.M2.jar:/Users/charlesallen/.m2/repository/org/eclipse/aether/aether-connector-file/0.9.0.M2/aether-connector-file-0.9.0.M2.jar:/Users/charlesallen/.m2/repository/io/tesla/aether/aether-connector-okhttp/0.0.9/aether-connector-okhttp-0.0.9.jar:/Users/charlesallen/.m2/repository/com/squareup/okhttp/okhttp/1.0.2/okhttp-1.0.2.jar:/Users/charlesallen/.m2/repository/org/apache/maven/wagon/wagon-provider-api/2.4/wagon-provider-api-2.4.jar:/Users/charlesallen/.m2/repository/org/apache/maven/maven-aether-provider/3.1.1/maven-aether-provider-3.1.1.jar:/Users/charlesallen/.m2/repository/org/apache/maven/maven-model/3.1.1/maven-model-3.1.1.jar:/Users/charlesallen/.m2/repository/org/apache/maven/maven-model-builder/3.1.1/maven-model-builder-3.1.1.jar:/Users/charlesallen/.m2/repository/org/apache/maven/maven-repository-metadata/3.1.1/maven-repository-metadata-3.1.1.jar:/Users/charlesallen/.m2/repository/org/codehaus/plexus/plexus-utils/3.0.15/plexus-utils-3.0.15.jar:/Users/charlesallen/.m2/repository/org/apache/maven/maven-settings-builder/3.1.1/maven-settings-builder-3.1.1.jar:/Users/charlesallen/.m2/repository/org/codehaus/plexus/plexus-interpolation/1.19/plexus-interpolation-1.19.jar:/Users/charlesallen/.m2/repository/org/apache/maven/maven-settings/3.1.1/maven-settings-3.1.1.jar:/Users/charlesallen/.m2/repository/org/eclipse/aether/aether-api/0.9.0.M2/aether-api-0.9.0.M2.jar:/Users/charlesallen/.m2/repository/org/antlr/antlr4-runtime/4.0/antlr4-runtime-4.0.jar:/Users/charlesallen/.m2/repository/org/abego/treelayout/org.abego.treelayout.core/1.0.1/org.abego.treelayout.core-1.0.1.jar:/Users/charlesallen/.m2/repository/net/spy/spymemcached/2.11.4/spymemcached-2.11.4.jar:/Users/charlesallen/.m2/repository/net/jpountz/lz4/lz4/1.3.0/lz4-1.3.0.jar:/Users/charlesallen/.m2/repository/org/eclipse/jetty/jetty-servlet/9.2.5.v20141112/jetty-servlet-9.2.5.v20141112.jar:/Users/charlesallen/.m2/repository/org/eclipse/jetty/jetty-security/9.2.5.v20141112/jetty-security-9.2.5.v20141112.jar:/Users/charlesallen/.m2/repository/org/eclipse/jetty/jetty-servlets/9.2.5.v20141112/jetty-servlets-9.2.5.v20141112.jar:/Users/charlesallen/.m2/repository/org/eclipse/jetty/jetty-continuation/9.2.5.v20141112/jetty-continuation-9.2.5.v20141112.jar:/Users/charlesallen/.m2/repository/com/ircclouds/irc/irc-api/1.0-0011/irc-api-1.0-0011.jar:/Users/charlesallen/.m2/repository/com/maxmind/geoip2/geoip2/0.4.0/geoip2-0.4.0.jar:/Users/charlesallen/.m2/repository/com/maxmind/maxminddb/maxminddb/0.2.0/maxminddb-0.2.0.jar:/Users/charlesallen/.m2/repository/com/google/http-client/google-http-client/1.15.0-rc/google-http-client-1.15.0-rc.jar:/Users/charlesallen/.m2/repository/xpp3/xpp3/1.1.4c/xpp3-1.1.4c.jar:/Users/charlesallen/.m2/repository/com/google/http-client/google-http-client-jackson2/1.15.0-rc/google-http-client-jackson2-1.15.0-rc.jar:/Users/charlesallen/.m2/repository/org/apache/derby/derby/10.11.1.1/derby-10.11.1.1.jar:/Users/charlesallen/.m2/repository/org/apache/derby/derbynet/10.11.1.1/derbynet-10.11.1.1.jar:/Users/charlesallen/.m2/repository/org/apache/derby/derbyclient/10.11.1.1/derbyclient-10.11.1.1.jar:/Users/charlesallen/src/druid/indexing-hadoop/target/classes:/Users/charlesallen/.m2/repository/commons-io/commons-io/2.0.1/commons-io-2.0.1.jar:/Users/charlesallen/.m2/repository/net/java/dev/jets3t/jets3t/0.9.1/jets3t-0.9.1.jar:/Users/charlesallen/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.9.12/jackson-core-asl-1.9.12.jar:/Users/charlesallen/.m2/repository/javax/activation/activation/1.1.1/activation-1.1.1.jar:/Users/charlesallen/.m2/repository/mx4j/mx4j/3.0.2/mx4j-3.0.2.jar:/Users/charlesallen/.m2/repository/javax/mail/mail/1.4.7/mail-1.4.7.jar:/Users/charlesallen/.m2/repository/org/bouncycastle/bcprov-jdk15/1.46/bcprov-jdk15-1.46.jar:/Users/charlesallen/.m2/repository/com/jamesmurty/utils/java-xmlbuilder/0.4/java-xmlbuilder-0.4.jar:/Users/charlesallen/.m2/repository/org/apache/httpcomponents/httpclient/4.2/httpclient-4.2.jar:/Users/charlesallen/.m2/repository/org/apache/httpcomponents/httpcore/4.2/httpcore-4.2.jar:/Users/charlesallen/src/druid/indexing-service/target/classes:/Users/charlesallen/.m2/repository/io/airlift/airline/0.6/airline-0.6.jar:/Applications/IntelliJ IDEA 14 CE.app/Contents/lib/idea_rt.jar"

``

My jets3t.properties is the same as the one you posted, and I get the following when it is run:

org.jets3t.service.ServiceException: Request Error: s3.redborder.cluster: nodename nor servname provided, or not known

``

Which cis actually the error I was hoping for. It means the property properly propogates.

I’ll be filing an issue about this to allow easy usage of custom properties or log4j.xml stuff.

Sorry for the confusion.

Let me know if this workaround suffices.

Regards,

Charles Allen

Hi,

Hi Barney,

I finally solved the problem change the properties file’s position on my classpath, some like this:

This doesn’t work :

java -Xmn400420k -Xloggc:/var/log/druid/realtime/gc.log -verbose:gc -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1m -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintCommandLineFlags -XX:+PrintFlagsFinal -server -Xmx800841k -Xms800841k -Djava.io.tmpdir=/opt/rb/var/druid/realtime -XX:OnOutOfMemoryError=/opt/rb/var/sv/druid_realtime/kill -Dlog4j.configuration=file:/tmp/druid/realtime/log4j.properties -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djute.maxbuffer=8388608 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDetails -Djava.rmi.server.hostname=10.0.153.153 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=9082 -cp /opt/rb/var/druid/app/druid-services.jar:/opt/rb/var/druid/app/postgresql.jar:/tmp/druid/realtime io.druid.cli.Main server realtime

This work:

java -Xmn400420k -Xloggc:/var/log/druid/realtime/gc.log -verbose:gc -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1m -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintCommandLineFlags -XX:+PrintFlagsFinal -server -Xmx800841k -Xms800841k -Djava.io.tmpdir=/opt/rb/var/druid/realtime -XX:OnOutOfMemoryError=/opt/rb/var/sv/druid_realtime/kill -Dlog4j.configuration=file:/tmp/druid/realtime/log4j.properties -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djute.maxbuffer=8388608 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDetails -Djava.rmi.server.hostname=10.0.153.153 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=9082 -cp /tmp/druid/realtime:/opt/rb/var/druid/app/druid-services.jar:/opt/rb/var/druid/app/postgresql.jar io.druid.cli.Main server realtime

I have my jets3t.properties inside this directory /tmp/druid/realtime

Regards,

Andrés Gómez

Developer****

redborder.net / agomez@redborder.net

Phone: +34 955 60 11 60

0e6e8de_1.png

square-twitter-20.png square-google-plus-20.png square-linkedin-20.png

Piénsalo antes de imprimir este mensaje

Este correo electrónico, incluidos sus anexos, se dirige exclusivamente a su destinatario. Contiene información CONFIDENCIAL cuya divulgación está prohibida por la ley o puede estar sometida a secreto profesional. Si ha recibido este mensaje por error, le rogamos nos lo comunique inmediatamente y proceda a su destrucción.

This email, including attachments, is intended exclusively for its addressee. It contains information that is CONFIDENTIAL whose disclosure is prohibited by law and may be covered by legal privilege. If you have received this email in error, please notify the sender and delete it from your system.

En 20 de marzo de 2015 en 7:49:59, Barney Desmond (barneydesmond@gmail.com) escrito: